{"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef showDfs(d):\n if \"metadata\" in d:\n print(\"metadata\")\n if \"paleoData\" in d:\n try:\n for k, v in d[\"paleoData\"].items():\n print(k)\n except KeyError:\n pass\n except AttributeError:\n pass\n if \"chronData\" in d:\n try:\n for k, v in d[\"chronData\"].items():\n print(k)\n except KeyError:\n pass\n except AttributeError:\n pass\n # print(\"Process Complete\")\n return", "response": "Displays the available data frame names in a given data frame collection"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting a list of all LiPD names in the library", "response": "def getLipdNames(D=None):\n \"\"\"\n Get a list of all LiPD names in the library\n\n | Example\n | names = lipd.getLipdNames(D)\n\n :return list f_list: File list\n \"\"\"\n _names = []\n try:\n if not D:\n print(\"Error: LiPD data not provided. Pass LiPD data into the function.\")\n else:\n _names = D.keys()\n except Exception:\n pass\n return _names"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef getMetadata(L):\n _l = {}\n try:\n # Create a copy. Do not affect the original data.\n _l = copy.deepcopy(L)\n # Remove values fields\n _l = rm_values_fields(_l)\n except Exception as e:\n # Input likely not formatted correctly, though other problems can occur.\n print(\"Error: Unable to get data. Please check that input is LiPD data: {}\".format(e))\n return _l", "response": "Get metadata from a LiPD record in memory"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets CSV from LiPD metadata and data set", "response": "def getCsv(L=None):\n \"\"\"\n Get CSV from LiPD metadata\n\n | Example\n | c = lipd.getCsv(D[\"Africa-ColdAirCave.Sundqvist.2013\"])\n\n :param dict L: One LiPD record\n :return dict d: CSV data\n \"\"\"\n _c = {}\n try:\n if not L:\n print(\"Error: LiPD data not provided. Pass LiPD data into the function.\")\n else:\n _j, _c = get_csv_from_metadata(L[\"dataSetName\"], L)\n except KeyError as ke:\n print(\"Error: Unable to get data. Please check that input is one LiPD dataset: {}\".format(ke))\n except Exception as e:\n print(\"Error: Unable to get data. Something went wrong: {}\".format(e))\n logger_start.warn(\"getCsv: Exception: Unable to process lipd data: {}\".format(e))\n return _c"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nwrite LiPD data to file", "response": "def writeLipd(dat, path=\"\"):\n \"\"\"\n Write LiPD data to file(s)\n\n :param dict dat: Metadata\n :param str path: Destination (optional)\n :return none:\n \"\"\"\n global settings\n start = clock()\n __write_lipd(dat, path)\n end = clock()\n logger_benchmark.info(log_benchmark(\"writeLipd\", start, end))\n return"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nuses a file path to create file metadata and load a file in the appropriate way, according to the provided file type. :param str file_path: Path to file :param str file_type: One of approved file types: xls, xlsx, txt, lpd :return none:", "response": "def __universal_read(file_path, file_type):\n \"\"\"\n Use a file path to create file metadata and load a file in the appropriate way, according to the provided file type.\n\n :param str file_path: Path to file\n :param str file_type: One of approved file types: xls, xlsx, txt, lpd\n :return none:\n \"\"\"\n global files, cwd, settings\n\n # check that we are using the correct function to load this file type. (i.e. readNoaa for a .txt file)\n correct_ext = load_fn_matches_ext(file_path, file_type)\n\n # Check that this path references a file\n valid_path = path_type(file_path, \"file\")\n\n # is the path a file?\n if valid_path and correct_ext:\n\n # get file metadata for one file\n file_meta = collect_metadata_file(file_path)\n\n # append to global files, then load in D\n if file_type == \".lpd\":\n # add meta to global file meta\n files[\".lpd\"].append(file_meta)\n # append to global files\n elif file_type in [\".xls\", \".xlsx\"]:\n print(\"reading: {}\".format(print_filename(file_meta[\"full_path\"])))\n files[\".xls\"].append(file_meta)\n # append to global files\n elif file_type == \".txt\":\n print(\"reading: {}\".format(print_filename(file_meta[\"full_path\"])))\n files[\".txt\"].append(file_meta)\n\n\n # we want to move around with the files we load\n # change dir into the dir of the target file\n cwd = file_meta[\"dir\"]\n if cwd:\n os.chdir(cwd)\n\n return"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef __read(usr_path, file_type):\n # is there a file path specified ?\n if usr_path:\n # Is this a URL? Download the file and return the local path\n is_url = re.match(re_url, usr_path)\n if is_url:\n # The usr_path will now be a local path to a single file. It will trigger the \"elif\" statement below\n usr_path = download_from_url(usr_path, get_download_path())\n\n # Directory path\n if os.path.isdir(usr_path):\n __read_directory(usr_path, file_type)\n # File path\n elif os.path.isfile(usr_path):\n __read_file(usr_path, file_type)\n # Invalid path given\n else:\n print(\"Error: Path given is invalid\")\n\n # no path specified. ask if they want to load dir or file\n else:\n choice = \"\"\n count = 3\n while not choice:\n try:\n print(\"Choose a read option:\\n1. One file\\n2. Multi-file select\\n3. Directory\")\n choice = input(\"Option: \")\n print(\"\\n\")\n # now use the given file type and prompt answer to call _read_file or _read_dir\n if choice in [\"1\", \"2\", \"3\"]:\n # open directory picker\n if choice == \"3\":\n __read_directory(usr_path, file_type)\n else:\n # open a file picker\n __read_file(usr_path, file_type)\n break\n else:\n count -= 1\n if count == 0:\n print(\"Error: Too many failed attempts\")\n break\n except Exception as e:\n print(\"Error: Invalid input: {}\".format(e))\n return", "response": "Read in a single file in a tree tree."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nread in the LiPD file contents as a dataset library.", "response": "def __read_lipd_contents():\n \"\"\"\n Use the file metadata to read in the LiPD file contents as a dataset library\n\n :return dict: Metadata\n \"\"\"\n global files, settings\n _d = {}\n try:\n if len(files[\".lpd\"]) == 1:\n _d = lipd_read(files[\".lpd\"][0][\"full_path\"])\n if settings[\"verbose\"]:\n print(\"Finished read: 1 record\")\n else:\n for file in files[\".lpd\"]:\n _d[file[\"filename_no_ext\"]] = lipd_read(file[\"full_path\"])\n if settings[\"verbose\"]:\n print(\"Finished read: {} records\".format(len(_d)))\n except Exception as e:\n print(\"Error: read_lipd_contents: {}\".format(e))\n return _d"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef __read_file(usr_path, file_type):\n global files\n\n # no path provided. start gui browse\n if not usr_path:\n # src files could be a list of one, or a list of many. depending how many files the user selects\n src_dir, src_files = get_src_or_dst(\"read\", \"file\")\n # check if src_files is a list of multiple files\n if src_files:\n for file_path in src_files:\n __universal_read(file_path, file_type)\n else:\n print(\"No file(s) chosen\")\n else:\n __universal_read(usr_path, file_type)\n\n return", "response": "This function will read a file from the Universal read file system."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef __write_lipd(dat, usr_path):\n global settings\n # no path provided. start gui browse\n if not usr_path:\n # got dir path\n usr_path, _ignore = get_src_or_dst(\"write\", \"directory\")\n # Check if this is a valid directory path\n valid_path = path_type(usr_path, \"directory\")\n # If dir path is valid\n if valid_path:\n # Filename is given, write out one file\n if \"paleoData\" in dat:\n try:\n if settings[\"verbose\"]:\n print(\"writing: {}\".format(dat[\"dataSetName\"]))\n lipd_write(dat, usr_path)\n except KeyError as ke:\n print(\"Error: Unable to write file: unknown, {}\".format(ke))\n except Exception as e:\n print(\"Error: Unable to write file: {}, {}\".format(dat[\"dataSetName\"], e))\n # Filename is not given, write out whole library\n else:\n if dat:\n for name, lipd_dat in dat.items():\n try:\n if settings[\"verbose\"]:\n print(\"writing: {}\".format(name))\n lipd_write(lipd_dat, usr_path)\n except Exception as e:\n print(\"Error: Unable to write file: {}, {}\".format(name, e))\n\n return", "response": "Write LiPD data to file."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef __disclaimer(opt=\"\"):\n global settings\n if opt is \"update\":\n print(\"Disclaimer: LiPD files may be updated and modified to adhere to standards\\n\")\n settings[\"note_update\"] = False\n if opt is \"validate\":\n print(\"Note: Use lipd.validate() or www.LiPD.net/create \"\n \"to ensure that your new LiPD file(s) are valid\")\n settings[\"note_validate\"] = False\n return", "response": "Print the disclaimers once."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nadds a file to the internal list of resources.", "response": "def safe_add_file(filename, app):\n \"\"\"\n Adds files to builder resources only, if the given filename was not already registered.\n Needed mainly for tests to avoid multiple registration of the same file and therefore also multiple execution\n of e.g. a javascript file during page load.\n\n :param filename: filename to remove\n :param app: app object\n :return: None\n \"\"\"\n data_file = filename\n static_data_file = os.path.join(\"_static\", data_file)\n\n if data_file.split(\".\")[-1] == \"js\":\n if hasattr(app.builder, \"script_files\") and static_data_file not in app.builder.script_files:\n app.add_javascript(data_file)\n elif data_file.split(\".\")[-1] == \"css\":\n if hasattr(app.builder, \"css_files\") and static_data_file not in app.builder.css_files:\n app.add_stylesheet(data_file)\n else:\n raise NotImplementedError(\"File type {} not support by save_add_file\".format(data_file.split(\".\")[-1]))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nremoving a given file from the resource files list.", "response": "def safe_remove_file(filename, app):\n \"\"\"\n Removes a given resource file from builder resources.\n Needed mostly during test, if multiple sphinx-build are started.\n During these tests js/cass-files are not cleaned, so a css_file from run A is still registered in run B.\n\n :param filename: filename to remove\n :param app: app object\n :return: None\n \"\"\"\n data_file = filename\n static_data_file = os.path.join(\"_static\", data_file)\n\n if data_file.split(\".\")[-1] == \"js\":\n if hasattr(app.builder, \"script_files\") and static_data_file in app.builder.script_files:\n app.builder.script_files.remove(static_data_file)\n elif data_file.split(\".\")[-1] == \"css\":\n if hasattr(app.builder, \"css_files\") and static_data_file in app.builder.css_files:\n app.builder.css_files.remove(static_data_file)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_shreds(self, feature_extractors, sheet_name):\n if self._shreds is None:\n shreds = []\n _, contours, _ = cv2.findContours(self._foreground_mask,\n cv2.RETR_EXTERNAL,\n cv2.CHAIN_APPROX_SIMPLE)\n for i, contour in enumerate(contours):\n shred = self._make_shred(contour, i, feature_extractors,\n sheet_name)\n if shred is not None:\n shreds.append(shred)\n self._shreds = shreds\n return self._shreds", "response": "Detects shreds in the current sheet and constructs Shred instances."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _make_shred(self, c, name, feature_extractors, sheet_name):\n height, width, channels = self.orig_img.shape\n\n # bounding rect of currrent contour\n r_x, r_y, r_w, r_h = cv2.boundingRect(c)\n\n # Generating simplified contour to use it in html\n epsilon = 0.01 * cv2.arcLength(c, True)\n simplified_contour = cv2.approxPolyDP(c, epsilon, True)\n\n # filter out too small fragments\n if self.px_to_mm(r_w) <= 3 or self.px_to_mm(r_h) <= 3:\n print(\"Skipping piece #%s as too small (%spx x %s px)\" % (\n name, r_w, r_h))\n return None\n\n if self.px_to_mm(r_w) >= 100 and self.px_to_mm(r_h) >= 100:\n print(\"Skipping piece #%s as too big (%spx x %s px)\" % (\n name, r_w, r_h))\n return None\n\n # position of rect of min area.\n # this will provide us angle to straighten image\n box_center, bbox, angle = cv2.minAreaRect(c)\n\n # We want our pieces to be \"vertical\"\n if bbox[0] > bbox[1]:\n angle += 90\n bbox = (bbox[1], bbox[0])\n\n if bbox[1] / float(bbox[0]) > 70:\n print(\"Skipping piece #%s as too too long and narrow\" % name)\n return None\n\n # Coords of region of interest using which we should crop piece after\n # rotation\n y1 = math.floor(box_center[1] - bbox[1] / 2)\n x1 = math.floor(box_center[0] - bbox[0] / 2)\n bbox = tuple(map(int, map(math.ceil, bbox)))\n\n # A mask we use to show only piece we are currently working on\n piece_mask = np.zeros([height, width, 1], dtype=np.uint8)\n cv2.drawContours(piece_mask, [c], -1, 255, cv2.FILLED)\n\n # apply mask to original image\n img_crp = self.orig_img[r_y:r_y + r_h, r_x:r_x + r_w]\n piece_in_context = self.save_image(\n \"pieces/%s_ctx\" % name,\n self.orig_img[max(r_y - 10, 0):r_y + r_h + 10,\n max(r_x - 10, 0):r_x + r_w + 10])\n\n mask = piece_mask[r_y:r_y + r_h, r_x:r_x + r_w]\n img_roi = cv2.bitwise_and(img_crp, img_crp, mask=mask)\n\n # Add alpha layer and set it to the mask\n img_roi = cv2.cvtColor(img_roi, cv2.COLOR_BGR2BGRA)\n img_roi[:, :, 3] = mask[:, :, 0]\n\n # Straighten it\n # Because we crop original image before rotation we save us some memory\n # and a lot of time but we need to adjust coords of the center of\n # new min area rect\n M = cv2.getRotationMatrix2D((box_center[0] - r_x,\n box_center[1] - r_y), angle, 1)\n\n # And translate an image a bit to make it fit to the bbox again.\n # This is done with direct editing of the transform matrix.\n # (Wooohoo, I know matrix-fu)\n M[0][2] += r_x - x1\n M[1][2] += r_y - y1\n\n # Apply rotation/transform/crop\n img_roi = cv2.warpAffine(img_roi, M, bbox)\n piece_fname = self.save_image(\"pieces/%s\" % name, img_roi, \"png\")\n\n # FEATURES MAGIC BELOW\n #\n # Get our mask/contour back after the trasnform\n _, _, _, mask = cv2.split(img_roi)\n\n _, contours, _ = cv2.findContours(mask.copy(), cv2.RETR_TREE,\n cv2.CHAIN_APPROX_SIMPLE)\n\n if len(contours) != 1:\n print(\"Piece #%s has strange contours after transform\" % name)\n\n cnt = contours[0]\n\n features_fname = self.save_image(\"pieces/%s_mask\" % name, mask, \"png\")\n\n base_features = {\n # On_sheet_* features describe the min counding box on the sheet.\n \"on_sheet_x\": r_x,\n \"on_sheet_y\": r_y,\n \"on_sheet_width\": r_w,\n \"on_sheet_height\": r_h,\n \"on_sheet_angle\": angle,\n \"width\": img_roi.shape[1],\n \"height\": img_roi.shape[0],\n }\n\n tags_suggestions = []\n for feat in feature_extractors:\n fts, tags = feat.get_info(img_roi, cnt, name)\n base_features.update(fts)\n tags_suggestions += tags\n\n if tags_suggestions:\n print(name, tags_suggestions)\n\n return Shred(\n contour=c,\n features=base_features,\n features_fname=features_fname,\n img_roi=img_roi,\n name=name,\n piece_fname=piece_fname,\n piece_in_context_fname=piece_in_context,\n sheet=sheet_name,\n simplified_contour=simplified_contour,\n tags_suggestions=tags_suggestions,\n )", "response": "Creates a Shred instance from a given contour."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a WKT string for a POLYGON with given amount of sides and center and distance.", "response": "def geo_shape(self, sides=5, center=None, distance=None):\n \"\"\"\n Return a WKT string for a POLYGON with given amount of sides.\n The polygon is defined by its center (random point if not provided) and\n the distance (random distance if not provided; in km) of the points to\n its center.\n \"\"\"\n assert isinstance(sides, int)\n\n if distance is None:\n distance = self.random_int(100, 1000)\n else:\n # 6371 => earth radius in km\n # assert that shape radius is maximum half of earth's circumference\n assert isinstance(distance, int)\n assert distance <= EARTH_RADIUS * math.pi, \\\n 'distance must not be greater than half of earth\\'s circumference'\n\n if center is None:\n # required minimal spherical distance from north/southpole\n dp = distance * 180.0 / EARTH_RADIUS / math.pi\n center = self.geo_point(lat_min=-90.0 + dp, lat_max=90.0 - dp)\n else:\n assert -180.0 <= center[0] <= 180.0, 'Longitude out of bounds'\n assert -90.0 <= center[1] <= 90.0, 'Latitude out of bounds'\n\n angles = list(self.random_sample(range(360), sides))\n angles.sort()\n points = [_dest_point(center, distance, bearing, EARTH_RADIUS) for bearing in angles]\n # close polygon\n points.append(points[0])\n\n path = ', '.join([' '.join(p) for p in ([str(lon), str(lat)] for lon, lat in points)])\n return f'POLYGON (( {path} ))'"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef run_track(track,\n result_hosts=None,\n crate_root=None,\n output_fmt=None,\n logfile_info=None,\n logfile_result=None,\n failfast=False,\n sample_mode='reservoir'):\n \"\"\"Execute a track file\"\"\"\n with Logger(output_fmt=output_fmt,\n logfile_info=logfile_info,\n logfile_result=logfile_result) as log:\n executor = Executor(\n track_dir=os.path.dirname(track),\n log=log,\n result_hosts=result_hosts,\n crate_root=crate_root,\n fail_fast=failfast,\n sample_mode=sample_mode\n )\n error = executor.execute(toml.load(track))\n if error:\n sys.exit(1)", "response": "Execute a Toml file and return a list of result_hosts and logfile_result."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconvert a NOAA text file into a lipds file.", "response": "def main(self):\n \"\"\"\n Convert a NOAA text file into a lipds file.\n CSV files will be created if chronology or data sections are available.\n :return dict: Metadata Dictionary\n \"\"\"\n logger_noaa_lpd.info(\"enter main\")\n # Run the file through the parser\n # Sets self.metadata, Creates CSVs in dir_tmp\n os.chdir(self.dir_tmp)\n os.mkdir(\"bag\")\n os.chdir(self.dir_root)\n self.__parse()\n os.chdir(self.dir_bag)\n # Dump Metadata JSON to dir_tmp\n write_json_to_file(self.metadata)\n # Create a bagit bag\n finish_bag(self.dir_bag)\n logger_noaa_lpd.info(\"exit main\")\n return"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef __parse(self):\n logger_noaa_lpd.info(\"enter parse\")\n\n # Strings\n missing_str = ''\n data_filename = ''\n\n # Counters\n grant_id = 0\n funding_id = 0\n data_col_ct = 1\n line_num = 0\n\n # Boolean markers\n description_on = False\n publication_on = False\n abstract_on = False\n site_info_on = False\n chronology_on = False\n chron_vals_on = False\n variables_on = False\n data_vals_on = False\n data_on = False\n\n # Lists\n lat = []\n lon = []\n elev = []\n pub = []\n funding = []\n temp_abstract = []\n temp_description = []\n data_var_names = []\n data_col_list = []\n data_tables = []\n\n # All dictionaries needed to create JSON structure\n temp_funding = OrderedDict()\n temp_pub = OrderedDict()\n core_len = OrderedDict()\n geo_properties = OrderedDict()\n chron_dict = OrderedDict()\n data_dict_upper = OrderedDict()\n final_dict = OrderedDict()\n\n try:\n # Open the text file in read mode. We'll read one line at a time until EOF\n with open(self.filename_txt, 'r') as f:\n logger_noaa_lpd.info(\"opened noaa file: {}\".format(self.filename_txt))\n for line in iter(f):\n line_num += 1\n\n # PUBLICATION\n # There can be multiple publications. Create a dictionary for each one.\n if publication_on:\n # End of the section. Add the dictionary for this one publication to the overall list\n if '-----' in line:\n temp_pub = self.__reorganize_doi(temp_pub)\n pub.append(temp_pub.copy())\n temp_abstract.clear()\n temp_pub.clear()\n publication_on = False\n logger_noaa_lpd.info(\"end section: Publication\")\n elif abstract_on:\n # End of abstract: possibly more variables after.\n if \"#\" in line:\n abstract_on = False\n temp_pub['abstract'] = ''.join(temp_abstract)\n logger_noaa_lpd.info(\"end section: Abstract\")\n line = self.__str_cleanup(line)\n key, value = self.__slice_key_val(line)\n temp_pub[self.__camel_case(key)] = value\n else:\n temp_abstract.append(self.__str_cleanup(line))\n\n # Add all info into the current publication dictionary\n else:\n line = self.__str_cleanup(line)\n key, value = self.__slice_key_val(line)\n if key in (\"Author\", \"Authors\"):\n temp_pub[\"author\"] = self.__reorganize_authors(value)\n else:\n temp_pub[self.__camel_case(key)] = value\n if key == 'Abstract':\n logger_noaa_lpd.info(\"reading section: Abstract\")\n abstract_on = True\n temp_abstract.append(value)\n\n # DESCRIPTION AND NOTES\n # Descriptions are often long paragraphs spanning multiple lines, but don't follow the key/value format\n elif description_on:\n\n # End of the section. Turn marker off and combine all the lines in the section\n if '-------' in line:\n description_on = False\n value = ''.join(temp_description)\n final_dict['description'] = value\n logger_noaa_lpd.info(\"end section: Description_and_Notes\")\n\n # The first line in the section. Split into key, value\n elif 'Description:' in line:\n key, val = self.__slice_key_val(line)\n temp_description.append(val)\n\n # Keep a running list of all lines in the section\n else:\n line = self.__str_cleanup(line)\n temp_description.append(line)\n\n # SITE INFORMATION (Geo)\n elif site_info_on:\n\n if '-------' in line:\n site_info_on = False\n logger_noaa_lpd.info(\"end section: Site_Information\")\n\n else:\n line = self.__str_cleanup(line)\n key, value = self.__slice_key_val(line)\n if key.lower() in [\"northernmost_latitude\", \"southernmost_latitude\"]:\n lat.append(self.__convert_num(value))\n\n elif key.lower() in [\"easternmost_longitude\", \"westernmost_longitude\"]:\n lon.append(self.__convert_num(value))\n\n elif key.lower() in [\"site_name\", \"location\", \"country\", \"elevation\"]:\n if key.lower() == 'elevation':\n val, unit = self.__split_name_unit(value)\n elev.append(val)\n else:\n geo_properties[self.__camel_case(key)] = value\n\n # CHRONOLOGY\n elif chronology_on:\n \"\"\"\n HOW IT WORKS:\n Chronology will be started at \"Chronology:\" section header\n Every line starting with a \"#\" will be ignored\n The first line without a \"#\" is considered the variable header line. Variable names are parsed.\n Each following line will be considered column data and sorted accordingly.\n Once the \"-----\" barrier is reached, we exit the chronology section.\n \"\"\"\n # When reaching the end of the chron section, set the marker to off and close the CSV file\n if '-------' in line:\n # Turn off markers to exit section\n chronology_on = False\n chron_vals_on = False\n try:\n # If nothing between the chronology start and the end barrier, then there won't be a CSV\n if chron_start_line != line_num - 1:\n try:\n chron_csv.close()\n logger_noaa_lpd.info(\"parse: chronology: no data found in chronology section\")\n except NameError:\n logger_noaa_lpd.debug(\n \"parse: chronology_on: NameError: chron_csv ref before assignment, {}\".format(\n self.filename_txt))\n print(\n \"Chronology section is incorrectly formatted. \"\n \"Section data will not be converted\")\n\n logger_noaa_lpd.info(\"end section: Chronology\")\n except NameError:\n logger_noaa_lpd.debug(\n \"parse: chronology_on: NameError: chron_start_line ref before assignment, {}\".format(\n self.filename_txt))\n print(\"Chronology section is incorrectly formatted. Section data will not be converted\")\n\n # Data values line. Split, then write to CSV file\n elif chron_vals_on:\n values = line.split()\n try:\n cw.writerow(values)\n except NameError:\n logger_noaa_lpd.debug(\n \"parse: chronology_on: NameError: csv writer ref before assignment, {}\".format(\n self.filename_txt))\n print(\"Chronology section is incorrectly formatted. Section data will not be converted\")\n\n else:\n try:\n # Chron variable headers line\n if line and line[0] != \"#\":\n chron_filename = self.dsn + '.chron1.measurementTable1.csv'\n # Organize the var header into a dictionary\n variables = self.__reorganize_chron_header(line)\n\n # Create a dictionary of info for each column\n chron_col_list = self.__create_chron_cols(variables)\n chron_dict['filename'] = chron_filename\n chron_dict['chronTableName'] = 'Chronology'\n chron_dict['columns'] = chron_col_list\n\n # Open CSV for writing\n csv_path = os.path.join(self.dir_bag, chron_filename)\n chron_csv = open(csv_path, 'w+', newline='')\n logger_noaa_lpd.info(\"opened csv file: {}\".format(chron_filename))\n cw = csv.writer(chron_csv)\n # Turn the marker on to start processing the values columns\n chron_vals_on = True\n except IndexError:\n logger_noaa_lpd.debug(\"parse: chronology: IndexError when attempting chron var header\")\n\n # VARIABLES\n elif variables_on:\n \"\"\"\n HOW IT WORKS:\n Variable lines are the only lines that have a \"##\" in front of them.\n Ignore all lines that don't match the \"##\" regex.\n Once there's a match, start parsing the variable lines, and create a column entry for each line.\n \"\"\"\n process_line = False\n # End of the section. Turn marker off\n if \"------\" in line:\n variables_on = False\n logger_noaa_lpd.info(\"end section: Variables\")\n for item in NOAA_VAR_LINES:\n if item.lower() in line.lower():\n process_line = False\n for item in NOAA_EMPTY:\n if item == line:\n process_line = False\n m = re.match(re_var, line)\n if m:\n process_line = True\n\n # If the line isn't in the ignore list, then it's a variable line\n if process_line:\n\n # Split the line items, and cleanup\n cleaned_line = self.__separate_data_vars(line)\n\n # Add the items into a column dictionary\n data_col_dict = self.__create_paleo_col(cleaned_line, data_col_ct)\n\n # Keep a list of all variable names\n try:\n # Use this list later to cross check with the variable line in the Data section\n data_var_names.append(data_col_dict['variableName'])\n except KeyError:\n data_var_names.append('')\n logger_noaa_lpd.warn(\"parse: variables: \"\n \"KeyError: {} not found in {}\".format(\"variableName\", \"data_col_dict\"))\n # Add the column dictionary into a final dictionary\n data_col_list.append(data_col_dict)\n data_col_ct += 1\n\n # DATA\n # Missing Value, Create data columns, and output Data CSV\n elif data_on:\n \"\"\"\n HOW IT WORKS:\n Capture the \"Missing Value\" entry, if it exists.\n Data lines should not have a \"#\" in front of them.\n The first line without a \"#\" should be the variable header line\n All lines that follow should have column data.\n \"\"\"\n # Do not process blank or template lines\n process_line = True\n for item in NOAA_DATA_LINES:\n if item in line:\n process_line = False\n for item in NOAA_EMPTY:\n if item == line:\n process_line = False\n for item in ALTS_MV:\n # Missing value found. Store entry\n if item in line.lower():\n process_line = False\n line = self.__str_cleanup(line)\n key, missing_str = self.__slice_key_val(line)\n\n if process_line:\n # Split the line at each space (There SHOULD one space between each variable. Not always true)\n values = line.split()\n # Write all data values to CSV\n if data_vals_on:\n try:\n dw.writerow(values)\n except NameError:\n logger_noaa_lpd.debug(\n \"parse: data_on: NameError: csv writer ref before assignment, {}\".format(\n self.filename_txt))\n\n # Check for the line of variables\n else:\n var = self.__str_cleanup(values[0].lstrip())\n # Check if a variable name is in the current line\n if var.lower() in line.lower():\n data_vals_on = True\n logger_noaa_lpd.info(\"start section: Data_Values\")\n # Open CSV for writing\n data_filename = \"{}.paleoData1.measurementTable1.csv\".format(self.dsn)\n csv_path = os.path.join(self.dir_bag, data_filename)\n data_csv = open(csv_path, 'w+', newline='')\n logger_noaa_lpd.info(\"opened csv file: {}\".format(data_filename))\n dw = csv.writer(data_csv)\n\n # METADATA\n else:\n # Line Continuation: Sometimes there are items that span a few lines.\n # If this happens, we want to combine them all properly into one entry.\n if '#' not in line and line not in NOAA_EMPTY and old_val:\n if old_key in ('funding', 'agency'):\n try:\n temp_funding[old_key] = old_val + line\n except KeyError as e:\n logger_noaa_lpd.debug(\n \"parse: metadata: line continuation: {} not found in {}, {}\".format(old_key,\n \"temp_funding\",\n e))\n else:\n try:\n final_dict[old_key] = old_val + line\n except KeyError as e:\n logger_noaa_lpd.debug(\n \"parse: metadata: line continuation: {} not found in {}, {}\".format(old_key,\n \"temp_funding\",\n e))\n\n # No Line Continuation: This is the start or a new entry\n else:\n line = self.__str_cleanup(line)\n\n # Grab the key and value from the current line\n try:\n # Split the line into key, value pieces\n key, value = self.__slice_key_val(line)\n l_key = key.lower()\n cc_key= self.__camel_case(key)\n\n # If there is no value, then we are at a section header.\n # Data often has a blank value, so that is a special check.\n if not value or l_key == 'data':\n # Turn on markers if we run into section headers\n if l_key == 'description_and_notes':\n description_on = True\n logger_noaa_lpd.info(\"reading section: Description_and_Notes\")\n elif l_key == 'publication':\n publication_on = True\n logger_noaa_lpd.info(\"reading section: Publication\")\n elif l_key == 'site_information':\n site_info_on = True\n logger_noaa_lpd.info(\"reading section: Site_Information\")\n elif l_key == 'chronology':\n chronology_on = True\n logger_noaa_lpd.info(\"reading section: Chronology\")\n chron_start_line = line_num\n elif l_key == 'variables':\n variables_on = True\n logger_noaa_lpd.info(\"reading section: Variables\")\n elif l_key == 'data':\n data_on = True\n logger_noaa_lpd.info(\"reading section: Data\")\n # For all\n else:\n # Ignore any entries that are specified in the skip list\n _ignore = [item.lower() for item in NOAA_KEYS_BY_SECTION[\"Ignore\"]]\n if l_key not in _ignore:\n # There can be multiple funding agencies and grants. Keep a list of dict entries\n _funding = [item.lower() for item in NOAA_KEYS_BY_SECTION[\"Funding_Agency\"]]\n if l_key in _funding:\n if l_key == 'funding_agency_name':\n funding_id += 1\n key = 'agency'\n elif l_key == 'grant':\n grant_id += 1\n key = 'grant'\n temp_funding[key] = value\n # If both counters are matching, we are ready to add content to the funding list\n if grant_id == funding_id:\n funding.append(temp_funding.copy())\n temp_funding.clear()\n else:\n # There's likely two \"Online_Resource\"s, and we need both, so check and concat\n if cc_key == \"onlineResource\":\n # If it exists, append. If not, add entry as a list\n if cc_key in final_dict:\n final_dict[cc_key].append(value)\n else:\n final_dict[cc_key] = [value]\n else:\n final_dict[cc_key] = value\n # Keep track of old key in case we have a line continuation\n old_key = key\n old_val = value.strip()\n except TypeError as e:\n logger_noaa_lpd.warn(\n \"parse: TypeError: none type received from slice_key_val, {}\".format(e))\n\n # Wait to close the data CSV until we reached the end of the text file\n try:\n data_csv.close()\n logger_noaa_lpd.info(\"end section: Data_Values\")\n logger_noaa_lpd.info(\"end section: Data\")\n except NameError as e:\n print(\"Error: NOAA text file is contains format errors. Unable to process.\")\n logger_noaa_lpd.debug(\n \"parse: NameError: failed to close csv, invalid formatting in NOAA txt file, {}\".format(e))\n\n # Piece together measurements block\n logger_noaa_lpd.info(\"compiling final paleoData\")\n data_dict_upper['filename'] = data_filename\n data_dict_upper['paleoDataTableName'] = 'Data'\n data_dict_upper['missingValue'] = missing_str\n data_dict_upper['columns'] = data_col_list\n data_tables.append(data_dict_upper)\n\n # Piece together geo block\n logger_noaa_lpd.info(\"compiling final geo\")\n geo = self.__create_coordinates(lat, lon, elev)\n geo['properties'] = geo_properties\n\n # Piece together final dictionary\n logger_noaa_lpd.info(\"compiling final master\")\n final_dict['pub'] = pub\n final_dict['funding'] = funding\n final_dict['geo'] = geo\n final_dict['coreLength'] = core_len\n final_dict['chronData'] = [{\"chronMeasurementTable\": chron_dict}]\n final_dict['paleoData'] = data_tables\n self.metadata = final_dict\n logger_noaa_lpd.info(\"final dictionary compiled\")\n\n # Start cleaning up the metadata\n logger_noaa_lpd.info(\"removing empty fields\")\n self.metadata = rm_empty_fields(self.metadata)\n logger_noaa_lpd.info(\"removing empty doi\")\n self.metadata = rm_empty_doi(self.metadata)\n logger_noaa_lpd.info(\"removing irrelevant keys\")\n self.__remove_irr_fields()\n\n except Exception as e:\n logger_noaa_lpd.debug(\"parse: {}\".format(e))\n\n logger_noaa_lpd.info(\"exit parse\")\n return", "response": "Parse the noaa file and return a compiled dictionary"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef __create_paleo_col(l, col_count):\n # Format: what, material, error, units, seasonality, archive, detail, method,\n # C or N for Character or Numeric data, direction of relation to climate (positive or negative)\n d = OrderedDict()\n d['number'] = col_count\n for idx, var_name in enumerate(NOAA_KEYS_BY_SECTION[\"Variables\"]):\n try:\n value = l[idx]\n # These two cases are nested in the column, so treat them special\n if var_name == \"seasonality\":\n d[\"climateInterpretation\"] = {var_name: value}\n elif var_name == \"uncertainty\":\n d[\"calibration\"] = {var_name: value}\n # All other cases are root items in the column, so add normally\n else:\n d[var_name] = value\n except IndexError as e:\n logger_noaa_lpd.debug(\"create_var_col: IndexError: var: {}, {}\".format(var_name, e))\n return d", "response": "Create a column of paleo data."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts a number to a float.", "response": "def __convert_num(number):\n \"\"\"\n All path items are automatically strings. If you think it's an int or float, this attempts to convert it.\n :param str number:\n :return float or str:\n \"\"\"\n try:\n return float(number)\n except ValueError as e:\n logger_noaa_lpd.warn(\"convert_num: ValueError: {}\".format(e))\n return number"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef __camel_case(word):\n word = word.lower()\n if '_' in word:\n split_word = word.split('_')\n else:\n split_word = word.split()\n if len(split_word) > 0:\n for i, word in enumerate(split_word):\n if i > 0:\n split_word[i] = word.title()\n strings = ''.join(split_word)\n return strings", "response": "Convert underscore naming into camel case naming\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef __name_unit_regex(word):\n value = \"\"\n unit = \"\"\n r = re.findall(re_name_unit, word)\n try:\n value = r[0][0]\n except IndexError as e:\n logger_noaa_lpd.warn(\"name_unit_regex: IndexError: value: {}, {}, {}\".format(word, r, e))\n try:\n unit = r[0][1]\n # Replace unit with correct synonym.\n if unit.lower() in UNITS:\n unit = UNITS[unit]\n except IndexError as e:\n logger_noaa_lpd.warn(\"name_unit_regex: IndexError: unit: {}, {}, {}\".format(word, r, e))\n if value:\n try:\n value = float(value)\n except ValueError as e:\n logger_noaa_lpd.warn(\"name_unit_regex: ValueError: val: {}, {}\".format(value, e))\n return value, unit", "response": "Split a name and unit that are bunched together ( i. e. 250m"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef __split_name_unit(self, line):\n vals = []\n unit = ''\n if line != '' or line != ' ':\n # If there are parenthesis, remove them\n line = line.replace('(', '').replace(')', '')\n # When value and units are a range (i.e. '100 m - 200 m').\n if re.match(re_name_unit_range, line):\n m = re.findall(re_name_unit_range, line)\n if m:\n for group in m:\n for item in group:\n try:\n val = float(item)\n vals.append(val)\n except ValueError:\n if item:\n unit = item\n # Piece the number range back together.\n if len(vals) == 1:\n value = vals[0]\n else:\n value = str(vals[0]) + ' to ' + str(vals[1])\n else:\n value, unit = self.__name_unit_regex(line)\n return value, unit", "response": "Split a string that has value and unit as one."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nremoves unnecessary characters in the line that we don t want", "response": "def __str_cleanup(line):\n \"\"\"\n Remove the unnecessary characters in the line that we don't want\n :param str line:\n :return str:\n \"\"\"\n if '#' in line:\n line = line.replace(\"#\", \"\")\n line = line.strip()\n if '-----------' in line:\n line = ''\n return line"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nslicing the key and value items from a line.", "response": "def __slice_key_val(line):\n \"\"\"\n Get the key and value items from a line by looking for and lines that have a \":\"\n :param str line:\n :return str str: Key, Value\n \"\"\"\n position = line.find(\":\")\n # If value is -1, that means the item was not found in the string.\n if position != -1:\n key = line[:position]\n value = line[position + 1:]\n value = value.lstrip()\n return key, value\n else:\n key = line\n value = None\n return key, value"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef __create_coordinates(self, lat, lon, elev):\n # Sort lat an lon in numerical order\n lat.sort()\n lon.sort()\n geo_dict = {}\n # 4 coordinate values\n if len(lat) == 2 and len(lon) == 2:\n # Matching coordinate pairs. Not 4 unique values.\n if lat[0] == lat[1] and lon[0] == lon[1]:\n logger_noaa_lpd.info(\"coordinates found: {}\".format(\"2\"))\n lat.pop()\n lon.pop()\n geo_dict = self.__geo_point(lat, lon, elev)\n # 4 unique coordinates\n else:\n logger_noaa_lpd.info(\"coordinates found: {}\".format(\"4\"))\n geo_dict = self.__geo_multipoint(lat, lon, elev)\n # 2 coordinate values\n elif len(lat) == 1 and len(lon) == 1:\n logger_noaa_lpd.info(\"coordinates found: {}\".format(\"2\"))\n geo_dict = self.__geo_point(lat, lon, elev)\n # 0 coordinate values\n elif not lat and not lon:\n logger_noaa_lpd.info(\"coordinates found: {}\".format(\"0\"))\n else:\n geo_dict = {}\n logger_noaa_lpd.info(\"coordinates found: {}\".format(\"too many\"))\n return geo_dict", "response": "Create the coordinates for the current object."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef __geo_multipoint(lat, lon, elev):\n logger_noaa_lpd.info(\"enter geo_multipoint\")\n geo_dict = OrderedDict()\n geometry_dict = OrderedDict()\n coordinates = []\n # bbox = []\n temp = [None, None]\n # 4 unique values\n # # Creates bounding box\n # for index, point in enumerate(lat):\n # bbox.append(lat[index])\n # bbox.append(lon[index])\n\n # Creates coordinates list\n for i in lat:\n temp[0] = i\n for j in lon:\n temp[1] = j\n coordinates.append(copy.copy(temp))\n if elev:\n coordinates = coordinates + elev\n # Create geometry block\n geometry_dict['type'] = 'Polygon'\n geometry_dict['coordinates'] = coordinates\n # Create geo block\n geo_dict['type'] = 'Feature'\n # geo_dict['bbox'] = bbox\n geo_dict['geometry'] = geometry_dict\n\n return geo_dict", "response": "Create a geoJson MultiPoint - type dictionary"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef __geo_point(lat, lon, elev):\n logger_noaa_lpd.info(\"enter geo_point\")\n coordinates = []\n geo_dict = OrderedDict()\n geometry_dict = OrderedDict()\n for index, point in enumerate(lat):\n coordinates.append(lat[index])\n coordinates.append(lon[index])\n if elev:\n coordinates = coordinates + elev\n geometry_dict['type'] = 'Point'\n geometry_dict['coordinates'] = coordinates\n geo_dict['type'] = 'Feature'\n geo_dict['geometry'] = geometry_dict\n return geo_dict", "response": "Create a geoJson Point - type dictionary"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef __reorganize_doi(temp_pub):\n doi_out = \"\"\n doi_list = []\n rm = [\"doiId\", \"doi\", \"doiUrl\"]\n # Check if both entries exist\n if \"doi\" and \"doiId\" in temp_pub:\n if temp_pub[\"doi\"] == temp_pub[\"doiId\"]:\n # If the entries are the same, then pick one to use\n doi_out = temp_pub[\"doiId\"]\n else:\n # If entries are not the same, check if it matches the regex pattern.\n if re_doi.findall(temp_pub[\"doiId\"]):\n doi_out = temp_pub[\"doiId\"]\n # If it doesnt match the regex, just use the \"doi\" entry as-is\n else:\n doi_out = temp_pub[\"doi\"]\n # Check if \"doiId\" entry exists. Most common entry.\n elif \"doiId\" in temp_pub:\n doi_out = temp_pub[\"doiId\"]\n # Check if \"doi\" entry exists. Fallback.\n elif \"doi\" in temp_pub:\n doi_out = temp_pub[\"doi\"]\n # It's unlikely that ONLY the doi_url exists, but check if no doi found so far\n elif \"doiUrl\" in temp_pub:\n doi_out = temp_pub[\"doiUrl\"]\n\n # Log if the DOI is invalid or not.\n if not re_doi.match(doi_out):\n logger_noaa_lpd.info(\"reorganize_doi: invalid doi input from NOAA file\")\n\n # Get a list of all DOIs found in the string\n matches = re.findall(re_doi, doi_out)\n logger_noaa_lpd.info(\"reorganize_dois: found {} dois: {}\".format(len(matches), doi_out))\n\n # Add identifier block to publication dictionary\n for doi in matches:\n doi_list.append({\"type\": \"doi\", \"id\": doi})\n\n # Remove the other DOI entries\n for k in rm:\n try:\n del temp_pub[k]\n except KeyError:\n # If there's a KeyError, don't worry about it. It's likely that only one of these keys will be present.\n pass\n\n temp_pub[\"identifier\"] = doi_list\n return temp_pub", "response": "Reorganize the DOI from a dictionary."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef __create_chron_cols(metadata):\n chron_col_list = []\n chron_col_ct = 1\n for variableName, unit in metadata.items():\n temp_dict = OrderedDict()\n temp_dict['column'] = chron_col_ct\n temp_dict['variableName'] = variableName\n temp_dict['unit'] = unit\n chron_col_list.append(copy.deepcopy(temp_dict))\n chron_col_ct += 1\n\n return chron_col_list", "response": "Create the chron columns from the chron metadata."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef __reorganize_chron_header(line):\n d = {}\n # Header variables should be tab-delimited. Use regex to split by tabs\n m = re.split(re_tab_split, line)\n # If there was an output match from the line, then keep going\n if m:\n # Loop once for each variable in the line\n for s in m:\n # Match the variable to the 'variable (units)' regex to look for units\n m2 = re.match(re_var_w_units, s)\n # If there was a match\n if m2:\n # If no units were found, set to blank\n if m2.group(2) is None:\n d[m2.group(1)] = \"\"\n # Units were found\n else:\n # Set both values\n d[m2.group(1)] = m2.group(2)\n return d", "response": "Reorganize the list of variables given in the chron header."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef __reorganize_authors(authors):\n # String SHOULD be semi-colon separated names.\n l = []\n s = authors.split(\";\")\n for author in s:\n try:\n l.append({\"name\": author.strip()})\n except AttributeError:\n logger_noaa_lpd.warning(\"reorganize_authors: AttributeError: authors incorrectly formatted\")\n return l", "response": "Separate the string of authors and put it into a BibJSON compliant list\n "} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_lipd_version(L):\n version = 1.0\n _keys = [\"LipdVersion\", \"LiPDVersion\", \"lipdVersion\", \"liPDVersion\"]\n for _key in _keys:\n if _key in L:\n version = L[_key]\n # Cast the version number to a float\n try:\n version = float(version)\n except AttributeError:\n # If the casting failed, then something is wrong with the key so assume version is 1.0\n version = 1.0\n L.pop(_key)\n return L, version", "response": "Get the LiPD version from the metadata dictionary L."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nupdate the metadata of a single lipd version.", "response": "def update_lipd_version(L):\n \"\"\"\n Metadata is indexed by number at this step.\n\n Use the current version number to determine where to start updating from. Use \"chain versioning\" to make it\n modular. If a file is a few versions behind, convert to EACH version until reaching current. If a file is one\n version behind, it will only convert once to the newest.\n :param dict L: Metadata\n :return dict d: Metadata\n \"\"\"\n # Get the lipd version number.\n L, version = get_lipd_version(L)\n\n # Update from (N/A or 1.0) to 1.1\n if version in [1.0, \"1.0\"]:\n L = update_lipd_v1_1(L)\n version = 1.1\n\n # Update from 1.1 to 1.2\n if version in [1.1, \"1.1\"]:\n L = update_lipd_v1_2(L)\n version = 1.2\n if version in [1.2, \"1.2\"]:\n L = update_lipd_v1_3(L)\n version = 1.3\n\n L = fix_doi(L)\n L[\"lipdVersion\"] = 1.3\n return L"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nupdates LiPD v1. 1 to v1. 0", "response": "def update_lipd_v1_1(d):\n \"\"\"\n Update LiPD v1.0 to v1.1\n - chronData entry is a list that allows multiple tables\n - paleoData entry is a list that allows multiple tables\n - chronData now allows measurement, model, summary, modelTable, ensemble, calibratedAges tables\n - Added 'lipdVersion' key\n\n :param dict d: Metadata v1.0\n :return dict d: Metadata v1.1\n \"\"\"\n logger_versions.info(\"enter update_lipd_v1_1\")\n tmp_all = []\n\n try:\n # ChronData is the only structure update\n if \"chronData\" in d:\n # As of v1.1, ChronData should have an extra level of abstraction.\n # No longer shares the same structure of paleoData\n\n # If no measurement table, then make a measurement table list with the table as the entry\n for table in d[\"chronData\"]:\n if \"chronMeasurementTable\" not in table:\n tmp_all.append({\"chronMeasurementTable\": [table]})\n\n # If the table exists, but it is a dictionary, then turn it into a list with one entry\n elif \"chronMeasurementTable\" in table:\n if isinstance(table[\"chronMeasurementTable\"], dict):\n tmp_all.append({\"chronMeasurementTable\": [table[\"chronMeasurementTable\"]]})\n if tmp_all:\n d[\"chronData\"] = tmp_all\n\n # Log that this is now a v1.1 structured file\n d[\"lipdVersion\"] = 1.1\n except Exception as e:\n logger_versions.error(\"update_lipd_v1_1: Exception: {}\".format(e))\n logger_versions.info(\"exit update_lipd_v1_1\")\n return d"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update_lipd_v1_3(d):\n # sub routine (recursive): changes all the key names and merges interpretation\n d = update_lipd_v1_3_names(d)\n # sub routine: changes ensemble and summary table structure\n d = update_lipd_v1_3_structure(d)\n d[\"lipdVersion\"] = 1.3\n if \"LiPDVersion\" in d:\n del d[\"LiPDVersion\"]\n return d", "response": "Update LiPD v1. 2 to v1. 3."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update_lipd_v1_3_names(d):\n try:\n if isinstance(d, dict):\n for k, v in d.items():\n # dive down for dictionaries\n d[k] = update_lipd_v1_3_names(v)\n\n # see if the key is in the remove list\n if k in VER_1_3[\"swap\"]:\n # replace the key in the dictionary\n _key_swap = VER_1_3[\"swap\"][k]\n d[_key_swap] = d.pop(k)\n elif k in VER_1_3[\"tables\"]:\n d[k] = \"\"\n for _key in [\"climateInterpretation\", \"isotopeInterpretation\"]:\n if _key in d:\n d = _merge_interpretations(d)\n\n elif isinstance(d, list):\n # dive down for lists\n for idx, i in enumerate(d):\n d[idx] = update_lipd_v1_3_names(i)\n except Exception as e:\n print(\"Error: Unable to update file to LiPD v1.3: {}\".format(e))\n logger_versions.error(\"update_lipd_v1_3_names: Exception: {}\".format(e))\n return d", "response": "Update the key names and merge interpretations in a list of LiPD v1. 3."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update_lipd_v1_3_structure(d):\n for key in [\"paleoData\", \"chronData\"]:\n if key in d:\n for entry1 in d[key]:\n if \"model\" in entry1:\n for entry2 in entry1[\"model\"]:\n for key_table in [\"summaryTable\", \"ensembleTable\"]:\n if key_table in entry2:\n if isinstance(entry2[key_table], dict):\n try:\n _tmp = entry2[key_table]\n entry2[key_table] = []\n entry2[key_table].append(_tmp)\n except Exception as e:\n logger_versions.error(\"update_lipd_v1_3_structure: Exception: {}\".format(e))\n return d", "response": "Update the structure for summary and ensemble tables\n "} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nmerging CSV files into a metadata dictionary.", "response": "def merge_csv_metadata(d, csvs):\n \"\"\"\n Using the given metadata dictionary, retrieve CSV data from CSV files, and insert the CSV\n values into their respective metadata columns. Checks for both paleoData and chronData tables.\n\n :param dict d: Metadata\n :return dict: Modified metadata dictionary\n \"\"\"\n logger_csvs.info(\"enter merge_csv_metadata\")\n\n # Add CSV to paleoData\n if \"paleoData\" in d:\n d[\"paleoData\"] = _merge_csv_section(d[\"paleoData\"], \"paleo\", csvs)\n\n # Add CSV to chronData\n if \"chronData\" in d:\n d[\"chronData\"] = _merge_csv_section(d[\"chronData\"], \"chron\", csvs)\n\n logger_csvs.info(\"exit merge_csv_metadata\")\n return d"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _merge_csv_section(sections, pc, csvs):\n logger_csvs.info(\"enter merge_csv_section\")\n\n try:\n # Loop through each table_data in paleoData\n for _name, _section in sections.items():\n\n if \"measurementTable\" in _section:\n sections[_name][\"measurementTable\"] = _merge_csv_table(_section[\"measurementTable\"], pc, csvs)\n\n if \"model\" in _section:\n sections[_name][\"model\"] = _merge_csv_model(_section[\"model\"], pc, csvs)\n\n except Exception as e:\n print(\"Error: There was an error merging CSV data into the metadata \")\n logger_csvs.error(\"merge_csv_section: {}\".format(e))\n\n logger_csvs.info(\"exit merge_csv_section\")\n return sections", "response": "Add csv data to all paleo data tables and model data into the metadata tables"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nadding csv data to each column in chron model", "response": "def _merge_csv_model(models, pc, csvs):\n \"\"\"\n Add csv data to each column in chron model\n\n :param dict models: Metadata\n :return dict models: Metadata\n \"\"\"\n logger_csvs.info(\"enter merge_csv_model\")\n\n try:\n for _name, _model in models.items():\n\n if \"summaryTable\" in _model:\n models[_name][\"summaryTable\"] = _merge_csv_table(_model[\"summaryTable\"], pc, csvs)\n\n if \"ensembleTable\" in _model:\n models[_name][\"ensembleTable\"] = _merge_csv_table(_model[\"ensembleTable\"], pc, csvs)\n\n if \"distributionTable\" in _model:\n models[_name][\"distributionTable\"] = _merge_csv_table(_model[\"distributionTable\"], pc, csvs)\n\n except Exception as e:\n logger_csvs.error(\"merge_csv_model: {}\",format(e))\n\n logger_csvs.info(\"exit merge_csv_model\")\n return models"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _merge_csv_column(table, csvs):\n\n # Start putting CSV data into corresponding column \"values\" key\n try:\n ensemble = is_ensemble(table[\"columns\"])\n if ensemble:\n # realization columns\n if len(table[\"columns\"]) == 1:\n for _name, _column in table[\"columns\"].items():\n _column[\"values\"] = csvs\n # depth column + realization columns\n elif len(table[\"columns\"]) == 2:\n _multi_column = False\n for _name, _column in table[\"columns\"].items():\n if isinstance(_column[\"number\"], (int, float)):\n col_num = cast_int(_column[\"number\"])\n _column['values'] = csvs[col_num - 1]\n elif isinstance(_column[\"number\"], list):\n if _multi_column:\n raise Exception(\"Error: merge_csv_column: This jsonld metadata looks wrong!\\n\"\n \"\\tAn ensemble table depth should not reference multiple columns of CSV data.\\n\"\n \"\\tPlease manually fix the ensemble columns in 'metadata.jsonld' inside of your LiPD file.\")\n else:\n _multi_column = True\n _column[\"values\"] = csvs[2:]\n else:\n for _name, _column in table['columns'].items():\n col_num = cast_int(_column[\"number\"])\n _column['values'] = csvs[col_num - 1]\n except IndexError:\n logger_csvs.warning(\"merge_csv_column: IndexError: index out of range of csv_data list\")\n except KeyError:\n logger_csvs.error(\"merge_csv_column: KeyError: missing columns key\")\n except Exception as e:\n logger_csvs.error(\"merge_csv_column: Unknown Error: {}\".format(e))\n print(\"Quitting...\")\n exit(1)\n\n # We want to keep one missing value ONLY at the table level. Remove MVs if they're still in column-level\n return table, ensemble", "response": "Add csv data to each column in a list of columns."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreads the target CSV file and creates a dictionary with one list for each CSV column.", "response": "def read_csv_from_file(filename):\n \"\"\"\n Opens the target CSV file and creates a dictionary with one list for each CSV column.\n\n :param str filename:\n :return list of lists: column values\n \"\"\"\n logger_csvs.info(\"enter read_csv_from_file\")\n d = {}\n l = []\n try:\n logger_csvs.info(\"open file: {}\".format(filename))\n with open(filename, 'r') as f:\n r = csv.reader(f, delimiter=',')\n\n # Create a dict with X lists corresponding to X columns\n for idx, col in enumerate(next(r)):\n d[idx] = []\n d = cast_values_csvs(d, idx, col)\n\n # Start iter through CSV data\n for row in r:\n for idx, col in enumerate(row):\n # Append the cell to the correct column list\n d = cast_values_csvs(d, idx, col)\n\n # Make a list of lists out of the dictionary instead\n for idx, col in d.items():\n l.append(col)\n except FileNotFoundError as e:\n print('CSV FileNotFound: ' + filename)\n logger_csvs.warn(\"read_csv_to_columns: FileNotFound: {}, {}\".format(filename, e))\n logger_csvs.info(\"exit read_csv_from_file\")\n return l"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nwrite columns of data to a target CSV file.", "response": "def write_csv_to_file(d):\n \"\"\"\n Writes columns of data to a target CSV file.\n\n :param dict d: A dictionary containing one list for every data column. Keys: int, Values: list\n :return None:\n \"\"\"\n logger_csvs.info(\"enter write_csv_to_file\")\n\n try:\n for filename, data in d.items():\n try:\n l_columns = _reorder_csv(data, filename)\n rows = zip(*l_columns)\n with open(filename, 'w+') as f:\n w = csv.writer(f)\n for row in rows:\n row2 = decimal_precision(row)\n w.writerow(row2)\n except TypeError as e:\n print(\"Error: Unable to write values to CSV file, {}:\\n\"\n \"(1) The data table may have 2 or more identical variables. Please correct the LiPD file manually\\n\"\n \"(2) There may have been an error trying to prep the values for file write. The 'number' field in the data columns may be a 'string' instead of an 'integer' data type\".format(filename))\n print(e)\n except Exception as e:\n print(\"Error: CSV file not written, {}, {}:\\n\"\n \"The data table may have 2 or more identical variables. Please correct the LiPD file manually\".format(filename, e))\n except AttributeError as e:\n logger_csvs.error(\"write_csv_to_file: Unable to write CSV File: {}\".format(e, exc_info=True))\n logger_csvs.info(\"exit write_csv_to_file\")\n return"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_csv_from_metadata(dsn, d):\n logger_csvs.info(\"enter get_csv_from_metadata\")\n _csvs = OrderedDict()\n _d = copy.deepcopy(d)\n\n try:\n if \"paleoData\" in _d:\n # Process paleoData section\n _d[\"paleoData\"], _csvs = _get_csv_from_section(_d[\"paleoData\"], \"{}.paleo\".format(dsn), _csvs)\n\n if \"chronData\" in _d:\n _d[\"chronData\"], _csvs = _get_csv_from_section(_d[\"chronData\"], \"{}.chron\".format(dsn), _csvs)\n\n except Exception as e:\n print(\"Error: get_csv_from_metadata: {}, {}\".format(dsn, e))\n logger_csvs.error(\"get_csv_from_metadata: {}, {}\".format(dsn, e))\n\n logger_csvs.info(\"exit get_csv_from_metadata\")\n return _d, _csvs", "response": "Get all csv from metadata and return new metadata with generated filenames to match files."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _get_csv_from_section(sections, crumbs, csvs):\n logger_csvs.info(\"enter get_csv_from_section: {}\".format(crumbs))\n _idx = 0\n try:\n # Process the tables in section\n for _name, _section in sections.items():\n\n # Process each entry sub-table below if they exist\n if \"measurementTable\" in _section:\n sections[_name][\"measurementTable\"], csvs = _get_csv_from_table(_section[\"measurementTable\"],\"{}{}{}\".format(crumbs, _idx, \"measurement\") , csvs)\n if \"model\" in _section:\n sections[_name][\"model\"], csvs = _get_csv_from_model(_section[\"model\"], \"{}{}{}\".format(crumbs, _idx, \"model\") , csvs)\n _idx += 1\n\n except Exception as e:\n logger_csvs.error(\"get_csv_from_section: {}, {}\".format(crumbs, e))\n print(\"Error: get_csv_from_section: {}, {}\".format(crumbs, e))\n\n logger_csvs.info(\"exit get_csv_from_section: {}\".format(crumbs))\n return sections, csvs", "response": "Get the CSVs from the metadata and the measurements and models in the main section."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets csv from model data", "response": "def _get_csv_from_model(models, crumbs, csvs):\n \"\"\"\n Get csv from model data\n\n :param dict models: Metadata\n :param str crumbs: Crumbs\n :param dict csvs: Csv\n :return dict models: Metadata\n :return dict csvs: Csv\n \"\"\"\n logger_csvs.info(\"enter get_csv_from_model: {}\".format(crumbs))\n _idx = 0\n try:\n for _name, _model in models.items():\n if \"distributionTable\" in _model:\n models[_name][\"distributionTable\"], csvs = _get_csv_from_table(_model[\"distributionTable\"], \"{}{}{}\".format(crumbs, _idx, \"distribution\"), csvs)\n\n if \"summaryTable\" in _model:\n models[_name][\"summaryTable\"], csvs = _get_csv_from_table(_model[\"summaryTable\"], \"{}{}{}\".format(crumbs, _idx, \"summary\"), csvs)\n\n if \"ensembleTable\" in _model:\n models[_name][\"ensembleTable\"], csvs = _get_csv_from_table(_model[\"ensembleTable\"], \"{}{}{}\".format(crumbs, _idx, \"ensemble\"), csvs)\n _idx += 1\n except Exception as e:\n print(\"Error: get_csv_from_model: {}, {}\".format(crumbs, e))\n logger_csvs.error(\"Error: get_csv_from_model: {}, {}\".format(crumbs, e))\n return models, csvs"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_csv_from_columns(table, filename, csvs):\n csvs[filename] = OrderedDict()\n try:\n if \"columns\" in table:\n try:\n for _name, _column in table[\"columns\"].items():\n csvs[filename][_name] = {\"number\": _column[\"number\"], \"values\": _column[\"values\"]}\n except KeyError as ke:\n print(\"Error: get_csv_from_columns: {}, {}\".format(filename, ke))\n except Exception as e:\n print(\"Error: get_csv_from_columns: inner: {}, {}\".format(filename, e))\n logger_csvs.error(\"get_csv_from_columns: inner: {}, {}\".format(filename, e))\n except Exception as e:\n print(\"Error: get_csv_from_columns: {}, {}\".format(filename, e))\n logger_csvs.error(\"get_csv_from_columns: {}, {}\".format(filename, e))\n\n return csvs", "response": "Search a data tables for column values. Return a dict of column values. ref by var name"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting the filename from a data table.", "response": "def _get_filename(table):\n \"\"\"\n Get the filename from a data table. If it doesn't exist, create a new one based on table hierarchy in metadata file.\n format: .
.csv\n example: ODP1098B.Chron1.ChronMeasurementTable.csv\n\n :param dict table: Table data\n :param str crumbs: Crumbs\n :return str filename: Filename\n \"\"\"\n try:\n filename = table[\"filename\"]\n except KeyError:\n logger_csvs.info(\"get_filename: KeyError: missing filename for a table\")\n print(\"Error: Missing filename for a table\")\n filename = \"\"\n except Exception as e:\n logger_csvs.error(\"get_filename: {}\".format(e))\n filename = \"\"\n return filename"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _reorder_csv(d, filename=\"\"):\n _ensemble = is_ensemble(d)\n _d2 = []\n try:\n if _ensemble:\n # 1 column ensemble: realizations\n if len(d) == 1:\n for var, data in d.items():\n if \"values\" in data:\n _d2 = data[\"values\"]\n # 2 column ensemble: depth and realizations\n else:\n _count = 0\n # count up how many columns total, and how many placeholders to make in our list\n for var, data in d.items():\n if isinstance(data[\"number\"], list):\n _curr_count = len(data[\"number\"])\n _count += _curr_count\n elif isinstance(data[\"number\"], (int, float, str)):\n _count += 1\n # make a list with X number of placeholders\n _d2 = [None for i in range(0, _count)]\n # Loop again and start combining all columns into one list of lists\n for var, data in d.items():\n # realizations: insert at (hopefully) index 1,2...1001\n if isinstance(data[\"number\"], list):\n for idx, number in enumerate(data[\"number\"]):\n # we can't trust the number entries. sometimes they start at \"number 1\",\n # which isn't true, because DEPTH is number 1. Use enumerate index instead.\n _insert_at = int(idx) + 1\n # Insert at one above the index. Grab values at exact index\n _d2[_insert_at] = data[\"values\"][idx-1]\n\n # depth column: insert at (hopefully) index 0\n else:\n # we can trust to use the number entry as an index placement\n _insert_at = int(data[\"number\"]) - 1\n # insert at one below number, to compensate for 0-index\n _d2[_insert_at] = data[\"values\"]\n else:\n _count = len(d)\n _d2 = [None for i in range(0, _count)]\n for key, data in d.items():\n _insert_at = int(data[\"number\"]) - 1\n _d2[_insert_at] = data[\"values\"]\n\n except Exception as e:\n print(\"Error: Unable to write CSV: There was an error trying to prep the values for file write: {}\".format(e))\n logger_csvs.error(\"reorder_csvs: Unable to write CSV file: {}, {}\".format(filename, e))\n return _d2", "response": "Reorder the CSV file d in JSONLD alphabetical order."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncheck if the data contains numeric lists.", "response": "def _is_numeric_data(ll):\n \"\"\"\n List of lists of csv values data\n :param list ll:\n :return bool: True, all lists are numeric lists, False, data contains at least one numeric list.\n \"\"\"\n for l in ll:\n try:\n if any(math.isnan(float(i)) or isinstance(i, str) for i in l):\n return False\n # if not all(isinstance(i, (int, float)) or math.isnan(float(i)) for i in l):\n # # There is an entry that is a non-numeric entry in this list\n # return False\n except ValueError:\n # Trying to case a str as a float didnt work, and we got an error\n return False\n # All arrays are 100% numeric or \"nan\" entries.\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_validator_format(L):\n _api_data = []\n\n _j, _csvs = get_csv_from_metadata(L[\"dataSetName\"], L)\n _j = rm_values_fields(copy.deepcopy(L))\n _j = idx_name_to_num(_j)\n\n # All the filenames being processed\n _filenames = [\"metadata.jsonld\", \"bagit.txt\", \"bag-info.txt\", \"manifest-md5.txt\", \"tagmanifest-md5.txt\"]\\\n + [k for k,v in _csvs.items()]\n\n # Loop for each filename\n for filename in _filenames:\n # Create a blank template\n _file = {\"type\": \"\", \"filenameFull\": \"\", \"filenameShort\": \"\", \"data\": \"\", \"pretty\": \"\"}\n # filename, no path prefix\n # _short = os.path.basename(filename)\n _short = filename\n # Bagit files\n if filename.endswith(\".txt\"):\n _file = {\"type\": \"bagit\", \"filenameFull\": filename, \"filenameShort\": _short}\n # JSONLD files\n elif filename.endswith(\".jsonld\"):\n _file = {\"type\": \"json\", \"filenameFull\": filename, \"filenameShort\": _short, \"data\": _j}\n\n # CSV files\n elif filename.endswith(\".csv\"):\n _cols_rows = {\"cols\": 0, \"rows\": 0}\n ensemble = is_ensemble(_csvs[_short])\n # special case for calculating ensemble rows and columns\n if ensemble:\n _cols_rows = get_ensemble_counts(_csvs[_short])\n\n # all other non-ensemble csv files.\n else:\n _cols_rows[\"cols\"] = len(_csvs[_short])\n for k, v in _csvs[_short].items():\n _cols_rows[\"rows\"] = len(v[\"values\"])\n break\n\n # take what we've gathered for this file, and add it to the list.\n _file = {\"type\": \"csv\", \"filenameFull\": filename, \"filenameShort\": _short,\n \"data\": _cols_rows}\n\n _api_data.append(_file)\n\n return _api_data", "response": "Returns the format of the LIPD data in the layout that the Lipd. net validator accepts."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_detailed_results(result):\n string = \"\"\n # Validation Response output\n # string += \"VALIDATION RESPONSE\\n\"\n string += \"STATUS: {}\\n\".format(result[\"status\"])\n if result[\"feedback\"]:\n string += \"WARNINGS: {}\\n\".format(len(result[\"feedback\"][\"wrnMsgs\"]))\n # Loop through and output the Warnings\n for msg in result[\"feedback\"][\"wrnMsgs\"]:\n string += \"- {}\\n\".format(msg)\n string += \"ERRORS: {}\\n\".format(len(result[\"feedback\"][\"errMsgs\"]))\n # Loop through and output the Errors\n for msg in result[\"feedback\"][\"errMsgs\"]:\n string += \"- {}\\n\".format(msg)\n return string", "response": "Create a detailed string output for the file containing the results from the API call."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndisplays the results from the validator in a brief or off state", "response": "def display_results(results, detailed=False):\n \"\"\"\n Display the results from the validator in a brief or detailed output\n\n :param list results: API results, sorted by dataset name (multiple)\n :param bool detailed: Detailed results on or off\n :return none:\n \"\"\"\n # print(\"\\nVALIDATOR RESULTS\")\n # print(\"======================\\n\")\n\n if not detailed:\n print('FILENAME......................................... STATUS..........')\n\n for entry in results:\n try:\n if detailed:\n print(\"\\n{}\".format(entry[\"filename\"]))\n print(create_detailed_results(entry))\n else:\n print(\"{:<50}{}\".format(entry[\"filename\"], entry[\"status\"]))\n except Exception as e:\n logger_validator_api.debug(\"display_results: Exception: {}\".format(e))\n print(\"Error: display_results: {}\".format(e))\n\n return"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalling the LiPD. net validator API to get a list of files from one dataset.", "response": "def call_validator_api(dsn, api_data):\n \"\"\"\n Single call to the lipd.net validator API\n\n 'api_data' format:\n [\n {\"type\": \"csv\", \"filenameFull\": /path/to/filename.csv, \"data\": \"\", ...},\n {\"type\": \"json\", \"filenameFull\": /path/to/metadata.jsonld, \"data\": \"\", ...},\n ...\n ]\n Result format:\n {\"dat\": , \"feedback\": , \"filename\": \"\", \"status\": \"\"}\n\n :param str dsn: Dataset name\n :param list api_data: Prepared payload for one LiPD dataset. All the sorted files (txt, jsonld, csv), API formatted\n :return list result: Validator result for one file\n \"\"\"\n _filename = dsn + \".lpd\"\n\n try:\n # Contact server and send LiPD metadata as the payload\n # print(\"Sending request to LiPD.net validator...\\n\")\n api_data = json.dumps(api_data)\n\n # The payload that is going to be sent with the JSON request\n payload = {'json_payload': api_data, 'apikey': 'lipd_linked'}\n\n # Development Link\n # response = requests.post('http://localhost:3000/api/validator', data=payload)\n\n # Production Link\n response = requests.post('http://www.lipd.net/api/validator', data=payload)\n\n if response.status_code == 413:\n result = {\"dat\": {}, \"feedback\": {}, \"filename\": _filename,\n \"status\": \"HTTP 413: Request Entity Too Large\"}\n elif response.status_code == 404:\n result = {\"dat\": {}, \"feedback\": {}, \"filename\": _filename,\n \"status\": \"HTTP 404: Not Found\"}\n elif response.status_code == 400:\n result = {\"dat\": {}, \"feedback\": {}, \"filename\": _filename,\n \"status\": response.text}\n\n # For an example of the JSON Response, reference the \"sample_data_response\" below\n\n # Convert JSON string into a Python dictionary\n # print(\"Converting response to json...\\n\")\n else:\n result = json.loads(response.text)\n\n except TypeError as e:\n logger_validator_api.warning(\"get_validator_results: TypeError: {}\".format(e))\n result = {\"dat\": {}, \"feedback\": {}, \"filename\": _filename, \"status\": \"JSON DECODE ERROR\"}\n except requests.exceptions.ConnectionError as e:\n logger_validator_api.warning(\"get_validator_results: ConnectionError: {}\".format(e))\n result = {\"dat\": {}, \"feedback\": {}, \"filename\": _filename, \"status\": \"UNABLE TO REACH SERVER\"}\n except Exception as e:\n logger_validator_api.debug(\"get_validator_results: Exception: {}\".format(e))\n result = {\"dat\": {}, \"feedback\": {}, \"filename\": _filename, \"status\": \"ERROR BEFORE VALIDATION, {}\".format(e)}\n if not result:\n result = {\"dat\": {}, \"feedback\": {}, \"filename\": _filename, \"status\": \"EMPTY RESPONSE\"}\n\n result[\"filename\"] = _filename\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfilters all needs with given configuration and returns all needs which pass the filters", "response": "def procces_filters(all_needs, current_needlist):\n \"\"\"\n Filters all needs with given configuration\n\n :param current_needlist: needlist object, which stores all filters\n :param all_needs: List of all needs inside document\n\n :return: list of needs, which passed the filters\n \"\"\"\n\n if current_needlist[\"sort_by\"] is not None:\n if current_needlist[\"sort_by\"] == \"id\":\n all_needs = sorted(all_needs, key=lambda node: node[\"id\"])\n elif current_needlist[\"sort_by\"] == \"status\":\n all_needs = sorted(all_needs, key=status_sorter)\n\n found_needs_by_options = []\n\n # Add all need_parts of given needs to the search list\n all_needs_incl_parts = prepare_need_list(all_needs)\n\n for need_info in all_needs_incl_parts:\n status_filter_passed = False\n if current_needlist[\"status\"] is None or len(current_needlist[\"status\"]) == 0:\n # Filtering for status was not requested\n status_filter_passed = True\n elif need_info[\"status\"] is not None and need_info[\"status\"] in current_needlist[\"status\"]:\n # Match was found\n status_filter_passed = True\n\n tags_filter_passed = False\n if len(set(need_info[\"tags\"]) & set(current_needlist[\"tags\"])) > 0 or len(current_needlist[\"tags\"]) == 0:\n tags_filter_passed = True\n\n type_filter_passed = False\n if need_info[\"type\"] in current_needlist[\"types\"] \\\n or need_info[\"type_name\"] in current_needlist[\"types\"] \\\n or len(current_needlist[\"types\"]) == 0:\n type_filter_passed = True\n\n if status_filter_passed and tags_filter_passed and type_filter_passed:\n found_needs_by_options.append(need_info)\n\n found_needs_by_string = filter_needs(all_needs_incl_parts, current_needlist[\"filter\"])\n\n # found_needs = [x for x in found_needs_by_string if x in found_needs_by_options]\n\n found_needs = check_need_list(found_needs_by_options, found_needs_by_string)\n\n return found_needs"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef filter_needs(needs, filter_string=\"\", filter_parts=True, merge_part_with_parent=True):\n\n if filter_string is None or filter_string == \"\":\n return needs\n\n found_needs = []\n\n for filter_need in needs:\n try:\n if filter_single_need(filter_need, filter_string):\n found_needs.append(filter_need)\n except Exception as e:\n logger.warning(\"Filter {0} not valid: Error: {1}\".format(filter_string, e))\n\n return found_needs", "response": "Filters given needs based on a given filter string."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchecks if a single need passes a filter_string", "response": "def filter_single_need(need, filter_string=\"\"):\n \"\"\"\n Checks if a single need/need_part passes a filter_string\n\n :param need: need or need_part\n :param filter_string: string, which is used as input for eval()\n :return: True, if need as passed the filter_string, else False\n \"\"\"\n filter_context = need.copy()\n filter_context[\"search\"] = re.search\n result = False\n try:\n result = bool(eval(filter_string, None, filter_context))\n except Exception as e:\n raise NeedInvalidFilter(\"Filter {0} not valid: Error: {1}\".format(filter_string, e))\n return result"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_logging_level(debug):\n level = logging.INFO\n if debug:\n level = logging.DEBUG\n return level", "response": "Returns logging level based on boolean"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nruns a spec file and return a list of Crate node entries.", "response": "def run_spec(spec,\n benchmark_hosts,\n result_hosts=None,\n output_fmt=None,\n logfile_info=None,\n logfile_result=None,\n action=None,\n fail_if=None,\n sample_mode='reservoir'):\n \"\"\"Run a spec file, executing the statements on the benchmark_hosts.\n\n Short example of a spec file:\n\n [setup]\n statement_files = [\"sql/create_table.sql\"]\n\n [[setup.data_files]]\n target = \"t\"\n source = \"data/t.json\"\n\n [[queries]]\n statement = \"select count(*) from t\"\n iterations = 2000\n concurrency = 10\n\n [teardown]\n statements = [\"drop table t\"]\n\n See https://github.com/mfussenegger/cr8/tree/master/specs\n for more examples.\n\n Args:\n spec: path to a spec file\n benchmark_hosts: hostname[:port] pairs of Crate nodes\n result_hosts: optional hostname[:port] Crate node pairs into which the\n runtime statistics should be inserted.\n output_fmt: output format\n action: Optional action to execute.\n Default is to execute all actions - setup, queries and teardown.\n If present only the specified action will be executed.\n The argument can be provided multiple times to execute more than\n one action.\n fail-if: An expression that causes cr8 to exit with a failure if it\n evaluates to true.\n The expression can contain formatting expressions for:\n - runtime_stats\n - statement\n - meta\n - concurrency\n - bulk_size\n For example:\n --fail-if \"{runtime_stats.mean} > 1.34\"\n \"\"\"\n with Logger(output_fmt=output_fmt,\n logfile_info=logfile_info,\n logfile_result=logfile_result) as log:\n do_run_spec(\n spec=spec,\n benchmark_hosts=benchmark_hosts,\n log=log,\n result_hosts=result_hosts,\n action=action,\n fail_if=fail_if,\n sample_mode=sample_mode\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef download_from_url(src_url, dst_dir):\n dst_path = \"\"\n if \"MD982181\" not in src_url:\n dsn = input(\"Please enter the dataset name for this file (Name.Location.Year) : \")\n dst_path = os.path.join(dst_dir, dsn + \".lpd\")\n try:\n print(\"downloading file from url...\")\n urllib.request.urlretrieve(src_url, dst_path)\n except Exception as e:\n print(\"Error: unable to download from url: {}\".format(e))\n else:\n print(\"Error: That file cannot be download due to server restrictions\")\n return dst_path", "response": "Download a file from a given URL and save it to a local path."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nprocessing a single LPD file.", "response": "def process_lpd(name, dir_tmp):\n \"\"\"\n Opens up json file, invokes doi_resolver, closes file, updates changelog, cleans directory, and makes new bag.\n :param str name: Name of current .lpd file\n :param str dir_tmp: Path to tmp directory\n :return none:\n \"\"\"\n logger_doi_main.info(\"enter process_lpd\")\n dir_root = os.getcwd()\n dir_bag = os.path.join(dir_tmp, name)\n dir_data = os.path.join(dir_bag, 'data')\n\n # Navigate down to jLD file\n # dir : dir_root -> dir_data\n os.chdir(dir_data)\n\n # Open jld file and read in the contents. Execute DOI Resolver.\n jld_data = read_json_from_file(os.path.join(dir_data, name + '.jsonld'))\n\n # Overwrite data with new data\n jld_data = DOIResolver(dir_root, name, jld_data).main()\n # Open the jld file and overwrite the contents with the new data.\n write_json_to_file(jld_data)\n\n # Open changelog. timestamp it. Prompt user for short description of changes. Close and save\n # update_changelog()\n\n # Delete old bag files, and move files to bag root for re-bagging\n # dir : dir_data -> dir_bag\n dir_cleanup(dir_bag, dir_data)\n finish_bag(dir_bag)\n\n logger_doi_main.info(\"exit process_lpd\")\n return"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nasks the user if they want to force update files that were previously resolved.", "response": "def prompt_force():\n \"\"\"\n Ask the user if they want to force update files that were previously resolved\n :return bool: response\n \"\"\"\n logger_doi_main.info(\"enter prompt_force\")\n count = 0\n print(\"Do you want to force updates for previously resolved files? (y/n)\")\n while True:\n force = input(\"> \")\n try:\n if count == 2:\n return True\n elif force.lower() in ('y', 'yes'):\n return True\n elif force.lower() in ('n', 'no'):\n return False\n else:\n print(\"invalid response\")\n except AttributeError as e:\n print(\"invalid response\")\n logger_doi_main.warn(\"invalid response: {}, {}\".format(force, e))\n count += 1\n logger_doi_main.info(\"force update: {}\".format(force))\n logger_doi_main.info(\"exit prompt_force\")\n return True"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef process_needtables(app, doctree, fromdocname):\n env = app.builder.env\n\n for node in doctree.traverse(Needtable):\n if not app.config.needs_include_needs:\n # Ok, this is really dirty.\n # If we replace a node, docutils checks, if it will not lose any attributes.\n # But this is here the case, because we are using the attribute \"ids\" of a node.\n # However, I do not understand, why losing an attribute is such a big deal, so we delete everything\n # before docutils claims about it.\n for att in ('ids', 'names', 'classes', 'dupnames'):\n node[att] = []\n node.replace_self([])\n continue\n\n id = node.attributes[\"ids\"][0]\n current_needtable = env.need_all_needtables[id]\n all_needs = env.needs_all_needs\n\n if current_needtable[\"style\"] == \"\" or current_needtable[\"style\"].upper() not in [\"TABLE\", \"DATATABLES\"]:\n if app.config.needs_table_style == \"\":\n style = \"DATATABLES\"\n else:\n style = app.config.needs_table_style.upper()\n else:\n style = current_needtable[\"style\"].upper()\n\n # Prepare table\n classes = [\"NEEDS_{style}\".format(style=style)]\n content = nodes.table(classes=classes)\n tgroup = nodes.tgroup()\n\n # Define Table column width\n # ToDo: Find a way to chosen to perfect width automatically.\n for col in current_needtable[\"columns\"]:\n if col == \"TITLE\":\n tgroup += nodes.colspec(colwidth=15)\n else:\n tgroup += nodes.colspec(colwidth=5)\n\n node_columns = []\n for col in current_needtable[\"columns\"]:\n header_name = col.title() if col != \"ID\" else col\n header_name = header_name.replace(\"_\", \" \")\n node_columns.append(nodes.entry('', nodes.paragraph('', header_name)))\n\n tgroup += nodes.thead('', nodes.row(\n '', *node_columns))\n tbody = nodes.tbody()\n tgroup += tbody\n content += tgroup\n\n all_needs = list(all_needs.values())\n\n if current_needtable[\"sort_by\"] is not None:\n if current_needtable[\"sort_by\"] == \"id\":\n all_needs = sorted(all_needs, key=lambda node: node[\"id\"])\n elif current_needtable[\"sort_by\"] == \"status\":\n all_needs = sorted(all_needs, key=status_sorter)\n\n # Perform filtering of needs\n found_needs = procces_filters(all_needs, current_needtable)\n\n for need_info in found_needs:\n temp_need = need_info.copy()\n if temp_need['is_need']:\n row = nodes.row(classes=['need'])\n prefix = ''\n else:\n row = nodes.row(classes=['need_part'])\n temp_need['id'] = temp_need['id_complete']\n prefix = app.config.needs_part_prefix\n temp_need['title'] = temp_need['content']\n\n for col in current_needtable[\"columns\"]:\n if col == \"ID\":\n row += row_col_maker(app, fromdocname, env.needs_all_needs, temp_need, \"id\", make_ref=True,\n prefix=prefix)\n elif col == \"TITLE\":\n row += row_col_maker(\n app, fromdocname, env.needs_all_needs, temp_need, \"title\",\n prefix=app.config.needs_part_prefix)\n elif col == \"INCOMING\":\n row += row_col_maker(app, fromdocname, env.needs_all_needs, temp_need,\n \"links_back\", ref_lookup=True)\n elif col == \"OUTGOING\":\n row += row_col_maker(app, fromdocname, env.needs_all_needs, temp_need, \"links\", ref_lookup=True)\n else:\n row += row_col_maker(app, fromdocname, env.needs_all_needs, temp_need, col.lower())\n tbody += row\n\n # Need part rows\n if current_needtable[\"show_parts\"] and need_info['is_need']:\n for key, part in need_info[\"parts\"].items():\n row = nodes.row(classes=['need_part'])\n temp_part = part.copy() # The dict needs to be manipulated, so that row_col_maker() can be used\n temp_part['docname'] = need_info['docname']\n\n for col in current_needtable[\"columns\"]:\n if col == \"ID\":\n temp_part['id'] = '.'.join([need_info['id'], part['id']])\n row += row_col_maker(\n app, fromdocname, env.needs_all_needs, temp_part, \"id\",\n make_ref=True, prefix=app.config.needs_part_prefix)\n elif col == \"TITLE\":\n row += row_col_maker(\n app, fromdocname, env.needs_all_needs, temp_part, \"content\",\n prefix=app.config.needs_part_prefix)\n elif col == \"INCOMING\":\n row += row_col_maker(\n app, fromdocname, env.needs_all_needs, temp_part, \"links_back\", ref_lookup=True)\n else:\n row += row_col_maker(\n app, fromdocname, env.needs_all_needs, temp_part, col.lower())\n\n tbody += row\n\n if len(found_needs) == 0:\n content.append(no_needs_found_paragraph())\n\n # add filter information to output\n if current_needtable[\"show_filters\"]:\n content.append(used_filter_paragraph(current_needtable))\n\n node.replace_self(content)", "response": "Process all needtables nodes and add them to the target node."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_sections(need_info):\n sections = []\n current_node = need_info['target_node']\n while current_node:\n if isinstance(current_node, nodes.section):\n title = current_node.children[0].astext()\n # If using auto-section numbering, then Sphinx inserts\n # multiple non-breaking space unicode characters into the title\n # we'll replace those with a simple space to make them easier to\n # use in filters\n title = NON_BREAKING_SPACE.sub(' ', title)\n sections.append(title)\n current_node = getattr(current_node, 'parent', None)\n return sections", "response": "Gets the hierarchy of the section nodes of the current need and then its parent sections"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef purge_needs(app, env, docname):\n if not hasattr(env, 'needs_all_needs'):\n return\n env.needs_all_needs = {key: need for key, need in env.needs_all_needs.items() if need['docname'] != docname}", "response": "Delete all needs for a given docname."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nadds section titles to the needs as additional attributes that can be used in tables and filters", "response": "def add_sections(app, doctree, fromdocname):\n \"\"\"Add section titles to the needs as additional attributes that can\n be used in tables and filters\"\"\"\n needs = getattr(app.builder.env, 'needs_all_needs', {})\n for key, need_info in needs.items():\n sections = get_sections(need_info)\n need_info['sections'] = sections\n need_info['section_name'] = sections[0] if sections else \"\""} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef process_need_nodes(app, doctree, fromdocname):\n if not app.config.needs_include_needs:\n for node in doctree.traverse(Need):\n node.parent.remove(node)\n return\n\n env = app.builder.env\n\n # If no needs were defined, we do not need to do anything\n if not hasattr(env, \"needs_all_needs\"):\n return\n\n needs = env.needs_all_needs\n\n # Call dynamic functions and replace related note data with their return values\n resolve_dynamic_values(env)\n\n create_back_links(env)\n\n for node_need in doctree.traverse(Need):\n need_id = node_need.attributes[\"ids\"][0]\n need_data = needs[need_id]\n\n find_and_replace_node_content(node_need, env, need_data)\n\n node_headline = construct_headline(need_data, app)\n node_meta = construct_meta(need_data, env)\n\n # Collapse check\n if need_data[\"collapse\"] and \"HTML\" in app.builder.name.upper():\n # HEADER\n node_need_toogle_container = nodes.container(classes=['toggle'])\n node_need_toogle_head_container = nodes.container(classes=['header'])\n node_need_toogle_head_container += node_headline.children\n\n node_need_toogle_container.append(node_need_toogle_head_container)\n\n # Only add node_meta(line_block), if it has lines in it\n # Otherwise the pdf/latex build will claim about an empty line_block.\n if node_meta.children:\n node_need_toogle_container.append(node_meta)\n\n node_need.insert(0, node_need_toogle_container)\n else:\n node_meta.insert(0, node_headline)\n node_need.insert(0, node_meta)", "response": "Process the need nodes."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates back - links in all found needs.", "response": "def create_back_links(env):\n \"\"\"\n Create back-links in all found needs.\n But do this only once, as all needs are already collected and this sorting is for all\n needs and not only for the ones of the current document.\n\n :param env: sphinx enviroment\n :return: None\n \"\"\"\n if env.needs_workflow['backlink_creation']:\n return\n\n needs = env.needs_all_needs\n for key, need in needs.items():\n for link in need[\"links\"]:\n link_main = link.split('.')[0]\n try:\n link_part = link.split('.')[1]\n except IndexError:\n link_part = None\n\n if link_main in needs:\n if key not in needs[link_main][\"links_back\"]:\n needs[link_main][\"links_back\"].append(key)\n\n # Handling of links to need_parts inside a need\n if link_part is not None:\n if link_part in needs[link_main]['parts']:\n if 'links_back' not in needs[link_main]['parts'][link_part].keys():\n needs[link_main]['parts'][link_part]['links_back'] = []\n needs[link_main]['parts'][link_part]['links_back'].append(key)\n\n env.needs_workflow['backlink_creation'] = True"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef construct_headline(need_data, app):\n # need title calculation\n title_type = '{}: '.format(need_data[\"type_name\"])\n title_headline = need_data[\"title\"]\n title_id = \"{}\".format(need_data[\"id\"])\n title_spacer = \" \"\n\n # need title\n node_type = nodes.inline(title_type, title_type, classes=[\"needs-type\"])\n node_title = nodes.inline(title_headline, title_headline, classes=[\"needs-title\"])\n\n nodes_id = nodes.inline(classes=[\"needs-id\"])\n\n nodes_id_text = nodes.Text(title_id, title_id)\n id_ref = make_refnode(app.builder,\n fromdocname=need_data['docname'],\n todocname=need_data['docname'],\n targetid=need_data['id'],\n child=nodes_id_text.deepcopy(),\n title=title_id)\n nodes_id += id_ref\n\n node_spacer = nodes.inline(title_spacer, title_spacer, classes=[\"needs-spacer\"])\n\n headline_line = nodes.line(classes=[\"headline\"])\n headline_line.append(node_type)\n headline_line.append(node_spacer)\n headline_line.append(node_title)\n headline_line.append(node_spacer)\n headline_line.append(nodes_id)\n\n return headline_line", "response": "Constructs the node - structure for the headline container."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef construct_meta(need_data, env):\n\n hide_options = env.config.needs_hide_options\n if not isinstance(hide_options, list):\n raise SphinxError('Config parameter needs_hide_options must be of type list')\n\n node_meta = nodes.line_block(classes=['needs_meta'])\n # need parameters\n param_status = \"status: \"\n param_tags = \"tags: \"\n\n if need_data[\"status\"] is not None and 'status' not in hide_options:\n status_line = nodes.line(classes=['status'])\n # node_status = nodes.line(param_status, param_status, classes=['status'])\n node_status = nodes.inline(param_status, param_status, classes=['status'])\n status_line.append(node_status)\n status_line.append(nodes.inline(need_data[\"status\"], need_data[\"status\"],\n classes=[\"needs-status\", str(need_data['status'])]))\n node_meta.append(status_line)\n\n if need_data[\"tags\"] and 'tags' not in hide_options:\n tag_line = nodes.line(classes=['tags'])\n # node_tags = nodes.line(param_tags, param_tags, classes=['tags'])\n node_tags = nodes.inline(param_tags, param_tags, classes=['tags'])\n tag_line.append(node_tags)\n for tag in need_data['tags']:\n # node_tags.append(nodes.inline(tag, tag, classes=[\"needs-tag\", str(tag)]))\n # node_tags.append(nodes.inline(' ', ' '))\n tag_line.append(nodes.inline(tag, tag, classes=[\"needs-tag\", str(tag)]))\n tag_line.append(nodes.inline(' ', ' '))\n node_meta.append(tag_line)\n\n # Links incoming\n if need_data['links_back'] and 'links_back' not in hide_options:\n node_incoming_line = nodes.line(classes=['links', 'incoming'])\n prefix = \"links incoming: \"\n node_incoming_prefix = nodes.inline(prefix, prefix)\n node_incoming_line.append(node_incoming_prefix)\n node_incoming_links = Need_incoming(reftarget=need_data['id'])\n node_incoming_links.append(nodes.inline(need_data['id'], need_data['id']))\n node_incoming_line.append(node_incoming_links)\n node_meta.append(node_incoming_line)\n\n # # Links outgoing\n if need_data['links'] and 'links' not in hide_options:\n node_outgoing_line = nodes.line(classes=['links', 'outgoing'])\n prefix = \"links outgoing: \"\n node_outgoing_prefix = nodes.inline(prefix, prefix)\n node_outgoing_line.append(node_outgoing_prefix)\n node_outgoing_links = Need_outgoing(reftarget=need_data['id'])\n node_outgoing_links.append(nodes.inline(need_data['id'], need_data['id']))\n node_outgoing_line.append(node_outgoing_links)\n node_meta.append(node_outgoing_line)\n\n extra_options = getattr(env.config, 'needs_extra_options', {})\n node_extra_options = []\n for key, value in extra_options.items():\n if key in hide_options:\n continue\n param_data = need_data[key]\n if param_data is None or not param_data:\n continue\n param_option = '{}: '.format(key)\n option_line = nodes.line(classes=['extra_option'])\n option_line.append(nodes.inline(param_option, param_option, classes=['extra_option']))\n option_line.append(nodes.inline(param_data, param_data,\n classes=[\"needs-extra-option\", str(key)]))\n node_extra_options.append(option_line)\n\n node_meta += node_extra_options\n\n global_options = getattr(env.config, 'needs_global_options', {})\n node_global_options = []\n for key, value in global_options.items():\n # If a global option got locally overwritten, it must already part of extra_options.\n # In this skipp output, as this is done during extra_option handling\n if key in extra_options or key in hide_options:\n continue\n\n param_data = need_data[key]\n if param_data is None or not param_data:\n continue\n param_option = '{}: '.format(key)\n global_option_line = nodes.line(classes=['global_option'])\n global_option_line.append(nodes.inline(param_option, param_option, classes=['global_option']))\n global_option_line.append(nodes.inline(param_data, param_data,\n classes=[\"needs-global-option\", str(key)]))\n node_global_options.append(global_option_line)\n\n node_meta += node_global_options\n\n return node_meta", "response": "Constructs the node - structure for the status container"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _fix_list_dyn_func(list):\n open_func_string = False\n new_list = []\n for element in list:\n if '[[' in element:\n open_func_string = True\n new_link = [element]\n elif ']]' in element:\n new_link.append(element)\n open_func_string = False\n element = \",\".join(new_link)\n new_list.append(element)\n elif open_func_string:\n new_link.append(element)\n else:\n new_list.append(element)\n return new_list", "response": "This function fixes the list of dynamic function fragments in a list."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef merge_extra_options(self, needs_info):\n extra_keys = set(self.options.keys()).difference(set(needs_info.keys()))\n for key in extra_keys:\n needs_info[key] = self.options[key]\n\n # Finally add all not used extra options with empty value to need_info.\n # Needed for filters, which need to access these empty/not used options.\n for key in self.option_spec:\n if key not in needs_info.keys():\n needs_info[key] = \"\"\n\n return extra_keys", "response": "Add any extra options introduced via options_ext to needs_info"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nadd all global options to needs_info", "response": "def merge_global_options(self, needs_info):\n \"\"\"Add all global defined options to needs_info\"\"\"\n global_options = getattr(self.env.app.config, 'needs_global_options', None)\n if global_options is None:\n return\n for key, value in global_options.items():\n\n # If key already exists in needs_info, this global_option got overwritten manually in current need\n if key in needs_info.keys():\n continue\n\n needs_info[key] = value"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_full_title(self):\n if len(self.arguments) > 0: # a title was passed\n if 'title_from_content' in self.options:\n self.log.warning(\n 'Needs: need \"{}\" has :title_from_content: set, '\n 'but a title was provided. (see file {})'\n .format(self.arguments[0], self.docname)\n )\n return self.arguments[0]\n elif self.title_from_content:\n first_sentence = ' '.join(self.content).split('.', 1)[0]\n if not first_sentence:\n raise NeedsInvalidException(':title_from_content: set, but '\n 'no content provided. '\n '(Line {} of file {}'\n .format(self.lineno, self.docname))\n return first_sentence\n else:\n return ''", "response": "Determines the full title for the need in order of precedence."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nprocessing all needlists in the doctree.", "response": "def process_needlist(app, doctree, fromdocname):\n \"\"\"\n Replace all needlist nodes with a list of the collected needs.\n Augment each need with a backlink to the original location.\n \"\"\"\n env = app.builder.env\n\n for node in doctree.traverse(Needlist):\n if not app.config.needs_include_needs:\n # Ok, this is really dirty.\n # If we replace a node, docutils checks, if it will not lose any attributes.\n # But this is here the case, because we are using the attribute \"ids\" of a node.\n # However, I do not understand, why losing an attribute is such a big deal, so we delete everything\n # before docutils claims about it.\n for att in ('ids', 'names', 'classes', 'dupnames'):\n node[att] = []\n node.replace_self([])\n continue\n\n id = node.attributes[\"ids\"][0]\n current_needfilter = env.need_all_needlists[id]\n all_needs = env.needs_all_needs\n content = []\n all_needs = list(all_needs.values())\n\n if current_needfilter[\"sort_by\"] is not None:\n if current_needfilter[\"sort_by\"] == \"id\":\n all_needs = sorted(all_needs, key=lambda node: node[\"id\"])\n elif current_needfilter[\"sort_by\"] == \"status\":\n all_needs = sorted(all_needs, key=status_sorter)\n\n found_needs = procces_filters(all_needs, current_needfilter)\n\n line_block = nodes.line_block()\n for need_info in found_needs:\n para = nodes.line()\n description = \"%s: %s\" % (need_info[\"id\"], need_info[\"title\"])\n\n if current_needfilter[\"show_status\"] and need_info[\"status\"] is not None:\n description += \" (%s)\" % need_info[\"status\"]\n\n if current_needfilter[\"show_tags\"] and need_info[\"tags\"] is not None:\n description += \" [%s]\" % \"; \".join(need_info[\"tags\"])\n\n title = nodes.Text(description, description)\n\n # Create a reference\n if not need_info[\"hide\"]:\n ref = nodes.reference('', '')\n ref['refdocname'] = need_info['docname']\n ref['refuri'] = app.builder.get_relative_uri(\n fromdocname, need_info['docname'])\n ref['refuri'] += '#' + need_info['target_node']['refid']\n ref.append(title)\n para += ref\n else:\n para += title\n line_block.append(para)\n content.append(line_block)\n\n if len(content) == 0:\n content.append(no_needs_found_paragraph())\n\n if current_needfilter[\"show_filters\"]:\n content.append(used_filter_paragraph(current_needfilter))\n\n node.replace_self(content)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nparse a string formatted X. Y. Z version number into a tuple of integers.", "response": "def parse_version(version: str) -> tuple:\n \"\"\"Parse a string formatted X[.Y.Z] version number into a tuple\n\n >>> parse_version('10.2.3')\n (10, 2, 3)\n\n >>> parse_version('12')\n (12, 0, 0)\n \"\"\"\n if not version:\n return None\n parts = version.split('.')\n missing = 3 - len(parts)\n return tuple(int(i) for i in parts + ([0] * missing))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse_table(fq_table: str) -> Tuple[str, str]:\n parts = fq_table.split('.')\n if len(parts) == 1:\n return 'doc', parts[0]\n elif len(parts) == 2:\n return parts[0], parts[1]\n else:\n raise ValueError", "response": "Parse a tablename into a tuple of schema and table name."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef as_bulk_queries(queries, bulk_size):\n stmt_dict = defaultdict(list)\n for stmt, args in queries:\n bulk_args = stmt_dict[stmt]\n bulk_args.append(args)\n if len(bulk_args) == bulk_size:\n yield stmt, bulk_args\n del stmt_dict[stmt]\n for stmt, bulk_args in stmt_dict.items():\n yield stmt, bulk_args", "response": "Yields a list of tuples grouped by stmt into a list of args grouped by stmt."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_lines(filename: str) -> Iterator[str]:\n if filename.endswith('.gz'):\n with gzip.open(filename, 'r') as f:\n for line in f:\n yield line.decode('utf-8')\n else:\n with open(filename, 'r', encoding='utf-8') as f:\n for line in f:\n yield line", "response": "Create an iterator that returns the lines of a utf - 8 encoded file."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef as_statements(lines: Iterator[str]) -> Iterator[str]:\n lines = (l.strip() for l in lines if l)\n lines = (l for l in lines if l and not l.startswith('--'))\n parts = []\n for line in lines:\n parts.append(line.rstrip(';'))\n if line.endswith(';'):\n yield ' '.join(parts)\n parts.clear()\n if parts:\n yield ' '.join(parts)", "response": "Create an iterator that transforms lines into sql statements."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef break_iterable(iterable, pred):\n sublist = []\n for i in iterable:\n if pred(i):\n yield sublist\n sublist = []\n else:\n sublist.append(i)\n yield sublist", "response": "Break an iterable on the item that matches the predicate into lists."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a Bag out of a directory.", "response": "def create_bag(dir_bag):\n \"\"\"\n Create a Bag out of given files.\n :param str dir_bag: Directory that contains csv, jsonld, and changelog files.\n :return obj: Bag\n \"\"\"\n logger_bagit.info(\"enter create_bag\")\n # if not dir_bag:\n # dir_bag = os.getcwd()\n try:\n bag = bagit.make_bag(dir_bag, {'Name': 'LiPD Project', 'Reference': 'www.lipds.net', 'DOI-Resolved': 'True'})\n logger_bagit.info(\"created bag\")\n return bag\n except FileNotFoundError as e:\n print(\"Error: directory not found to create bagit\")\n logger_bagit.debug(\"create_bag: FileNotFoundError: failed to create bagit, {}\".format(e))\n except Exception as e:\n print(\"Error: failed to create bagit bag\")\n logger_bagit.debug(\"create_bag: Exception: failed to create bag, {}\".format(e))\n return None"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nopens a Bag at the given path", "response": "def open_bag(dir_bag):\n \"\"\"\n Open Bag at the given path\n :param str dir_bag: Path to Bag\n :return obj: Bag\n \"\"\"\n logger_bagit.info(\"enter open_bag\")\n try:\n bag = bagit.Bag(dir_bag)\n logger_bagit.info(\"opened bag\")\n return bag\n except Exception as e:\n print(\"Error: failed to open bagit bag\")\n logger_bagit.debug(\"failed to open bag, {}\".format(e))\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef validate_md5(bag):\n logger_bagit.info(\"validate_md5\")\n if bag.is_valid():\n print(\"Valid md5\")\n # for path, fixity in bag.entries.items():\n # print(\"path:{}\\nmd5:{}\\n\".format(path, fixity[\"md5\"]))\n else:\n print(\"Invalid md5\")\n logger_bagit.debug(\"invalid bag\")\n return", "response": "Check if Bag is valid and print out entries and md5"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nchecks if DOI - Resolved flag in bag. info to see if doi_resolver has been previously run", "response": "def resolved_flag(bag):\n \"\"\"\n Check DOI flag in bag.info to see if doi_resolver has been previously run\n :param obj bag: Bag\n :return bool: Flag\n \"\"\"\n if 'DOI-Resolved' in bag.info:\n logger_bagit.info(\"bagit resolved_flag: true\")\n return True\n logger_bagit.info(\"bagit resolved_flag: false\")\n return False"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef finish_bag(dir_bag):\n logger_bagit.info(\"enter finish_bag\")\n # Create a bag for the 3 files\n new_bag = create_bag(dir_bag)\n open_bag(dir_bag)\n new_bag.save(manifests=True)\n logger_bagit.info(\"exit finish_bag\")\n return", "response": "Closes steps for creating a bag and saves it"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _ask_how_many():\n batch = True\n invalid = True\n _option = \"\"\n\n try:\n while invalid:\n print(\"\\nChoose a loading option:\\n1. Select specific file(s)\\n2. Load entire folder\")\n _option = input(\"Option: \")\n # these are the only 2 valid options. if this is true, then stop prompting\n if _option in [\"1\", \"2\"]:\n invalid = False\n # indicate whether to leave batch on or turn off\n if _option == \"1\":\n batch = False\n except Exception:\n logger_directory.info(\"_askHowMany: Couldn't get a valid input from the user.\")\n\n return batch", "response": "Ask user if they want to load in one file or do a batch process of a whole directory."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef browse_dialog_dir():\n _go_to_package()\n logger_directory.info(\"enter browse_dialog\")\n _path_bytes = subprocess.check_output(['python', 'gui_dir_browse.py'], shell=False)\n _path = _fix_path_bytes(_path_bytes, file=False)\n if len(_path) >= 1:\n _path = _path[0]\n else:\n _path = \"\"\n logger_directory.info(\"chosen path: {}\".format(_path))\n logger_directory.info(\"exit browse_dialog\")\n return _path", "response": "Open a GUI browse dialog window and let to user pick a target directory."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef browse_dialog_file():\n logger_directory.info(\"enter browse_dialog\")\n\n # We make files a list, because the user can multi-select files.\n _files = []\n _path = \"\"\n try:\n _go_to_package()\n _path_bytes = subprocess.check_output(['python', 'gui_file_browse.py'])\n _path = _fix_path_bytes(_path_bytes)\n _files = [i for i in _path]\n _path = os.path.dirname(_path[0])\n logger_directory.info(\"chosen path: {}, chosen file: {}\".format(_path, _files))\n\n except IndexError:\n logger_directory.warn(\"directory: browse_dialog_file: IndexError: no file chosen\")\n except Exception as e:\n logger_directory.error(\"directory: browse_dialog_file: UnknownError: {}\".format(e))\n\n logger_directory.info(\"exit browse_dialog_file\")\n\n return _path, _files", "response": "Open a GUI browse dialog window and let to select one or more files"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef check_file_age(filename, days):\n logger_directory.info(\"enter check_file_age\")\n # Multiply days given by time for one day.\n t = days * 60 * 60 * 24\n now = time.time()\n specified_time = now - t\n try:\n if os.path.getctime(filename) < specified_time:\n # File found and out of date\n logger_directory.info(\"{} not up-to-date\".format(filename))\n logger_directory.info(\"exiting check_file_age()\")\n return True\n # File found, and not out of date\n logger_directory.info(\"{} and up-to-date\".format(filename))\n logger_directory.info(\"exiting check_file_age()\")\n return False\n except FileNotFoundError:\n # File not found. Need to download it.\n logger_directory.info(\"{} not found in {}\".format(filename, os.getcwd()))\n logger_directory.info(\"exiting check_file_age()\")\n return True", "response": "Check if the target file has an older creation date than X amount of time."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef collect_metadata_files(cwd, new_files, existing_files):\n obj = {}\n try:\n os.chdir(cwd)\n\n # Special case: User uses gui to mult-select 2+ files. You'll be given a list of file paths.\n if new_files:\n for full_path in new_files:\n # Create the file metadata for one file, and append it to the existing files.\n obj = collect_metadata_file(full_path)\n\n # directory: get all files in the directory and sort by type\n else:\n for file_type in [\".lpd\", \".xls\", \".txt\"]:\n # get all files in cwd of this file extension\n files_found = list_files(file_type)\n # if looking for excel files, also look for the alternate extension.\n if file_type == \".xls\":\n files_found += list_files(\".xlsx\")\n # for each file found, build it's metadata and append it to files_by_type\n for file in files_found:\n fn = os.path.splitext(file)[0]\n existing_files[file_type].append({\"full_path\": os.path.join(cwd, file), \"filename_ext\": file, \"filename_no_ext\": fn, \"dir\": cwd})\n except Exception:\n logger_directory.info(\"directory: collect_files: there's a problem\")\n\n return obj", "response": "Collect all files from a given path and return one list for each type\n "} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates the file metadata and add it to the appropriate section by file - type", "response": "def collect_metadata_file(full_path):\n \"\"\"\n Create the file metadata and add it to the appropriate section by file-type\n :param str full_path:\n :param dict existing_files:\n :return dict existing files:\n \"\"\"\n fne = os.path.basename(full_path)\n fn = os.path.splitext(fne)[0]\n obj = {\"full_path\": full_path, \"filename_ext\": fne, \"filename_no_ext\": fn, \"dir\": os.path.dirname(full_path)}\n return obj"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndeletes all the data files in the data directory and the metadata bag.", "response": "def dir_cleanup(dir_bag, dir_data):\n \"\"\"\n Moves JSON and csv files to bag root, then deletes all the metadata bag files. We'll be creating a new bag with\n the data files, so we don't need the other text files and such.\n :param str dir_bag: Path to root of Bag\n :param str dir_data: Path to Bag /data subdirectory\n :return None:\n \"\"\"\n logger_directory.info(\"enter dir_cleanup\")\n # dir : dir_data -> dir_bag\n os.chdir(dir_bag)\n\n # Delete files in dir_bag\n for file in os.listdir(dir_bag):\n if file.endswith('.txt'):\n os.remove(os.path.join(dir_bag, file))\n logger_directory.info(\"deleted files in dir_bag\")\n # Move dir_data files up to dir_bag\n for file in os.listdir(dir_data):\n shutil.move(os.path.join(dir_data, file), dir_bag)\n logger_directory.info(\"moved dir_data contents to dir_bag\")\n # Delete empty dir_data folder\n shutil.rmtree(dir_data)\n logger_directory.info(\"exit dir_cleanup\")\n return"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nextracting the file name from a given file path.", "response": "def filename_from_path(path):\n \"\"\"\n Extract the file name from a given file path.\n :param str path: File path\n :return str: File name with extension\n \"\"\"\n head, tail = ntpath.split(path)\n return head, tail or ntpath.basename(head)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsearches for the directory containing jsonld and csv files. chdir and then quit.", "response": "def find_files():\n \"\"\"\n Search for the directory containing jsonld and csv files. chdir and then quit.\n :return none:\n \"\"\"\n _dir = os.getcwd()\n _files = os.listdir()\n # Look for a jsonld file\n for _file in _files:\n if _file.endswith(\".jsonld\"):\n return os.getcwd()\n # No jsonld file found, try to chdir into \"bag\" (LiPD v1.3)\n if \"bag\" in _files:\n os.chdir(\"bag\")\n _dir = find_files()\n # No \"bag\" dir. Try chdir into whatever dataset name dir we find (< LiPD v1.2)\n else:\n for _file in _files:\n if os.path.isdir(_file):\n os.chdir(_file)\n _dir = find_files()\n return _dir"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_filenames_generated(d, name=\"\", csvs=\"\"):\n filenames = []\n try:\n filenames = d.keys()\n if name:\n filenames = [os.path.join(name, \"data\", filename) for filename in filenames]\n if csvs:\n lst = [i for i in csvs if not i.endswith(\"csv\")]\n filenames = lst + filenames\n except Exception as e:\n logger_directory.debug(\"get_filenames_generated: Exception: {}\".format(e))\n return filenames", "response": "Get the filenames that the LiPD utilities has generated."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting all the files contained in the LiPD archive. Bagit JSON and CSV files are included in the LiPD archive.", "response": "def get_filenames_in_lipd(path, name=\"\"):\n \"\"\"\n List all the files contained in the LiPD archive. Bagit, JSON, and CSV\n :param str path: Directory to be listed\n :param str name: LiPD dataset name, if you want to prefix it to show file hierarchy\n :return list: Filenames found\n \"\"\"\n _filenames = []\n try:\n # in the top level, list all files and skip the \"data\" directory\n _top = [os.path.join(name, f) for f in os.listdir(path) if f != \"data\"]\n # in the data directory, list all files\n _dir_data = [os.path.join(name, \"data\", f) for f in os.listdir(os.path.join(path, \"data\"))]\n # combine the two lists\n _filenames = _top + _dir_data\n except Exception:\n pass\n return _filenames"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the proper prompt for reading or writing a file.", "response": "def get_src_or_dst_prompt(mode):\n \"\"\"\n String together the proper prompt based on the mode\n :param str mode: \"read\" or \"write\"\n :return str prompt: The prompt needed\n \"\"\"\n _words = {\"read\": \"from\", \"write\": \"to\"}\n # print(os.getcwd())\n prompt = \"Where would you like to {} your file(s) {}?\\n\" \\\n \"1. Desktop ({})\\n\" \\\n \"2. Downloads ({})\\n\" \\\n \"3. Current ({})\\n\" \\\n \"4. Browse\".format(mode, _words[mode],\n os.path.expanduser('~/Desktop'),\n os.path.expanduser('~/Downloads'),\n os.getcwd())\n return prompt"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_src_or_dst_path(prompt, count):\n _path = \"\"\n print(prompt)\n option = input(\"Option: \")\n print(\"\\n\")\n if option == '1':\n # Set the path to the system desktop folder.\n logger_directory.info(\"1: desktop\")\n _path = os.path.expanduser('~/Desktop')\n elif option == '2':\n # Set the path to the system downloads folder.\n logger_directory.info(\"2: downloads\")\n _path = os.path.expanduser('~/Downloads')\n elif option == '3':\n # Current directory\n logger_directory.info(\"3: current\")\n _path = os.getcwd()\n elif option == '4':\n # Open up the GUI browse dialog\n logger_directory.info(\"4: browse \")\n _path = browse_dialog_dir()\n\n else:\n # Something went wrong. Prompt again. Give a couple tries before defaulting to downloads folder\n if count == 2:\n logger_directory.warn(\"too many attempts\")\n print(\"Too many failed attempts. Defaulting to current working directory.\")\n _path = os.getcwd()\n else:\n count += 1\n logger_directory.warn(\"failed attempts: {}\".format(count))\n print(\"Invalid option. Try again.\")\n\n return _path, count", "response": "Get the path to the source directory or destination directory."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef list_files(x, path=\"\"):\n logger_directory.info(\"enter list_files\")\n file_list = []\n if path:\n # list files from target directory\n files = os.listdir(path)\n for file in files:\n if file.endswith(x) and not file.startswith((\"$\", \"~\", \".\")):\n # join the path and basename to create full path\n file_list.append(os.path.join(path, file))\n else:\n # list files from current working directory\n files = os.listdir()\n for file in files:\n if file.endswith(x) and not file.startswith((\"$\", \"~\", \".\")):\n # append basename. not full path\n file_list.append(file)\n logger_directory.info(\"exit list_files\")\n return file_list", "response": "Lists files in given path of the X type."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rm_files_in_dir(path):\n for f in os.listdir(path):\n try:\n os.remove(f)\n except PermissionError:\n os.chmod(f, 0o777)\n try:\n os.remove(f)\n except Exception:\n pass\n return", "response": "Removes all files within a directory but does not delete the directory"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nremove a file if it exists in the specified directory.", "response": "def rm_file_if_exists(path, filename):\n \"\"\"\n Remove a file if it exists. Useful for when we want to write a file, but it already exists in that locaiton.\n :param str filename: Filename\n :param str path: Directory\n :return none:\n \"\"\"\n _full_path = os.path.join(path, filename)\n if os.path.exists(_full_path):\n os.remove(_full_path)\n return"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngenerating the languages Python module.", "response": "def regenerate(location='http://www.iana.org/assignments/language-subtag-registry',\n filename=None, default_encoding='utf-8'):\n \"\"\"\n Generate the languages Python module.\n \"\"\"\n paren = re.compile('\\([^)]*\\)')\n\n # Get the language list.\n data = urllib2.urlopen(location)\n if ('content-type' in data.headers and\n 'charset=' in data.headers['content-type']):\n encoding = data.headers['content-type'].split('charset=')[-1]\n else:\n encoding = default_encoding\n content = data.read().decode(encoding)\n languages = []\n info = {}\n p = None\n for line in content.splitlines():\n if line == '%%':\n if 'Type' in info and info['Type'] == 'language':\n languages.append(info)\n info = {}\n elif ':' not in line and p:\n info[p[0]] = paren.sub('', p[2]+line).strip()\n else:\n p = line.partition(':')\n if not p[0] in info: # Keep the first description as it should be the most common\n info[p[0]] = paren.sub('', p[2]).strip()\n\n languages_lines = map(lambda x:'(\"%s\", _(u\"%s\")),'%(x['Subtag'],x['Description']), languages)\n\n # Generate and save the file.\n if not filename:\n filename = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'languages.py')\n # TODO: first make a backup of the file if it exists already.\n f = codecs.open(filename, 'w', 'utf-8')\n f.write(TEMPLATE % {\n 'languages': '\\n '.join(languages_lines),\n })\n f.close()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\npatching a class to give it a generic __repr__ method that works by inspecting the instance dictionary.", "response": "def give_repr(cls): # pragma: no cover\n r\"\"\"Patch a class to give it a generic __repr__ method\n that works by inspecting the instance dictionary.\n\n Parameters\n ----------\n cls: type\n The class to add a generic __repr__ to.\n\n Returns\n -------\n cls: type\n The passed class is returned\n \"\"\"\n def reprer(self):\n attribs = ', '.join([\"%s=%r\" % (k, v) for k, v in self.__dict__.items() if not k.startswith(\"_\")])\n wrap = \"{self.__class__.__name__}({attribs})\".format(self=self, attribs=attribs)\n return wrap\n cls.__repr__ = reprer\n return cls"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef vietes(coefficients):\n r'''\n Given the coefficients of a polynomial of a single variable,\n compute an elementary symmetric polynomial of the roots of the\n input polynomial by Viete's Formula:\n\n .. math::\n \\sum_{1\\le i_150 and memlim):\n imin = imdic['im']\n imshape = imdic['shape']\n affine = imdic['affine']\n fldrin = fims\n fnms = [os.path.basename(f).split('.nii')[0] for f in imdic['files'] if f!=None]\n # number of images/frames\n Nim = imdic['N']\n using_multiple_files = True\n\n # case when input file is a 3D or 4D NIfTI image\n elif isinstance(fims, basestring) and os.path.isfile(fims) and (fims.endswith('nii') or fims.endswith('nii.gz')):\n imdic = imio.getnii(fims, output='all')\n imin = imdic['im']\n if imin.ndim==3:\n imin.shape = (1, imin.shape[0], imin.shape[1], imin.shape[2])\n imdtype = imdic['dtype']\n imshape = imdic['shape'][-3:]\n affine = imdic['affine']\n fldrin = os.path.dirname(fims)\n fnms = imin.shape[0] * [ os.path.basename(fims).split('.nii')[0] ]\n # number of images/frames\n Nim = imin.shape[0]\n\n # case when a list of input files is given\n elif isinstance(fims, list) and all([os.path.isfile(k) for k in fims]):\n imdic = imio.niisort(fims, memlim=memlim)\n if not (imdic['N']>50 and memlim):\n imin = imdic['im']\n imshape = imdic['shape']\n affine = imdic['affine']\n imdtype = imdic['dtype']\n fldrin = os.path.dirname(fims[0])\n fnms = [os.path.basename(f).split('.nii')[0] for f in imdic['files']]\n # number of images/frames\n Nim = imdic['N']\n using_multiple_files = True\n\n # case when an array [#frames, zdim, ydim, xdim]. Can be 3D or 4D\n elif isinstance(fims, (np.ndarray, np.generic)) and (fims.ndim==4 or fims.ndim==3):\n # check image affine\n if affine.shape!=(4,4):\n raise ValueError('Affine should be a 4x4 array.')\n # create a copy to avoid mutation when only one image (3D)\n imin = np.copy(fims)\n if fims.ndim==3:\n imin.shape = (1, imin.shape[0], imin.shape[1], imin.shape[2])\n imshape = imin.shape[-3:]\n fldrin = os.path.join(os.path.expanduser('~'), 'NIMPA_output')\n if fname=='':\n fnms = imin.shape[0] * [ 'NIMPA' ]\n else:\n fnms = imin.shape[0] * [ fname ]\n # number of images/frames\n Nim = imin.shape[0]\n\n else:\n raise TypeError('Wrong data type input.')\n\n \n\n #-------------------------------------------------------\n # store images in this folder\n if outpath=='':\n petudir = os.path.join(fldrin, 'trimmed')\n else:\n petudir = os.path.join(outpath, 'trimmed')\n imio.create_dir( petudir )\n #-------------------------------------------------------\n\n #-------------------------------------------------------\n # scale is preferred to be integer\n try:\n scale = int(scale)\n except ValueError:\n raise ValueError('e> scale has to be an integer.')\n # scale factor as the inverse of scale\n sf = 1/float(scale)\n if verbose:\n print 'i> upsampling scale {}, giving resolution scale factor {} for {} images.'.format(scale, sf, Nim)\n #-------------------------------------------------------\n\n #-------------------------------------------------------\n # scaled input image and get a sum image as the base for trimming\n if scale>1:\n newshape = (scale*imshape[0], scale*imshape[1], scale*imshape[2])\n imsum = np.zeros(newshape, dtype=imdtype)\n if not memlim:\n imscl = np.zeros((Nim,)+newshape, dtype=imdtype)\n for i in range(Nim):\n imscl[i,:,:,:] = ndi.interpolation.zoom(imin[i,:,:,:], (scale, scale, scale), order=int_order )\n imsum += imscl[i,:,:,:]\n else:\n for i in range(Nim):\n if Nim>50 and using_multiple_files:\n imin_temp = imio.getnii(imdic['files'][i])\n imsum += ndi.interpolation.zoom(imin_temp, (scale, scale, scale), order=0 )\n if verbose:\n print 'i> image sum: read', imdic['files'][i]\n else:\n imsum += ndi.interpolation.zoom(imin[i,:,:,:], (scale, scale, scale), order=0 )\n else:\n imscl = imin\n imsum = np.sum(imin, axis=0)\n\n # smooth the sum image for improving trimming (if any)\n #imsum = ndi.filters.gaussian_filter(imsum, imio.fwhm2sig(4.0, voxsize=abs(affine[0,0])), mode='mirror')\n #-------------------------------------------------------\n \n # find the object bounding indexes in x, y and z axes, e.g., ix0-ix1 for the x axis\n qx = np.sum(imsum, axis=(0,1))\n ix0 = np.argmax( qx>(fmax*np.nanmax(qx)) )\n ix1 = ix0+np.argmin( qx[ix0:]>(fmax*np.nanmax(qx)) )\n\n qy = np.sum(imsum, axis=(0,2))\n iy0 = np.argmax( qy>(fmax*np.nanmax(qy)) )\n iy1 = iy0+np.argmin( qy[iy0:]>(fmax*np.nanmax(qy)) )\n\n qz = np.sum(imsum, axis=(1,2))\n iz0 = np.argmax( qz>(fmax*np.nanmax(qz)) )\n\n # find the maximum voxel range for x and y axes\n IX = ix1-ix0+1\n IY = iy1-iy0+1\n tmp = max(IX, IY)\n #> get the range such that it is divisible by \n #> divdim (64 by default) for GPU execution\n IXY = divdim * ((tmp+divdim-1)/divdim)\n div = (IXY-IX)/2\n # x\n ix0 -= div\n ix1 += (IXY-IX)-div\n # y\n div = (IXY-IY)/2\n iy0 -= div\n iy1 += (IXY-IY)-div\n # z\n tmp = (len(qz)-iz0+1)\n IZ = divdim * ((tmp+divdim-1)/divdim)\n iz0 -= IZ-tmp+1\n \n # save the trimming parameters in a dic\n trimpar = {'x':(ix0, ix1), 'y':(iy0, iy1), 'z':(iz0), 'fmax':fmax}\n\n # new dims (z,y,x)\n newdims = (imsum.shape[0]-iz0, iy1-iy0+1, ix1-ix0+1)\n imtrim = np.zeros((Nim,)+newdims, dtype=imdtype)\n imsumt = np.zeros(newdims, dtype=imdtype)\n # in case of needed padding (negative indx resulting above)\n # the absolute values are supposed to work like padding in case the indx are negative\n iz0s, iy0s, ix0s = iz0, iy0, ix0\n iz0t, iy0t, ix0t = 0,0,0\n if iz0<0: \n iz0s=0; \n iz0t = abs(iz0)\n print '-----------------------------------------------------------------'\n print 'w> Correcting for trimming outside the original image (z-axis)'\n print '-----------------------------------------------------------------'\n if iy0<0: \n iy0s=0; \n iy0t = abs(iy0)\n print '-----------------------------------------------------------------'\n print 'w> Correcting for trimming outside the original image (y-axis)'\n print '-----------------------------------------------------------------'\n if ix0<0: \n ix0s=0; \n ix0t = abs(ix0)\n print '-----------------------------------------------------------------'\n print 'w> Correcting for trimming outside the original image (x-axis)'\n print '-----------------------------------------------------------------'\n \n\n #> in case the upper index goes beyond the scaled but untrimmed image\n iy1t = imsumt.shape[1]\n if iy1>=imsum.shape[1]:\n iy1t -= iy1+1\n\n #> the same for x\n ix1t = imsumt.shape[2]\n if ix1>=imsum.shape[2]:\n ix1t -= ix1+1\n\n\n # first trim the sum image\n imsumt[iz0t:, iy0t:iy1t, ix0t:ix1t] = imsum[iz0s:, iy0s:iy1+1, ix0s:ix1+1]\n\n #> new affine matrix for the scaled and trimmed image\n A = np.diag(sf*np.diag(affine))\n #> note half of new voxel offset is used for the new centre of voxels\n A[0,3] = affine[0,3] + A[0,0]*(ix0-0.5) \n A[1,3] = affine[1,3] + (affine[1,1]*(imshape[1]-1) - A[1,1]*(iy1-0.5))\n A[2,3] = affine[2,3] - A[1,1]*0.5\n A[3,3] = 1\n\n\n # output dictionary\n dctout = {'affine': A, 'trimpar':trimpar, 'imsum':imsumt}\n\n # NIfTI image description (to be stored in the header)\n niidescr = 'trimm(x,y,z):' \\\n + str(trimpar['x'])+',' \\\n + str(trimpar['y'])+',' \\\n + str((trimpar['z'],)) \\\n + ';scale='+str(scale) \\\n + ';fmx='+str(fmax)\n \n # store the sum image\n if store_avg:\n fsum = os.path.join(petudir, 'avg_trimmed-upsampled-scale-'+str(scale)+fcomment+'.nii.gz')\n imio.array2nii( imsumt[::-1,::-1,:], A, fsum, descrip=niidescr)\n if verbose: print 'i> saved averaged image to:', fsum\n dctout['fsum'] = fsum\n\n # list of file names for the upsampled and trimmed images\n fpetu = []\n # perform the trimming and save the intermediate images if requested\n for i in range(Nim):\n\n # memory saving option, second time doing interpolation\n if memlim:\n if Nim>50 and using_multiple_files:\n imin_temp = imio.getnii(imdic['files'][i])\n im = ndi.interpolation.zoom(imin_temp, (scale, scale, scale), order=int_order )\n if verbose:\n print 'i> image scaling:', imdic['files'][i]\n else:\n im = ndi.interpolation.zoom(imin[i,:,:,:], (scale, scale, scale), order=int_order )\n else:\n im = imscl[i,:,:,:]\n\n # trim the scaled image\n imtrim[i, iz0t:, iy0t:iy1t, ix0t:ix1t] = im[iz0s:, iy0s:iy1+1, ix0s:ix1+1]\n \n # save the up-sampled and trimmed PET images\n if store_img_intrmd:\n _frm = '_trmfrm'+str(i)\n _fstr = '_trimmed-upsampled-scale-'+str(scale) + _frm*(Nim>1) +fcomment\n fpetu.append( os.path.join(petudir, fnms[i]+_fstr+'_i.nii.gz') )\n imio.array2nii( imtrim[i,::-1,::-1,:], A, fpetu[i], descrip=niidescr)\n if verbose: print 'i> saved upsampled PET image to:', fpetu[i]\n\n if store_img:\n _nfrm = '_nfrm'+str(Nim)\n fim = os.path.join(petudir, 'trimmed-upsampled-scale-'+str(scale))+_nfrm*(Nim>1)+fcomment+'.nii.gz'\n imio.array2nii( np.squeeze(imtrim[:,::-1,::-1,:]), A, fim, descrip=niidescr)\n dctout['fim'] = fim\n\n # file names (with paths) for the intermediate PET images\n dctout['fimi'] = fpetu\n dctout['im'] = np.squeeze(imtrim)\n dctout['N'] = Nim\n dctout['affine'] = A\n \n return dctout", "response": "This function trims and upsamples PET images from the input images in a NIfTI file."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngeneralizes convolution function for the GPU device.", "response": "def psf_general(vx_size=(1,1,1), fwhm=(5, 5, 6), hradius=8, scale=2):\n '''\n Separable kernels for convolution executed on the GPU device\n The outputted kernels are in this order: z, y, x\n '''\n xSig = (scale*fwhm[0]/vx_size[0]) / (2*(2*np.log(2))**.5)\n ySig = (scale*fwhm[1]/vx_size[1]) / (2*(2*np.log(2))**.5)\n zSig = (scale*fwhm[2]/vx_size[2]) / (2*(2*np.log(2))**.5)\n\n # get the separated kernels\n x = np.arange(-hradius,hradius+1)\n xKrnl = np.exp(-(x**2/(2*xSig**2)))\n yKrnl = np.exp(-(x**2/(2*ySig**2)))\n zKrnl = np.exp(-(x**2/(2*zSig**2)))\n\n # normalise kernels\n xKrnl /= np.sum(xKrnl)\n yKrnl /= np.sum(yKrnl)\n zKrnl /= np.sum(zKrnl)\n\n krnl = np.array([zKrnl, yKrnl, xKrnl], dtype=np.float32)\n\n # for checking if the normalisation worked\n # np.prod( np.sum(krnl,axis=1) )\n\n # return all kernels together\n return krnl"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef pvc_iyang(\n petin,\n mridct,\n Cnt,\n pvcroi,\n krnl,\n itr=5,\n tool='niftyreg',\n faff='',\n outpath='',\n fcomment='',\n store_img=False,\n store_rois=False,\n ):\n ''' Perform partial volume (PVC) correction of PET data (petin) using MRI data (mridct).\n The PVC method uses iterative Yang method.\n GPU based convolution is the key routine of the PVC. \n Input:\n -------\n petin: either a dictionary containing image data, file name and affine transform,\n or a string of the path to the NIfTI file of the PET data.\n mridct: a dictionary of MRI data, including the T1w image, which can be given\n in DICOM (field 'T1DCM') or NIfTI (field 'T1nii'). The T1w image data\n is needed for co-registration to PET if affine is not given in the text\n file with its path in faff.\n Cnt: a dictionary of paths for third-party tools:\n * dcm2niix: Cnt['DCM2NIIX']\n * niftyreg, resample: Cnt['RESPATH']\n * niftyreg, rigid-reg: Cnt['REGPATH']\n * verbose mode on/off: Cnt['VERBOSE'] = True/False\n pvcroi: list of regions (also a list) with number label to distinguish\n the parcellations. The numbers correspond to the image values\n of the parcellated T1w image. E.g.:\n pvcroi = [\n [36], # ROI 1 (single parcellation region)\n [35], # ROI 2\n [39, 40, 72, 73, 74], # ROI 3 (region consisting of multiple parcellation regions)\n ...\n ]\n kernel: the point spread function (PSF) specific for the camera and the object. \n It is given as a 3x17 matrix, a 17-element kernel for each dimension (x,y,z).\n It is used in the GPU-based convolution using separable kernels.\n outpath:path to the output of the resulting PVC images\n faff: a text file of the affine transformations needed to get the MRI into PET space.\n If not provided, it will be obtained from the performed rigid transformation.\n For this the MR T1w image is required. If faff and T1w image are not provided,\n it will results in exception/error.\n fcomment:a string used in naming the produced files, helpful for distinguishing them.\n tool: co-registration tool. By default it is NiftyReg, but SPM is also \n possible (needs Matlab engine and more validation)\n itr: number of iterations used by the PVC. 5-10 should be enough (5 default)\n '''\n\n # get all the input image properties\n if isinstance(petin, dict):\n im = imdic['im']\n fpet = imdic['fpet']\n B = imdic['affine']\n elif isinstance(petin, basestring) and os.path.isfile(petin):\n imdct = imio.getnii(petin, output='all')\n im = imdct['im']\n B = imdct['affine']\n fpet = petin\n \n if im.ndim!=3:\n raise IndexError('Only 3D images are expected in this method of partial volume correction.')\n\n # check if brain parcellations exist in NIfTI format\n if not os.path.isfile(mridct['T1lbl']):\n raise NameError('MissingLabels')\n\n #> output dictionary\n outdct = {}\n\n # establish the output folder\n if outpath=='': \n prcl_dir = os.path.dirname(mridct['T1lbl'])\n pvc_dir = os.path.join(os.path.dirname(fpet), 'PVC')\n else:\n prcl_dir = outpath\n pvc_dir = os.path.join(os.path.dirname(fpet), 'PVC')\n\n #> create folders\n imio.create_dir(prcl_dir)\n imio.create_dir(pvc_dir)\n\n #==================================================================\n #if affine transf. (faff) is given then take the T1 and resample it too.\n if isinstance(faff, basestring) and not os.path.isfile(faff):\n # faff is not given; get it by running the affine; get T1w to PET space\n\n ft1w = imio.pick_t1w(mridct)\n\n if tool=='spm':\n regdct = regseg.coreg_spm(\n fpet,\n ft1w,\n fcomment = fcomment,\n outpath=os.path.join(outpath,'PET', 'positioning')\n )\n elif tool=='niftyreg':\n regdct = regseg.affine_niftyreg(\n fpet,\n ft1w,\n outpath=os.path.join(outpath,'PET', 'positioning'),\n fcomment = fcomment,\n executable = Cnt['REGPATH'],\n omp = multiprocessing.cpu_count()/2,\n rigOnly = True,\n affDirect = False,\n maxit=5,\n speed=True,\n pi=50, pv=50,\n smof=0, smor=0,\n rmsk=True,\n fmsk=True,\n rfwhm=15., #millilitres\n rthrsh=0.05,\n ffwhm = 15., #millilitres\n fthrsh=0.05,\n verbose=Cnt['VERBOSE']\n )\n faff = regdct['faff']\n\n # resample the GIF T1/labels to upsampled PET\n # file name of the parcellation (GIF-based) upsampled to PET\n fgt1u = os.path.join(\n prcl_dir,\n os.path.basename(mridct['T1lbl']).split('.')[0]\\\n +'_registered_trimmed'+fcomment+'.nii.gz')\n \n if tool=='niftyreg':\n if os.path.isfile( Cnt['RESPATH'] ):\n cmd = [Cnt['RESPATH'], '-ref', fpet, '-flo', mridct['T1lbl'],\n '-trans', faff, '-res', fgt1u, '-inter', '0']\n if not Cnt['VERBOSE']: cmd.append('-voff')\n call(cmd)\n else:\n raise IOError('e> path to resampling executable is incorrect!')\n \n elif tool=='spm':\n fout = regseg.resample_spm(\n fpet,\n mridct['T1lbl'],\n faff,\n fimout = fgt1u,\n matlab_eng = regdct['matlab_eng'],\n intrp = 0.,\n del_ref_uncmpr = True,\n del_flo_uncmpr = True,\n del_out_uncmpr = True,\n )\n\n #==================================================================\n\n # Get the parcellation labels in the upsampled PET space\n dprcl = imio.getnii(fgt1u, output='all')\n prcl = dprcl['im']\n\n #> path to parcellations in NIfTI format\n prcl_pth = os.path.split(mridct['T1lbl'])\n\n #---------------------------------------------------------------------------\n #> get the parcellation specific for PVC based on the current parcellations\n imgroi = prcl.copy(); imgroi[:] = 0\n \n #> number of segments, without the background\n nSeg = len(pvcroi)\n \n #> create the image of numbered parcellations\n for k in range(nSeg):\n for m in pvcroi[k]:\n imgroi[prcl==m] = k+1\n\n #> save the PCV ROIs to a new NIfTI file\n if store_rois:\n froi = os.path.join(prcl_dir, prcl_pth[1].split('.nii')[0]+'_PVC-ROIs-inPET.nii.gz')\n imio.array2nii(\n imgroi,\n dprcl['affine'],\n froi,\n trnsp = (dprcl['transpose'].index(0),\n dprcl['transpose'].index(1),\n dprcl['transpose'].index(2)),\n flip = dprcl['flip'])\n outdct['froi'] = froi\n #---------------------------------------------------------------------------\n\n\n #---------------------------------------------------------------------------\n # run iterative Yang PVC\n imgpvc, m_a = iyang(im, krnl, imgroi, Cnt, itr=itr)\n #---------------------------------------------------------------------------\n\n outdct['im'] = imgpvc\n outdct['imroi'] = imgroi\n outdct['faff'] = faff\n\n if store_img:\n fpvc = os.path.join( pvc_dir,\n os.path.split(fpet)[1].split('.nii')[0]+'_PVC'+fcomment+'.nii.gz')\n imio.array2nii( imgpvc[::-1,::-1,:], B, fpvc, descrip='pvc=iY')\n outdct['fpet'] = fpvc\n\n return outdct", "response": "This function is used to perform partial volume correction of the PET data using iterative Yang method."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting CT image to mu - values", "response": "def ct2mu(im):\n '''HU units to 511keV PET mu-values\n https://link.springer.com/content/pdf/10.1007%2Fs00259-002-0796-3.pdf\n C. Burger, et al., PET attenuation coefficients from CT images, \n '''\n\n # convert nans to -1024 for the HU values only\n im[np.isnan(im)] = -1024\n # constants\n muwater = 0.096\n mubone = 0.172\n rhowater = 0.184\n rhobone = 0.428\n uim = np.zeros(im.shape, dtype=np.float32)\n uim[im<=0] = muwater * ( 1+im[im<=0]*1e-3 )\n uim[im> 0] = muwater+im[im>0]*(rhowater*(mubone-muwater)/(1e3*(rhobone-rhowater)))\n # remove negative values\n uim[uim<0] = 0\n return uim"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef centre_mass_img(imdct):\n ''' Calculate the centre of mass of an image along each axes (x,y,z),\n separately.\n Arguments:\n imdct - the image dictionary with the image and header data.\n Output the list of the centre of mass for each axis.\n '''\n \n #> initialise centre of mass array\n com = np.zeros(3, dtype=np.float32)\n #> total image sum\n imsum = np.sum(imdct['im'])\n\n for ind_ax in [-1, -2, -3]:\n #> list of axes\n axs = range(imdct['im'].ndim)\n del axs[ind_ax]\n #> indexed centre of mass\n icom = np.sum( np.sum(imdct['im'], axis=tuple(axs)) \\\n * np.arange(imdct['im'].shape[ind_ax]))/imsum\n #> centre of mass in mm\n com[abs(ind_ax)-1] = icom * imdct['hdr']['pixdim'][abs(ind_ax)]\n\n return com", "response": "Calculates the centre of mass of an image along each axes."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nmodifying the NIfTI image given either as a file path or a dictionary of NIfTI images obtained by nimpa. getnii.", "response": "def nii_modify(\n nii,\n fimout = '',\n outpath = '',\n fcomment = '',\n voxel_range=[]):\n\n ''' Modify the NIfTI image given either as a file path or a dictionary,\n obtained by nimpa.getnii(file_path).\n '''\n\n if isinstance(nii, basestring) and os.path.isfile(nii):\n dctnii = imio.getnii(nii, output='all')\n fnii = nii\n if isinstance(nii, dict) and 'im' in nii:\n dctnii = nii\n if 'fim' in dctnii:\n fnii = dctnii['fim']\n\n #---------------------------------------------------------------------------\n #> output path\n if outpath=='' and fimout!='' and '/' in fimout:\n opth = os.path.dirname(fimout)\n if opth=='' and isinstance(fnii, basestring) and os.path.isfile(fnii):\n opth = os.path.dirname(nii)\n fimout = os.path.basename(fimout)\n\n elif outpath=='' and isinstance(fnii, basestring) and os.path.isfile(fnii):\n opth = os.path.dirname(fnii)\n else:\n opth = outpath\n imio.create_dir(opth)\n\n #> output floating and affine file names\n if fimout=='':\n\n if fcomment=='':\n fcomment += '_nimpa-modified'\n\n fout = os.path.join(\n opth,\n os.path.basename(fnii).split('.nii')[0]+fcomment+'.nii.gz')\n else:\n fout = os.path.join(\n opth,\n fimout.split('.')[0]+'.nii.gz')\n #---------------------------------------------------------------------------\n\n\n #> reduce the max value to 255\n if voxel_range and len(voxel_range)==1:\n im = voxel_range[0] * dctnii['im']/np.max(dctnii['im'])\n elif voxel_range and len(voxel_range)==2:\n #> normalise into range 0-1\n im = (dctnii['im'] - np.min(dctnii['im'])) / np.ptp(dctnii['im'])\n #> convert to voxel_range \n im = voxel_range[0] + im*(voxel_range[1] - voxel_range[0])\n else:\n return None\n\n #> output file name for the extra reference image\n imio.array2nii(\n im,\n dctnii['affine'],\n fout,\n trnsp = (dctnii['transpose'].index(0),\n dctnii['transpose'].index(1),\n dctnii['transpose'].index(2)),\n flip = dctnii['flip'])\n\n return {'fim':fout, 'im':im, 'affine':dctnii['affine']}"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncorrects for bias field in MR image(s) given in as a string (single file) or as a list of strings (multiple files). Output dictionary with the bias corrected file names. Options: - fimout: The name (with path) of the output file. It's ignored when multiple files are given as input. If given for a single file name, the and options are ignored. - outpath: Path to the output folder - fcomment: A prefix comment to the file name - executable: The path to the executable, overrides the above choice of software; if 'sitk' is given instead of the path, the Python module SimpleITK will be used if it is available. - exe_options: Options for the executable in the form of a list of strings. - sitk_image_mask: Image masking will be used if SimpleITK is chosen.", "response": "def bias_field_correction(\n fmr,\n fimout = '',\n outpath = '',\n fcomment = '_N4bias',\n executable = '',\n exe_options = [],\n sitk_image_mask = True,\n verbose = False,):\n\n ''' Correct for bias field in MR image(s) given in as a string\n (single file) or as a list of strings (multiple files). \n\n Output dictionary with the bias corrected file names.\n\n Options:\n - fimout: The name (with path) of the output file. It's \n ignored when multiple files are given as input. If\n given for a single file name, the and\n options are ignored.\n - outpath: Path to the output folder\n - fcomment: A prefix comment to the file name\n - executable: The path to the executable, overrides the above\n choice of software; if 'sitk' is given instead\n of the path, the Python module SimpleITK will be \n used if it is available.\n - exe_options: Options for the executable in the form of a list of\n strings.\n - sitk_image_mask: Image masking will be used if SimpleITK is\n chosen.\n '''\n\n if executable=='sitk' and not 'SimpleITK' in sys.modules:\n print 'e> if SimpleITK module is required for bias correction' \\\n +' it needs to be first installed.\\n' \\\n +' Install the module by:\\n' \\\n +' conda install -c https://conda.anaconda.org/simpleitk SimpleITK=1.2.0\\n' \\\n +' or pip install SimpleITK'\n return None\n\n #---------------------------------------------------------------------------\n # INPUT\n #---------------------------------------------------------------------------\n #> path to a single file \n if isinstance(fmr, basestring) and os.path.isfile(fmr):\n fins = [fmr]\n\n #> list of file paths\n elif isinstance(fmr, list) and all([os.path.isfile(f) for f in fmr]):\n fins = fmr\n print 'i> multiple input files => ignoring the single output file name.'\n fimout = ''\n\n #> path to a folder\n elif isinstance(fmr, basestring) and os.path.isdir(fmr):\n fins = [os.path.join(fmr, f) for f in os.listdir(fmr) if f.endswith(('.nii', '.nii.gz'))]\n print 'i> multiple input files from input folder.'\n fimout = ''\n\n else:\n raise ValueError('could not decode the input of floating images.')\n #---------------------------------------------------------------------------\n\n\n\n\n #---------------------------------------------------------------------------\n # OUTPUT\n #---------------------------------------------------------------------------\n #> output path\n if outpath=='' and fimout!='':\n opth = os.path.dirname(fimout)\n if opth=='':\n opth = os.path.dirname(fmr)\n fimout = os.path.join(opth, fimout)\n n4opth = opth\n fcomment = ''\n\n elif outpath=='':\n opth = os.path.dirname(fmr)\n #> N4 bias correction specific folder\n n4opth = os.path.join(opth, 'N4bias')\n\n else:\n opth = outpath\n #> N4 bias correction specific folder\n n4opth = os.path.join(opth, 'N4bias')\n\n imio.create_dir(n4opth)\n\n outdct = {}\n #---------------------------------------------------------------------------\n\n\n\n for fin in fins:\n\n if verbose:\n print 'i> input for bias correction:\\n', fin\n \n if fimout=='':\n # split path\n fspl = os.path.split(fin)\n \n # N4 bias correction file output paths\n fn4 = os.path.join( n4opth, fspl[1].split('.nii')[0]\\\n + fcomment +'.nii.gz')\n else:\n fn4 = fimout\n\n if not os.path.exists(fn4):\n\n if executable=='sitk':\n #==============================================\n # SimpleITK Bias field correction for T1 and T2\n #==============================================\n #> initialise the corrector\n corrector = sitk.N4BiasFieldCorrectionImageFilter()\n # numberFilltingLevels = 4\n\n # read input file \n im = sitk.ReadImage(fin)\n\n #> create a object specific mask\n fmsk = os.path.join( n4opth, fspl[1].split('.nii')[0] +'_sitk_mask.nii.gz')\n msk = sitk.OtsuThreshold(im, 0, 1, 200)\n sitk.WriteImage(msk, fmsk)\n \n #> cast to 32-bit float\n im = sitk.Cast( im, sitk.sitkFloat32 )\n\n #-------------------------------------------\n print 'i> correcting bias field for', fin \n n4out = corrector.Execute(im, msk)\n sitk.WriteImage(n4out, fn4)\n #-------------------------------------------\n\n if sitk_image_mask:\n if not 'fmsk' in outdct: outdct['fmsk'] = []\n outdct['fmsk'].append(fmsk)\n\n\n elif os.path.basename(executable)=='N4BiasFieldCorrection' \\\n and os.path.isfile(executable):\n \n cmd = [executable, '-i', fin, '-o', fn4]\n \n if verbose and os.path.basename(executable)=='N4BiasFieldCorrection':\n cmd.extend(['-v', '1'])\n\n cmd.extend(exe_options)\n \n call(cmd)\n\n if 'command' not in outdct:\n outdct['command'] = []\n outdct['command'].append(cmd)\n\n elif os.path.isfile(executable):\n\n cmd = [executable]\n cmd.extend(exe_options)\n call(cmd)\n\n if 'command' not in outdct:\n outdct['command'] = cmd\n \n else:\n print ' N4 bias corrected file seems already existing.'\n\n #> output to dictionary \n if not 'fim' in outdct: outdct['fim'] = []\n outdct['fim'].append(fn4)\n\n\n return outdct"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nextract ROI values of upsampled PET. If not provided it will be upsampled. imdic: dictionary of all image parts (image, path to file and affine transformation) > fpet: file name for quantitative PET > faff: file name and path to the affine (rigid) transformations, most likely for upscaled PET amyroi: module with ROI definitions in numbers datain: dictionary of all the input data", "response": "def roi_extraction(imdic, amyroi, datain, Cnt, use_stored=False):\n '''\n Extracting ROI values of upsampled PET. If not provided it will be upsampled.\n imdic: dictionary of all image parts (image, path to file and affine transformation) \n > fpet: file name for quantitative PET\n > faff: file name and path to the affine (rigid) transformations, most likely for upscaled PET\n amyroi: module with ROI definitions in numbers\n datain: dictionary of all the input data\n '''\n\n fpet = imdic['fpet']\n faff = imdic['faff']\n im = imdic['im']\n\n if Cnt['VERBOSE']: print 'i> extracting ROI values'\n\n # split the labels path\n lbpth = os.path.split(datain['T1lbl'])\n # # MR T1w NIfTI\n # if os.path.isfile(datain['T1nii']):\n # ft1w = datain['T1nii']\n # elif os.path.isfile(datain['T1bc']):\n # ft1w = datain['T1nii']\n # else:\n # print 'e> no T1w NIfTI!'\n # sys.exit()\n\n prcl_dir = os.path.dirname(datain['T1lbl'])\n\n dscrp = imio.getnii_descr(fpet)\n impvc = False\n if 'pvc' in dscrp.keys(): \n impvc = True\n if Cnt['VERBOSE']: print 'i> the provided image is partial volume corrected.'\n\n #------------------------------------------------------------------------------------------------------------ \n # next steps after having sorted out the upsampled input image and the rigid transformation for the T1w -> PET:\n\n # Get the labels before resampling to PET space so that the regions can be separated (more difficult to separate after resampling)\n if os.path.isfile(datain['T1lbl']):\n nilb = nib.load(datain['T1lbl'])\n A = nilb.get_sform()\n imlb = nilb.get_data()\n else:\n print 'e> parcellation label image not present!'\n sys.exit()\n\n # ===============================================================\n # get the segmentation/parcellation by creating an image for ROI extraction\n # (still in the original T1 orientation)\n roi_img = np.copy(imlb)\n\n # sum of all voxel values in any given ROI\n roisum = {}\n # sum of the mask values <0,1> used for getting the mean value of any ROI\n roimsk = {}\n for k in amyroi.rois.keys():\n roisum[k] = []\n roimsk[k] = []\n\n # extract the ROI values from PV corrected and uncorrected PET images\n for k in amyroi.rois.keys():\n roi_img[:] = 0\n for i in amyroi.rois[k]:\n roi_img[imlb==i] = 1.\n # now save the image mask to nii file \n froi1 = os.path.join(prcl_dir, lbpth[1].split('.')[0][:8]+'_'+k+'.nii.gz')\n nii_roi = nib.Nifti1Image(roi_img, A)\n nii_roi.header['cal_max'] = 1.\n nii_roi.header['cal_min'] = 0.\n nib.save(nii_roi, froi1)\n # file name for the resampled ROI to PET space\n froi2 = os.path.join(prcl_dir, os.path.basename(datain['T1lbl']).split('.')[0]+ '_toPET_'+k+'.nii.gz')\n if not use_stored and os.path.isfile( Cnt['RESPATH'] ):\n cmd = [Cnt['RESPATH'],\n '-ref', fpet,\n '-flo', froi1,\n '-trans', faff,\n '-res', froi2]#,\n #'-pad', '0']\n if not Cnt['VERBOSE']: cmd.append('-voff') \n call(cmd)\n # get the resampled ROI mask image\n rmsk = imio.getnii(froi2)\n rmsk[rmsk>1.] = 1.\n rmsk[rmsk<0.] = 0.\n\n # erode the cerebral white matter region if no PVC image\n if k=='wm' and not impvc:\n if Cnt['VERBOSE']: print 'i> eroding white matter as PET image not partial volume corrected.'\n nilb = nib.load(froi2)\n B = nilb.get_sform()\n tmp = ndi.filters.gaussian_filter(rmsk, imio.fwhm2sig(12,Cnt['VOXY']), mode='mirror')\n rmsk = np.float32(tmp>0.85)\n froi3 = os.path.join(prcl_dir, os.path.basename(datain['T1lbl']).split('.')[0]+ '_toPET_'+k+'_eroded.nii.gz')\n savenii(rmsk, froi3, B, Cnt)\n\n # ROI value and mask sums:\n rvsum = np.sum(im*rmsk)\n rmsum = np.sum(rmsk)\n roisum[k].append( rvsum )\n roimsk[k].append( rmsum )\n if Cnt['VERBOSE']: \n print ''\n print '================================================================'\n print 'i> ROI extracted:', k\n print ' > value sum:', rvsum\n print ' > # voxels :', rmsum\n print '================================================================'\n # --------------------------------------------------------------\n\n return roisum, roimsk"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget PET image from NIfTI file.", "response": "def getnii(fim, nan_replace=None, output='image'):\n '''Get PET image from NIfTI file.\n fim: input file name for the nifty image\n nan_replace: the value to be used for replacing the NaNs in the image.\n by default no change (None).\n output: option for choosing output: image, affine matrix or a dictionary with all info.\n ----------\n Return:\n 'image': outputs just an image (4D or 3D)\n 'affine': outputs just the affine matrix\n 'all': outputs all as a dictionary\n '''\n\n import numbers\n\n nim = nib.load(fim)\n\n dim = nim.header.get('dim')\n dimno = dim[0]\n\n if output=='image' or output=='all':\n imr = nim.get_data()\n # replace NaNs if requested\n if isinstance(nan_replace, numbers.Number): imr[np.isnan(imr)]\n\n imr = np.squeeze(imr)\n if dimno!=imr.ndim and dimno==4:\n dimno = imr.ndim\n \n #> get orientations from the affine\n ornt = nib.orientations.axcodes2ornt(nib.aff2axcodes(nim.affine))\n trnsp = tuple(np.int8(ornt[::-1,0]))\n flip = tuple(np.int8(ornt[:,1]))\n\n #> flip y-axis and z-axis and then transpose. Depends if dynamic (4 dimensions) or static (3 dimensions)\n if dimno==4:\n imr = np.transpose(imr[::-flip[0],::-flip[1],::-flip[2],:], (3,)+trnsp)\n elif dimno==3:\n imr = np.transpose(imr[::-flip[0],::-flip[1],::-flip[2]], trnsp)\n \n if output=='affine' or output=='all':\n # A = nim.get_sform()\n # if not A[:3,:3].any():\n # A = nim.get_qform()\n A = nim.affine\n\n if output=='all':\n out = { 'im':imr,\n 'affine':A,\n 'fim':fim,\n 'dtype':nim.get_data_dtype(),\n 'shape':imr.shape,\n 'hdr':nim.header,\n 'transpose':trnsp,\n 'flip':flip}\n elif output=='image':\n out = imr\n elif output=='affine':\n out = A\n else:\n raise NameError('Unrecognised output request!')\n\n return out"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nextract the custom description header field to dictionary", "response": "def getnii_descr(fim):\n '''\n Extracts the custom description header field to dictionary\n '''\n nim = nib.load(fim)\n hdr = nim.header\n rcnlst = hdr['descrip'].item().split(';')\n rcndic = {}\n \n if rcnlst[0]=='':\n # print 'w> no description in the NIfTI header'\n return rcndic\n \n for ci in range(len(rcnlst)):\n tmp = rcnlst[ci].split('=')\n rcndic[tmp[0]] = tmp[1]\n return rcndic"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nstores the numpy array im to a NIfTI file fnii.", "response": "def array2nii(im, A, fnii, descrip='', trnsp=(), flip=(), storage_as=[]):\n '''Store the numpy array 'im' to a NIfTI file 'fnii'.\n ----\n Arguments:\n 'im': image to be stored in NIfTI\n 'A': affine transformation\n 'fnii': output NIfTI file name.\n 'descrip': the description given to the file\n 'trsnp': transpose/permute the dimensions.\n In NIfTI it has to be in this order: [x,y,z,t,...])\n 'flip': flip tupple for flipping the direction of x,y,z axes. \n (1: no flip, -1: flip)\n 'storage_as': uses the flip and displacement as given by the following\n NifTI dictionary, obtained using\n nimpa.getnii(filepath, output='all').\n '''\n\n if not len(trnsp) in [0,3,4] and not len(flip) in [0,3]:\n raise ValueError('e> number of flip and/or transpose elements is incorrect.')\n\n #---------------------------------------------------------------------------\n #> TRANSLATIONS and FLIPS\n #> get the same geometry as the input NIfTI file in the form of dictionary,\n #>>as obtained from getnii(..., output='all')\n\n #> permute the axis order in the image array\n if isinstance(storage_as, dict) and 'transpose' in storage_as \\\n and 'flip' in storage_as:\n\n trnsp = (storage_as['transpose'].index(0),\n storage_as['transpose'].index(1),\n storage_as['transpose'].index(2))\n\n flip = storage_as['flip']\n \n\n if trnsp==():\n im = im.transpose()\n #> check if the image is 4D (dynamic) and modify as needed\n elif len(trnsp)==3 and im.ndim==4:\n trnsp = tuple([t+1 for t in trnsp] + [0])\n im = im.transpose(trnsp)\n else:\n im = im.transpose(trnsp)\n \n\n #> perform flip of x,y,z axes after transposition into proper NIfTI order\n if flip!=() and len(flip)==3:\n im = im[::-flip[0], ::-flip[1], ::-flip[2], ...]\n #---------------------------------------------------------------------------\n\n nii = nib.Nifti1Image(im, A)\n hdr = nii.header\n hdr.set_sform(None, code='scanner')\n hdr['cal_max'] = np.max(im) #np.percentile(im, 90) #\n hdr['cal_min'] = np.min(im)\n hdr['descrip'] = descrip\n nib.save(nii, fnii)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets the orientation from NIfTI sform. Not fully functional yet.", "response": "def orientnii(imfile):\n '''Get the orientation from NIfTI sform. Not fully functional yet.'''\n strorient = ['L-R', 'S-I', 'A-P']\n niiorient = []\n niixyz = np.zeros(3,dtype=np.int8)\n if os.path.isfile(imfile):\n nim = nib.load(imfile)\n pct = nim.get_data()\n A = nim.get_sform()\n for i in range(3):\n niixyz[i] = np.argmax(abs(A[i,:-1]))\n niiorient.append( strorient[ niixyz[i] ] )\n print niiorient"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef nii_gzip(imfile, outpath=''):\n '''Compress *.gz file'''\n import gzip\n with open(imfile, 'rb') as f:\n d = f.read()\n # Now store the compressed data\n if outpath=='':\n fout = imfile+'.gz'\n else:\n fout = os.path.join(outpath, os.path.basename(imfile)+'.gz')\n # store compressed file data from 'd' variable\n with gzip.open(fout, 'wb') as f:\n f.write(d)\n return fout", "response": "Compress the NII - IML file and store it in a file"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef pick_t1w(mri):\n ''' Pick the MR T1w from the dictionary for MR->PET registration.\n '''\n\n if isinstance(mri, dict):\n # check if NIfTI file is given\n if 'T1N4' in mri and os.path.isfile(mri['T1N4']):\n ft1w = mri['T1N4']\n # or another bias corrected\n elif 'T1bc' in mri and os.path.isfile(mri['T1bc']):\n ft1w = mri['T1bc']\n elif 'T1nii' in mri and os.path.isfile(mri['T1nii']):\n ft1w = mri['T1nii']\n elif 'T1DCM' in mri and os.path.exists(mri['MRT1W']):\n # create file name for the converted NIfTI image\n fnii = 'converted'\n call( [rs.DCM2NIIX, '-f', fnii, mri['T1nii'] ] )\n ft1nii = glob.glob( os.path.join(mri['T1nii'], '*converted*.nii*') )\n ft1w = ft1nii[0]\n else:\n print 'e> disaster: could not find a T1w image!'\n return None\n \n else:\n ('e> no correct input found for the T1w image')\n return None\n\n return ft1w", "response": "Pick the MR T1w from the dictionary for MR - > PET registration."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef dcminfo(dcmvar, verbose=True):\n ''' Get basic info about the DICOM file/header.\n '''\n\n if isinstance(dcmvar, basestring):\n if verbose:\n print 'i> provided DICOM file:', dcmvar\n dhdr = dcm.dcmread(dcmvar)\n elif isinstance(dcmvar, dict):\n dhdr = dcmvar\n elif isinstance(dcmvar, dcm.dataset.FileDataset):\n dhdr = dcmvar\n\n dtype = dhdr[0x08, 0x08].value\n if verbose:\n print ' Image Type:', dtype\n\n #-------------------------------------------\n #> scanner ID\n scanner_vendor = 'unknown'\n if [0x008, 0x070] in dhdr:\n scanner_vendor = dhdr[0x008, 0x070].value\n\n scanner_model = 'unknown'\n if [0x008, 0x1090] in dhdr:\n scanner_model = dhdr[0x008, 0x1090].value\n\n scanner_id = 'other'\n if any(s in scanner_model for s in ['mMR', 'Biograph']) and 'siemens' in scanner_vendor.lower():\n scanner_id = 'mmr'\n #-------------------------------------------\n\n #> CSA type (mMR)\n csatype = ''\n if [0x29, 0x1108] in dhdr:\n csatype = dhdr[0x29, 0x1108].value\n if verbose:\n print ' CSA Data Type:', csatype\n\n #> DICOM comment or on MR parameters\n cmmnt = ''\n if [0x20, 0x4000] in dhdr:\n cmmnt = dhdr[0x0020, 0x4000].value\n if verbose:\n print ' Comments:', cmmnt\n\n #> MR parameters (echo time, etc) \n TR = 0\n TE = 0\n if [0x18, 0x80] in dhdr:\n TR = float(dhdr[0x18, 0x80].value)\n if verbose: print ' TR:', TR\n if [0x18, 0x81] in dhdr:\n TE = float(dhdr[0x18, 0x81].value)\n if verbose: print ' TE:', TE\n\n\n #> check if it is norm file\n if any('PET_NORM' in s for s in dtype) or cmmnt=='PET Normalization data' or csatype=='MRPETNORM':\n out = ['raw', 'norm', scanner_id]\n\n elif any('PET_LISTMODE' in s for s in dtype) or cmmnt=='Listmode' or csatype=='MRPETLM_LARGE':\n out = ['raw', 'list', scanner_id]\n\n elif any('MRPET_UMAP3D' in s for s in dtype) or cmmnt=='MR based umap':\n out = ['raw', 'mumap', 'ute', 'mr', scanner_id]\n\n elif TR>400 and TR<2500 and TE<20:\n out = ['mr', 't1', scanner_id]\n\n elif TR>2500 and TE>50:\n out = ['mr', 't2', scanner_id]\n\n #> UTE's two sequences: UTE2\n elif TR<50 and TE<20 and TE>1:\n out = ['mr', 'ute', 'ute2', scanner_id]\n\n #> UTE1\n elif TR<50 and TE<20 and TE<0.1 and TR>0 and TE>0:\n out = ['mr', 'ute', 'ute1', scanner_id]\n\n #> physio data\n elif 'PET_PHYSIO' in dtype or 'physio' in cmmnt.lower():\n out = ['raw', 'physio', scanner_id]\n\n else:\n out = ['unknown', cmmnt]\n\n return out", "response": "Get basic info about the DICOM file."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nlist all DICOM files in the datain dictionary of input data.", "response": "def list_dcm_datain(datain):\n ''' List all DICOM file paths in the datain dictionary of input data.\n '''\n\n if not isinstance(datain, dict):\n raise ValueError('The input is not a dictionary!')\n\n dcmlst = []\n # list of mu-map DICOM files\n if 'mumapDCM' in datain:\n dcmump = os.listdir(datain['mumapDCM'])\n # accept only *.dcm extensions\n dcmump = [os.path.join(datain['mumapDCM'],d) for d in dcmump if d.endswith(dcmext)]\n dcmlst += dcmump\n\n if 'T1DCM' in datain:\n dcmt1 = os.listdir(datain['T1DCM'])\n # accept only *.dcm extensions\n dcmt1 = [os.path.join(datain['T1DCM'],d) for d in dcmt1 if d.endswith(dcmext)]\n dcmlst += dcmt1\n\n if 'T2DCM' in datain:\n dcmt2 = os.listdir(datain['T2DCM'])\n # accept only *.dcm extensions\n dcmt2 = [os.path.join(datain['T2DCM'],d) for d in dcmt2 if d.endswith(dcmext)]\n dcmlst += dcmt2\n\n if 'UTE1' in datain:\n dcmute1 = os.listdir(datain['UTE1'])\n # accept only *.dcm extensions\n dcmute1 = [os.path.join(datain['UTE1'],d) for d in dcmute1 if d.endswith(dcmext)]\n dcmlst += dcmute1\n\n if 'UTE2' in datain:\n dcmute2 = os.listdir(datain['UTE2'])\n # accept only *.dcm extensions\n dcmute2 = [os.path.join(datain['UTE2'],d) for d in dcmute2 if d.endswith(dcmext)]\n dcmlst += dcmute2\n\n #-list-mode data dcm\n if 'lm_dcm' in datain:\n dcmlst += [datain['lm_dcm']]\n\n if 'lm_ima' in datain:\n dcmlst += [datain['lm_ima']]\n\n #-norm\n if 'nrm_dcm' in datain:\n dcmlst += [datain['nrm_dcm']]\n\n if 'nrm_ima' in datain:\n dcmlst += [datain['nrm_ima']]\n\n return dcmlst"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef dcmanonym(\n dcmpth,\n displayonly=False,\n patient='anonymised',\n physician='anonymised',\n dob='19800101',\n verbose=True):\n\n ''' Anonymise DICOM file(s)\n Arguments:\n > dcmpth: it can be passed as a single DICOM file, or\n a folder containing DICOM files, or a list of DICOM file paths.\n > patient: the name of the patient.\n > physician:the name of the referring physician.\n > dob: patient's date of birth.\n > verbose: display processing output.\n '''\n\n #> check if a single DICOM file\n if isinstance(dcmpth, basestring) and os.path.isfile(dcmpth):\n dcmlst = [dcmpth]\n if verbose:\n print 'i> recognised the input argument as a single DICOM file.'\n\n #> check if a folder containing DICOM files\n elif isinstance(dcmpth, basestring) and os.path.isdir(dcmpth):\n dircontent = os.listdir(dcmpth)\n #> create a list of DICOM files inside the folder\n dcmlst = [os.path.join(dcmpth,d) for d in dircontent if os.path.isfile(os.path.join(dcmpth,d)) and d.endswith(dcmext)]\n if verbose:\n print 'i> recognised the input argument as the folder containing DICOM files.'\n\n #> check if a folder containing DICOM files\n elif isinstance(dcmpth, list):\n if not all([os.path.isfile(d) and d.endswith(dcmext) for d in dcmpth]):\n raise IOError('Not all files in the list are DICOM files.')\n dcmlst = dcmpth\n if verbose:\n print 'i> recognised the input argument as the list of DICOM file paths.'\n\n #> check if dictionary of data input \n elif isinstance(dcmpth, dict) and 'corepath' in dcmpth:\n dcmlst = list_dcm_datain(dcmpth)\n if verbose:\n print 'i> recognised the input argument as the dictionary of scanner data.'\n\n else:\n raise IOError('Unrecognised input!')\n\n\n\n for dcmf in dcmlst:\n\n #> read the file\n dhdr = dcm.dcmread(dcmf)\n\n #> get the basic info about the DICOM file\n dcmtype = dcminfo(dhdr, verbose=False)\n if verbose:\n print '-------------------------------'\n print 'i> the DICOM file is for:', dcmtype\n\n #> anonymise mMR data.\n if 'mmr' in dcmtype:\n\n if [0x029, 0x1120] in dhdr and dhdr[0x029, 0x1120].name=='[CSA Series Header Info]':\n csafield = dhdr[0x029, 0x1120]\n csa = csafield.value\n elif [0x029, 0x1020] in dhdr and dhdr[0x029, 0x1020].name=='[CSA Series Header Info]':\n csafield = dhdr[0x029, 0x1020]\n csa = csafield.value\n else:\n csa = ''\n\n # string length considered for replacement\n strlen = 200\n \n idx = [m.start() for m in re.finditer(r'([Pp]atients{0,1}[Nn]ame)', csa)]\n if idx and verbose:\n print ' > found sensitive information deep in DICOM headers:', dcmtype\n\n\n #> run the anonymisation \n iupdate = 0\n\n for i in idx:\n ci = i - iupdate\n\n if displayonly:\n print ' > sensitive info:'\n print ' ', csa[ci:ci+strlen]\n continue\n\n rplcmnt = re.sub( r'(\\{\\s*\\\"{1,2}\\W*\\w+\\W*\\w+\\W*\\\"{1,2}\\s*\\})',\n '{ \"\"' +patient+ '\"\" }',\n csa[ci:ci+strlen]\n )\n #> update string\n csa = csa[:ci] + rplcmnt + csa[ci+strlen:]\n print ' > removed sensitive information.'\n #> correct for the number of removed letters\n iupdate = strlen-len(rplcmnt)\n\n #> update DICOM\n if not displayonly and csa!='':\n csafield.value = csa\n\n\n #> Patient's name\n if [0x010,0x010] in dhdr:\n if displayonly:\n print ' > sensitive info:', dhdr[0x010,0x010].name\n print ' ', dhdr[0x010,0x010].value\n else:\n dhdr[0x010,0x010].value = patient\n if verbose: print ' > anonymised patients name'\n\n #> date of birth\n if [0x010,0x030] in dhdr:\n if displayonly:\n print ' > sensitive info:', dhdr[0x010,0x030].name\n print ' ', dhdr[0x010,0x030].value\n else:\n dhdr[0x010,0x030].value = dob\n if verbose: print ' > anonymised date of birh'\n\n #> physician's name\n if [0x008, 0x090] in dhdr:\n if displayonly:\n print ' > sensitive info:', dhdr[0x008,0x090].name\n print ' ', dhdr[0x008,0x090].value\n else:\n dhdr[0x008,0x090].value = physician\n if verbose: print ' > anonymised physician name'\n\n dhdr.save_as(dcmf)", "response": "Anonymise DICOM file(s)\n Arguments:\n > dcmpth: it can be passed as a single DICOM file, or\n a folder containing DICOM files, or a list of DICOM file paths.\n > patient: the name of the patient.\n > physician:the name of the referring physician.\n > dob: patient's date of birth.\n > verbose: display processing output."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef dcmsort(folder, copy_series=False, verbose=False):\n ''' Sort out the DICOM files in the folder according to the recorded series.\n '''\n\n # list files in the input folder\n files = os.listdir(folder)\n\n srs = {}\n for f in files:\n if os.path.isfile(os.path.join(folder, f)) and f.endswith(dcmext):\n dhdr = dcm.read_file(os.path.join(folder, f))\n #---------------------------------\n # image size\n imsz = np.zeros(2, dtype=np.int64)\n if [0x028,0x010] in dhdr:\n imsz[0] = dhdr[0x028,0x010].value\n if [0x028,0x011] in dhdr:\n imsz[1] = dhdr[0x028,0x011].value\n # voxel size\n vxsz = np.zeros(3, dtype=np.float64)\n if [0x028,0x030] in dhdr and [0x018,0x050] in dhdr:\n pxsz = np.array([float(e) for e in dhdr[0x028,0x030].value])\n vxsz[:2] = pxsz\n vxsz[2] = float(dhdr[0x018,0x050].value)\n # orientation\n ornt = np.zeros(6, dtype=np.float64)\n if [0x020,0x037] in dhdr:\n ornt = np.array([float(e) for e in dhdr[0x20,0x37].value])\n # seires description, time and study time\n srs_dcrp = dhdr[0x0008, 0x103e].value\n srs_time = dhdr[0x0008, 0x0031].value[:6]\n std_time = dhdr[0x0008, 0x0030].value[:6]\n\n if verbose:\n print 'series desciption:', srs_dcrp\n print 'series time:', srs_time\n print 'study time:', std_time\n print '---------------------------------------------------'\n\n #----------\n # series for any category (can be multiple scans within the same category)\n recognised_series = False\n srs_k = srs.keys()\n for s in srs_k:\n if np.array_equal(srs[s]['imorient'], ornt) and \\\n np.array_equal(srs[s]['imsize'], imsz) and \\\n np.array_equal(srs[s]['voxsize'], vxsz) and \\\n srs[s]['tseries'] == srs_time:\n recognised_series = True\n break\n # if series was not found, create one\n if not recognised_series:\n s = srs_dcrp + '_' + srs_time\n srs[s] = {}\n srs[s]['imorient'] = ornt\n srs[s]['imsize'] = imsz\n srs[s]['voxsize'] = vxsz\n srs[s]['tseries'] = srs_time\n # append the file name\n if 'files' not in srs[s]: srs[s]['files'] = []\n if copy_series:\n srsdir = os.path.join(folder, s)\n create_dir( srsdir )\n shutil.copy(os.path.join(folder, f), srsdir)\n srs[s]['files'].append( os.path.join(srsdir, f) )\n else:\n srs[s]['files'].append( os.path.join(folder, f) )\n\n return srs", "response": "Sort out the DICOM files in the folder according to the recorded series."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef niisort(\n fims,\n memlim=True\n ):\n\n ''' Sort all input NIfTI images and check their shape.\n Output dictionary of image files and their properties.\n Options:\n memlim -- when processing large numbers of frames the memory may\n not be large enough. memlim causes that the output does not contain\n all the arrays corresponding to the images.\n '''\n # number of NIfTI images in folder\n Nim = 0\n # sorting list (if frame number is present in the form '_frm
', where d is a digit)\n sortlist = []\n\n for f in fims:\n if f.endswith('.nii') or f.endswith('.nii.gz'):\n Nim += 1\n _match = re.search('(?<=_frm)\\d*', f)\n if _match:\n frm = int(_match.group(0))\n freelists = [frm not in l for l in sortlist]\n listidx = [i for i,f in enumerate(freelists) if f]\n if listidx:\n sortlist[listidx[0]].append(frm)\n else:\n sortlist.append([frm])\n else:\n sortlist.append([None])\n\n if len(sortlist)>1:\n # if more than one dynamic set is given, the dynamic mode is cancelled.\n dyn_flg = False\n sortlist = range(Nim)\n elif len(sortlist)==1:\n dyn_flg = True\n sortlist = sortlist[0]\n else:\n raise ValueError('e> niisort input error.')\n \n \n # number of frames (can be larger than the # images)\n Nfrm = max(sortlist)+1\n # sort the list according to the frame numbers\n _fims = ['Blank']*Nfrm\n # list of NIfTI image shapes and data types used\n shape = []\n dtype = []\n _nii = []\n for i in range(Nim):\n if dyn_flg:\n _fims[sortlist[i]] = fims[i]\n _nii = nib.load(fims[i])\n dtype.append(_nii.get_data_dtype()) \n shape.append(_nii.shape)\n else:\n _fims[i] = fims[i]\n _nii = nib.load(fims[i])\n dtype.append(_nii.get_data_dtype()) \n shape.append(_nii.shape)\n\n # check if all images are of the same shape and data type\n if _nii and not shape.count(_nii.shape)==len(shape):\n raise ValueError('Input images are of different shapes.')\n if _nii and not dtype.count(_nii.get_data_dtype())==len(dtype):\n raise TypeError('Input images are of different data types.')\n # image shape must be 3D\n if _nii and not len(_nii.shape)==3:\n raise ValueError('Input image(s) must be 3D.')\n\n out = {'shape':_nii.shape[::-1],\n 'files':_fims, \n 'sortlist':sortlist,\n 'dtype':_nii.get_data_dtype(), \n 'N':Nim}\n\n if memlim and Nfrm>50:\n imdic = getnii(_fims[0], output='all')\n affine = imdic['affine']\n else:\n # get the images into an array\n _imin = np.zeros((Nfrm,)+_nii.shape[::-1], dtype=_nii.get_data_dtype())\n for i in range(Nfrm):\n if i in sortlist:\n imdic = getnii(_fims[i], output='all')\n _imin[i,:,:,:] = imdic['im']\n affine = imdic['affine']\n out['im'] = _imin[:Nfrm,:,:,:]\n\n out['affine'] = affine\n\n return out", "response": "Sort all input NIfTI images and check their shape."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dcm2nii(\n dcmpth,\n fimout = '',\n fprefix = 'converted-from-DICOM_',\n fcomment = '',\n outpath = '',\n timestamp = True,\n executable = '',\n force = False,\n ):\n ''' Convert DICOM files in folder (indicated by ) using DCM2NIIX\n third-party software.\n '''\n\n # skip conversion if the output already exists and not force is selected\n if os.path.isfile(fimout) and not force:\n return fimout\n\n if executable=='':\n try:\n import resources\n executable = resources.DCM2NIIX\n except:\n raise NameError('e> could not import resources \\\n or find variable DCM2NIIX in resources.py')\n elif not os.path.isfile(executable):\n raise IOError('e> the executable is incorrect!')\n\n if not os.path.isdir(dcmpth):\n raise IOError('e> the provided DICOM path is not a folder!')\n\n #> output path\n if outpath=='' and fimout!='' and '/' in fimout:\n opth = os.path.dirname(fimout)\n if opth=='':\n opth = dcmpth\n fimout = os.path.basename(fimout)\n\n elif outpath=='':\n opth = dcmpth\n\n else:\n opth = outpath\n\n create_dir(opth)\n\n if fimout=='':\n fimout = fprefix\n if timestamp:\n fimout += time_stamp(simple_ascii=True)\n\n fimout = fimout.split('.nii')[0]\n\n\n # convert the DICOM mu-map images to nii\n call([executable, '-f', fimout, '-o', opth, dcmpth])\n\n fniiout = glob.glob( os.path.join(opth, '*'+fimout+'*.nii*') )\n\n if fniiout:\n return fniiout[0]\n else:\n raise ValueError('e> could not get the output file!')", "response": "Convert the DICOM files in the specified folder using DCM2NIIX."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef dcm2im(fpth):\n ''' Get the DICOM files from 'fpth' into an image with the affine transformation.\n fpth can be a list of DICOM files or a path (string) to the folder with DICOM files.\n '''\n\n # possible DICOM file extensions\n ext = ('dcm', 'DCM', 'ima', 'IMA') \n \n # case when given a folder path\n if isinstance(fpth, basestring) and os.path.isdir(fpth):\n SZ0 = len([d for d in os.listdir(fpth) if d.endswith(ext)])\n # list of DICOM files\n fdcms = os.listdir(fpth)\n fdcms = [os.path.join(fpth,f) for f in fdcms if f.endswith(ext)]\n \n # case when list of DICOM files is given\n elif isinstance(fpth, list) and os.path.isfile(os.path.join(fpth[0])):\n SZ0 = len(fpth)\n # list of DICOM files\n fdcms = fpth\n fdcms = [f for f in fdcms if f.endswith(ext)]\n else:\n raise NameError('Unrecognised input for DICOM files.')\n\n if SZ0<1:\n print 'e> no DICOM images in the specified path.'\n raise IOError('Input DICOM images not recognised')\n\n # pick single DICOM header\n dhdr = dcm.read_file(fdcms[0])\n\n #------------------------------------\n # some info, e.g.: patient position and series UID\n if [0x018, 0x5100] in dhdr:\n ornt = dhdr[0x18,0x5100].value\n else:\n ornt = 'unkonwn'\n # Series UID\n sruid = dhdr[0x0020, 0x000e].value\n #------------------------------------\n\n #------------------------------------\n # INIT\n # image position \n P = np.zeros((SZ0,3), dtype=np.float64)\n #image orientation\n Orn = np.zeros((SZ0,6), dtype=np.float64)\n #xy resolution\n R = np.zeros((SZ0,2), dtype=np.float64)\n #slice thickness\n S = np.zeros((SZ0,1), dtype=np.float64)\n #slope and intercept\n SI = np.ones((SZ0,2), dtype=np.float64)\n SI[:,1] = 0\n\n #image data as an list of array for now\n IM = []\n #------------------------------------\n\n c = 0\n for d in fdcms:\n dhdr = dcm.read_file(d)\n if [0x20,0x32] in dhdr and [0x20,0x37] in dhdr and [0x28,0x30] in dhdr:\n P[c,:] = np.array([float(f) for f in dhdr[0x20,0x32].value])\n Orn[c,:] = np.array([float(f) for f in dhdr[0x20,0x37].value])\n R[c,:] = np.array([float(f) for f in dhdr[0x28,0x30].value])\n S[c,:] = float(dhdr[0x18,0x50].value)\n else:\n print 'e> could not read all the DICOM tags.'\n return {'im':[], 'affine':[], 'shape':[], 'orient':ornt, 'sruid':sruid}\n \n if [0x28,0x1053] in dhdr and [0x28,0x1052] in dhdr:\n SI[c,0] = float(dhdr[0x28,0x1053].value)\n SI[c,1] = float(dhdr[0x28,0x1052].value)\n IM.append(dhdr.pixel_array)\n c += 1\n\n\n #check if orientation/resolution is the same for all slices\n if np.sum(Orn-Orn[0,:]) > 1e-6:\n print 'e> varying orientation for slices'\n else:\n Orn = Orn[0,:]\n if np.sum(R-R[0,:]) > 1e-6:\n print 'e> varying resolution for slices'\n else:\n R = R[0,:]\n\n # Patient Position\n #patpos = dhdr[0x18,0x5100].value\n # Rows and Columns\n if [0x28,0x10] in dhdr and [0x28,0x11] in dhdr:\n SZ2 = dhdr[0x28,0x10].value\n SZ1 = dhdr[0x28,0x11].value\n # image resolution\n SZ_VX2 = R[0]\n SZ_VX1 = R[1]\n\n #now sort the images along k-dimension\n k = np.argmin(abs(Orn[:3]+Orn[3:]))\n #sorted indeces\n si = np.argsort(P[:,k])\n Pos = np.zeros(P.shape, dtype=np.float64)\n im = np.zeros((SZ0, SZ1, SZ2 ), dtype=np.float32)\n\n #check if the detentions are in agreement (the pixel array could be transposed...)\n if IM[0].shape[0]==SZ1:\n for i in range(SZ0):\n im[i,:,:] = IM[si[i]]*SI[si[i],0] + SI[si[i],1]\n Pos[i,:] = P[si[i]]\n else:\n for i in range(SZ0):\n im[i,:,:] = IM[si[i]].T * SI[si[i],0] + SI[si[i],1]\n Pos[i,:] = P[si[i]]\n\n # proper slice thickness\n Zz = (P[si[-1],2] - P[si[0],2])/(SZ0-1)\n Zy = (P[si[-1],1] - P[si[0],1])/(SZ0-1)\n Zx = (P[si[-1],0] - P[si[0],0])/(SZ0-1)\n \n\n # dictionary for affine and image size for the image\n A = {\n 'AFFINE':np.array([[SZ_VX2*Orn[0], SZ_VX1*Orn[3], Zx, Pos[0,0]],\n [SZ_VX2*Orn[1], SZ_VX1*Orn[4], Zy, Pos[0,1]],\n [SZ_VX2*Orn[2], SZ_VX1*Orn[5], Zz, Pos[0,2]],\n [0., 0., 0., 1.]]),\n 'SHAPE':(SZ0, SZ1, SZ2)\n }\n\n #the returned image is already scaled according to the dcm header\n return {'im':im, 'affine':A['AFFINE'], 'shape':A['SHAPE'], 'orient':ornt, 'sruid':sruid}", "response": "Get the DICOM files from fpth into an image with the affine transformation."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets the path to the local folder for NiftyPET resources.", "response": "def path_niftypet_local():\n '''Get the path to the local (home) folder for NiftyPET resources.'''\n # if using conda put the resources in the folder with the environment name\n if 'CONDA_DEFAULT_ENV' in os.environ:\n env = os.environ['CONDA_DEFAULT_ENV']\n print 'i> conda environment found:', env\n else:\n env = ''\n # create the path for the resources files according to the OS platform\n if platform.system() == 'Linux' :\n path_resources = os.path.join( os.path.join(os.path.expanduser('~'), '.niftypet'), env )\n elif platform.system() == 'Windows' :\n path_resources = os.path.join( os.path.join(os.getenv('LOCALAPPDATA'), '.niftypet'), env )\n elif platform.system() == 'Darwin':\n path_resources = os.path.join( os.path.join(os.path.expanduser('~'), '.niftypet'), env )\n else:\n raise SystemError('e> the operating system is not recognised and therefore not supported!')\n\n return path_resources"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nlocating the CUDA environment on the system.", "response": "def find_cuda():\n '''Locate the CUDA environment on the system.'''\n # search the PATH for NVCC\n for fldr in os.environ['PATH'].split(os.pathsep):\n cuda_path = join(fldr, 'nvcc')\n if os.path.exists(cuda_path):\n cuda_path = os.path.dirname(os.path.dirname(cuda_path))\n break\n cuda_path = None\n \n if cuda_path is None:\n print 'w> nvcc compiler could not be found from the PATH!'\n return None\n\n # serach for the CUDA library path\n lcuda_path = os.path.join(cuda_path, 'lib64')\n if 'LD_LIBRARY_PATH' in os.environ.keys():\n if lcuda_path in os.environ['LD_LIBRARY_PATH'].split(os.pathsep):\n print 'i> found CUDA lib64 in LD_LIBRARY_PATH: ', lcuda_path\n elif os.path.isdir(lcuda_path):\n print 'i> found CUDA lib64 in : ', lcuda_path\n else:\n print 'w> folder for CUDA library (64-bit) could not be found!'\n\n\n return cuda_path, lcuda_path"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfiguring out what GPU devices are available and choose the supported ones.", "response": "def dev_setup():\n '''figure out what GPU devices are available and choose the supported ones.'''\n\n # check first if NiftyPET was already installed and use the choice of GPU\n path_resources = path_niftypet_local()\n # if so, import the resources and get the constants\n if os.path.isfile(os.path.join(path_resources,'resources.py')):\n sys.path.append(path_resources)\n try:\n import resources\n except ImportError as ie:\n print '----------------------------'\n print 'e> Import Error: NiftyPET''s resources file could not be imported. It should be in ''~/.niftypet/resources.py'' but likely it does not exists.'\n print '----------------------------'\n else:\n print 'e> resources file not found/installed.'\n return None\n\n # get all constants and check if device is already chosen\n Cnt = resources.get_setup()\n if 'CCARCH' in Cnt and 'DEVID' in Cnt:\n print 'i> using this CUDA architecture(s):', Cnt['CCARCH']\n return Cnt['CCARCH']\n\n # get the current locations\n path_current = os.path.dirname( os.path.realpath(__file__) )\n path_resins = os.path.join(path_current, 'resources')\n path_dinf = os.path.join(path_current, 'niftypet')\n path_dinf = os.path.join(path_dinf, 'nimpa')\n path_dinf = os.path.join(path_dinf, 'dinf')\n # temporary installation location for identifying the CUDA devices\n path_tmp_dinf = os.path.join(path_resins,'dinf')\n # if the folder 'path_tmp_dinf' exists, delete it\n if os.path.isdir(path_tmp_dinf):\n shutil.rmtree(path_tmp_dinf)\n # copy the device_info module to the resources folder within the installation package\n shutil.copytree( path_dinf, path_tmp_dinf)\n # create a build using cmake\n if platform.system()=='Windows':\n path_tmp_build = os.path.join(path_tmp_dinf, 'build')\n elif platform.system() in ['Linux', 'Darwin']:\n path_tmp_build = os.path.join(path_tmp_dinf, 'build')\n else:\n print 'w> the operating system {} is not supported for GPU'.format(platform.system())\n return ''\n \n os.makedirs(path_tmp_build)\n os.chdir(path_tmp_build)\n if platform.system()=='Windows':\n subprocess.call(\n ['cmake', '../', '-DPYTHON_INCLUDE_DIRS='+pyhdr,\n '-DPYTHON_PREFIX_PATH='+prefix, '-G', Cnt['MSVC_VRSN']]\n )\n subprocess.call(['cmake', '--build', './', '--config', 'Release'])\n path_tmp_build = os.path.join(path_tmp_build, 'Release')\n\n elif platform.system() in ['Linux', 'Darwin']:\n subprocess.call(\n ['cmake', '../', '-DPYTHON_INCLUDE_DIRS='+pyhdr,\n '-DPYTHON_PREFIX_PATH='+prefix]\n )\n subprocess.call(['cmake', '--build', './'])\n else:\n print 'e> This operating systems {} is not supported!'.format(platform.system())\n return ''\n \n # imoprt the new module for device properties\n sys.path.insert(0, path_tmp_build)\n try:\n import dinf\n except ImportError:\n print 'e> could not compile a CUDA C file--assuming no CUDA installation.'\n return ''\n \n # get the list of installed CUDA devices\n Ldev = dinf.dev_info(0)\n # extract the compute capability as a single number \n cclist = [int(str(e[2])+str(e[3])) for e in Ldev]\n # get the list of supported CUDA devices (with minimum compute capability)\n spprtd = [str(cc) for cc in cclist if cc>=mincc]\n ccstr = ''\n if spprtd:\n # best for the default CUDA device\n i = [int(s) for s in spprtd]\n devid = i.index(max(i))\n #-----------------------------------------------------------------------------------\n # form return list of compute capability numbers for which the software will be compiled\n \n for cc in spprtd:\n ccstr += '-gencode=arch=compute_'+cc+',code=compute_'+cc+';'\n #-----------------------------------------------------------------------------------\n else:\n devid = None\n\n # remove the temporary path\n sys.path.remove(path_tmp_build)\n # delete the build once the info about the GPUs has been obtained\n os.chdir(path_current)\n shutil.rmtree(path_tmp_dinf, ignore_errors=True)\n\n # passing this setting to resources.py\n fpth = os.path.join(path_resources,'resources.py') #resource_filename(__name__, 'resources/resources.py')\n f = open(fpth, 'r')\n rsrc = f.read()\n f.close()\n # get the region of keeping in synch with Python\n i0 = rsrc.find('### start GPU properties ###')\n i1 = rsrc.find('### end GPU properties ###')\n # list of constants which will be kept in sych from Python\n cnt_list = ['DEV_ID', 'CC_ARCH']\n val_list = [str(devid), '\\''+ccstr+'\\'']\n # update the resource.py file\n strNew = '### start GPU properties ###\\n'\n for i in range(len(cnt_list)):\n strNew += cnt_list[i]+' = '+val_list[i] + '\\n'\n rsrcNew = rsrc[:i0] + strNew + rsrc[i1:] \n f = open(fpth, 'w')\n f.write(rsrcNew)\n f.close()\n\n return ccstr"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef resources_setup():\n '''\n This function checks CUDA devices, selects some and installs resources.py\n '''\n print 'i> installing file into home directory if it does not exist.'\n path_current = os.path.dirname( os.path.realpath(__file__) )\n # path to the install version of resources.py.\n path_install = os.path.join(path_current, 'resources')\n # get the path to the local resources.py (on Linux machines it is in ~/.niftypet)\n path_resources = path_niftypet_local()\n print path_current\n\n # flag for the resources file if already installed (initially assumed not)\n flg_resources = False\n # does the local folder for niftypet exists? if not create one. \n if not os.path.exists(path_resources):\n os.makedirs(path_resources)\n # is resources.py in the folder?\n if not os.path.isfile(os.path.join(path_resources,'resources.py')):\n if os.path.isfile(os.path.join(path_install,'resources.py')):\n shutil.copyfile( os.path.join(path_install,'resources.py'), os.path.join(path_resources,'resources.py') )\n else:\n print 'e> could not fine file to be installed!'\n raise IOError('could not find should be already in the local NiftyPET folder.', path_resources\n # set the flag that the resources file is already there\n flg_resources = True\n sys.path.append(path_resources)\n try:\n import resources\n except ImportError as ie:\n print '----------------------------'\n print 'e> Import Error: NiftyPET''s resources file could not be imported. It should be in ''~/.niftypet/resources.py'' but likely it does not exists.'\n print '----------------------------'\n\n # find available GPU devices, select one or more and output the compilation flags\n gpuarch = dev_setup()\n\n # return gpuarch for cmake compilation\n return gpuarch", "response": "This function checks CUDA devices installs resources. py and installs it into the local NiftyPET folder."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef no_coroutine(f):\n\n @functools.wraps(f)\n def _no_coroutine(*args, **kwargs):\n generator = f(*args, **kwargs)\n\n # Special case for a function that returns immediately\n if not isinstance(generator, types.GeneratorType):\n return generator\n\n previous = None\n first = True\n\n while True:\n element = None\n\n try:\n if first:\n element = next(generator)\n else:\n element = generator.send(previous)\n except StopIteration as e:\n return getattr(e, \"value\", None)\n except ReturnValueException as e:\n return e.value\n else:\n previous = element\n first = False\n\n return _no_coroutine", "response": "A function decorator that returns a generator that returns the value of the next item in the sequence."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef maybe_coroutine(decide):\n\n def _maybe_coroutine(f):\n @functools.wraps(f)\n def __maybe_coroutine(*args, **kwargs):\n if decide(*args, **kwargs):\n return coroutine(f)(*args, **kwargs)\n else:\n return no_coroutine(f)(*args, **kwargs)\n return __maybe_coroutine\n return _maybe_coroutine", "response": "A function decorator that returns a generator that yields only Promises or whatever."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef makeCallbackPromise(function, *args, **kwargs):\n\n def _resolver(resolve, reject):\n function(\n lambda success, result: resolve(result) if success else reject(result),\n *args,\n **kwargs\n )\n\n return Promise(_resolver)", "response": "Create a Promise that resolves with the result of a function that reports its result using a callback and returns a Promise that resolves with the result."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef calculate_mass(composition, mass_data=None):\n mass = 0.0\n if mass_data is None:\n mass_data = nist_mass\n for element in composition:\n try:\n mass += (composition[element] * mass_data[element][0][0])\n except KeyError:\n match = re.search(r\"(\\S+)\\[(\\d+)\\]\", element)\n if match:\n element_ = match.group(1)\n isotope = int(match.group(2))\n mass += composition[element] * mass_data[element_][isotope][0]\n else:\n raise\n return mass", "response": "Calculates the monoisotopic mass of a chemical element."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse_formula(formula):\n if not formula_pattern.match(formula):\n raise ValueError(\"%r does not look like a formula\" % (formula,))\n composition = PyComposition()\n for elem, isotope, number in atom_pattern.findall(formula):\n composition[_make_isotope_string(elem, int(isotope) if isotope else 0)] += int(number)\n return composition", "response": "Parse a chemical formula and construct a : class:`PyComposition` object containing the chemical formula."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_devices(self, refresh=False):\n if refresh or self._devices is None:\n if self._devices is None:\n self._devices = {}\n\n _LOGGER.info(\"Updating all devices...\")\n response = self.send_request(\"get\", CONST.DEVICES_URL)\n response_object = json.loads(response.text)\n\n _LOGGER.debug(\"Get Devices Response: %s\", response.text)\n\n for device_json in response_object:\n # Attempt to reuse an existing device\n device = self._devices.get(device_json['id'])\n\n # No existing device, create a new one\n if device:\n device.update(device_json)\n else:\n device = SkybellDevice(device_json, self)\n self._devices[device.device_id] = device\n\n return list(self._devices.values())", "response": "Get all devices from Abode."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsends a request to Skybell.", "response": "def send_request(self, method, url, headers=None,\n json_data=None, retry=True):\n \"\"\"Send requests to Skybell.\"\"\"\n if not self.cache(CONST.ACCESS_TOKEN) and url != CONST.LOGIN_URL:\n self.login()\n\n if not headers:\n headers = {}\n\n if self.cache(CONST.ACCESS_TOKEN):\n headers['Authorization'] = 'Bearer ' + \\\n self.cache(CONST.ACCESS_TOKEN)\n\n headers['user-agent'] = (\n 'SkyBell/3.4.1 (iPhone9,2; iOS 11.0; loc=en_US; lang=en-US) '\n 'com.skybell.doorbell/1')\n headers['content-type'] = 'application/json'\n headers['accepts'] = '*/*'\n headers['x-skybell-app-id'] = self.cache(CONST.APP_ID)\n headers['x-skybell-client-id'] = self.cache(CONST.CLIENT_ID)\n\n _LOGGER.debug(\"HTTP %s %s Request with headers: %s\",\n method, url, headers)\n\n try:\n response = getattr(self._session, method)(\n url, headers=headers, json=json_data)\n _LOGGER.debug(\"%s %s\", response, response.text)\n\n if response and response.status_code < 400:\n return response\n except RequestException as exc:\n _LOGGER.warning(\"Skybell request exception: %s\", exc)\n\n if retry:\n self.login()\n\n return self.send_request(method, url, headers, json_data, False)\n\n raise SkybellException(ERROR.REQUEST, \"Retry failed\")"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update_cache(self, data):\n UTILS.update(self._cache, data)\n self._save_cache()", "response": "Update a cached value."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget a cached value for a device.", "response": "def dev_cache(self, device, key=None):\n \"\"\"Get a cached value for a device.\"\"\"\n device_cache = self._cache.get(CONST.DEVICES, {}).get(device.device_id)\n\n if device_cache and key:\n return device_cache.get(key)\n\n return device_cache"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef update_dev_cache(self, device, data):\n self.update_cache(\n {\n CONST.DEVICES: {\n device.device_id: data\n }\n })", "response": "Update cached values for a device."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _load_cache(self):\n if not self._disable_cache:\n if os.path.exists(self._cache_path):\n _LOGGER.debug(\"Cache found at: %s\", self._cache_path)\n if os.path.getsize(self._cache_path) > 0:\n loaded_cache = UTILS.load_cache(self._cache_path)\n UTILS.update(self._cache, loaded_cache)\n else:\n _LOGGER.debug(\"Cache file is empty. Removing it.\")\n os.remove(self._cache_path)\n\n self._save_cache()", "response": "Load existing cache and merge for updating if required."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef __encryptKeyTransportMessage(\n self,\n bare_jids,\n encryption_callback,\n bundles = None,\n expect_problems = None,\n ignore_trust = False\n ):\n \"\"\"\n bare_jids: iterable\n encryption_callback: A function which is called using an instance of cryptography.hazmat.primitives.ciphers.CipherContext, which you can use to encrypt any sort of data. You don't have to return anything.\n bundles: { [bare_jid: string] => { [device_id: int] => ExtendedPublicBundle } }\n expect_problems: { [bare_jid: string] => iterable }\n\n returns: {\n iv: bytes,\n sid: int,\n keys: {\n [bare_jid: string] => {\n [device: int] => {\n \"data\" : bytes,\n \"pre_key\" : boolean\n }\n }\n }\n }\n \"\"\"\n\n yield self.runInactiveDeviceCleanup()\n\n #########################\n # parameter preparation #\n #########################\n\n if isinstance(bare_jids, string_type):\n bare_jids = set([ bare_jids ])\n else:\n bare_jids = set(bare_jids)\n\n if bundles == None:\n bundles = {}\n\n if expect_problems == None:\n expect_problems = {}\n else:\n for bare_jid in expect_problems:\n expect_problems[bare_jid] = set(expect_problems[bare_jid])\n\n # Add the own bare jid to the set of jids\n bare_jids.add(self.__my_bare_jid)\n\n ########################################################\n # check all preconditions and prepare missing sessions #\n ########################################################\n\n problems = []\n\n # Prepare the lists of devices to encrypt for\n encrypt_for = {}\n\n for bare_jid in bare_jids:\n devices = yield self.__loadActiveDevices(bare_jid)\n\n if len(devices) == 0:\n problems.append(NoDevicesException(bare_jid))\n else:\n encrypt_for[bare_jid] = devices\n\n # Remove the sending devices from the list\n encrypt_for[self.__my_bare_jid].remove(self.__my_device_id)\n\n # Check whether all required bundles are available\n for bare_jid, devices in encrypt_for.items():\n missing_bundles = set()\n\n # Load all sessions\n sessions = yield self.__loadSessions(bare_jid, devices)\n\n for device in devices:\n session = sessions[device]\n\n if session == None:\n if not device in bundles.get(bare_jid, {}):\n missing_bundles.add(device)\n\n devices -= missing_bundles\n\n for device in missing_bundles:\n if not device in expect_problems.get(bare_jid, set()):\n problems.append(MissingBundleException(bare_jid, device))\n\n # Check for missing sessions and simulate the key exchange\n for bare_jid, devices in encrypt_for.items():\n key_exchange_problems = {}\n\n # Load all sessions\n sessions = yield self.__loadSessions(bare_jid, devices)\n\n for device in devices:\n session = sessions[device]\n\n # If no session exists, create a new session\n if session == None:\n # Get the required bundle\n bundle = bundles[bare_jid][device]\n\n try:\n # Build the session, discarding the result afterwards. This is\n # just to check that the key exchange works.\n self.__state.getSharedSecretActive(bundle)\n except x3dh.exceptions.KeyExchangeException as e:\n key_exchange_problems[device] = str(e)\n\n encrypt_for[bare_jid] -= set(key_exchange_problems.keys())\n\n for device, message in key_exchange_problems.items():\n if not device in expect_problems.get(bare_jid, set()):\n problems.append(KeyExchangeException(\n bare_jid,\n device,\n message\n ))\n\n if not ignore_trust:\n # Check the trust for each device\n for bare_jid, devices in encrypt_for.items():\n # Load all trust\n trusts = yield self.__loadTrusts(bare_jid, devices)\n\n # Load all sessions\n sessions = yield self.__loadSessions(bare_jid, devices)\n\n trust_problems = []\n\n for device in devices:\n trust = trusts[device]\n session = sessions[device]\n\n # Get the identity key of the recipient\n other_ik = (\n bundles[bare_jid][device].ik\n if session == None else\n session.ik\n )\n\n if trust == None:\n trust_problems.append((device, other_ik, \"undecided\"))\n elif not (trust[\"key\"] == other_ik and trust[\"trusted\"]):\n trust_problems.append((device, other_ik, \"untrusted\"))\n\n devices -= set(map(lambda x: x[0], trust_problems))\n\n for device, other_ik, problem_type in trust_problems:\n if not device in expect_problems.get(bare_jid, set()):\n problems.append(\n TrustException(bare_jid, device, other_ik, problem_type)\n )\n\n # Check for jids with no eligible devices\n for bare_jid, devices in list(encrypt_for.items()):\n # Skip this check for my own bare jid\n if bare_jid == self.__my_bare_jid:\n continue\n\n if len(devices) == 0:\n problems.append(NoEligibleDevicesException(bare_jid))\n del encrypt_for[bare_jid]\n\n # If there were and problems, raise an Exception with a list of those.\n if len(problems) > 0:\n raise EncryptionProblemsException(problems)\n\n ##############\n # encryption #\n ##############\n\n # Prepare AES-GCM key and IV\n aes_gcm_iv = os.urandom(16)\n aes_gcm_key = os.urandom(16)\n\n # Create the AES-GCM instance\n aes_gcm = Cipher(\n algorithms.AES(aes_gcm_key),\n modes.GCM(aes_gcm_iv),\n backend=default_backend()\n ).encryptor()\n\n # Encrypt the plain data\n encryption_callback(aes_gcm)\n\n # Store the tag\n aes_gcm_tag = aes_gcm.tag\n\n # {\n # [bare_jid: string] => {\n # [device: int] => {\n # \"data\" : bytes,\n # \"pre_key\" : boolean\n # }\n # }\n # }\n encrypted_keys = {}\n\n for bare_jid, devices in encrypt_for.items():\n encrypted_keys[bare_jid] = {}\n\n for device in devices:\n # Note whether this is a response to a PreKeyMessage\n if self.__state.hasBoundOTPK(bare_jid, device):\n self.__state.respondedTo(bare_jid, device)\n yield self._storage.storeState(self.__state.serialize())\n\n # Load the session\n session = yield self.__loadSession(bare_jid, device)\n\n # If no session exists, this will be a PreKeyMessage\n pre_key = session == None\n\n # Create a new session \n if pre_key:\n # Get the required bundle\n bundle = bundles[bare_jid][device]\n\n # Build the session\n session_and_init_data = self.__state.getSharedSecretActive(bundle)\n \n session = session_and_init_data[\"dr\"]\n session_init_data = session_and_init_data[\"to_other\"]\n\n # Encrypt the AES GCM key and tag\n encrypted_data = session.encryptMessage(aes_gcm_key + aes_gcm_tag)\n\n # Store the new/changed session\n yield self.__storeSession(bare_jid, device, session)\n\n # Serialize the data into a simple message format\n serialized = self.__backend.WireFormat.messageToWire(\n encrypted_data[\"ciphertext\"],\n encrypted_data[\"header\"],\n { \"DoubleRatchet\": encrypted_data[\"additional\"] }\n )\n\n # If it is a PreKeyMessage, apply an additional step to the serialization.\n if pre_key:\n serialized = self.__backend.WireFormat.preKeyMessageToWire(\n session_init_data,\n serialized,\n { \"DoubleRatchet\": encrypted_data[\"additional\"] }\n )\n\n # Add the final encrypted and serialized data.\n encrypted_keys[bare_jid][device] = {\n \"data\" : serialized,\n \"pre_key\" : pre_key\n }\n\n promise.returnValue({\n \"iv\" : aes_gcm_iv,\n \"sid\" : self.__my_device_id,\n \"keys\" : encrypted_keys\n })", "response": "This method is used to encrypt a message for a given set of bare jids."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef deleteInactiveDevicesByQuota(self, per_jid_max = 15, global_max = 0):\n\n if per_jid_max < 1 and global_max < 1:\n return\n\n if per_jid_max < 1:\n per_jid_max = None\n\n if global_max < 1:\n global_max = None\n\n bare_jids = yield self._storage.listJIDs()\n\n if not per_jid_max == None:\n for bare_jid in bare_jids:\n devices = yield self.__loadInactiveDevices(bare_jid)\n\n if len(devices) > per_jid_max:\n # This sorts the devices from smaller to bigger timestamp, which means\n # from old to young.\n devices = sorted(devices.items(), key = lambda device: device[1])\n\n # This gets the first (=oldest) n entries, so that only the\n # per_jid_max youngest entries are left.\n devices = devices[:-per_jid_max]\n\n # Get the device ids and discard the timestamps.\n devices = list(map(lambda device: device[0], devices))\n\n yield self.__deleteInactiveDevices(bare_jid, devices)\n \n if not global_max == None:\n all_inactive_devices = []\n\n for bare_jid in bare_jids:\n devices = yield self.__loadInactiveDevices(bare_jid)\n\n all_inactive_devices.extend(map(\n lambda device: (bare_jid, device[0], device[1]),\n devices.items()\n ))\n\n if len(all_inactive_devices) > global_max:\n # This sorts the devices from smaller to bigger timestamp, which means\n # from old to young.\n devices = sorted(all_inactive_devices, key = lambda device: device[2])\n\n # This gets the first (=oldest) n entries, so that only the global_max\n # youngest entries are left.\n devices = devices[:-global_max]\n\n # Get the list of devices to delete for each jid\n delete_devices = {}\n\n for device in devices:\n bare_jid = device[0]\n device_id = device[1]\n\n delete_devices[bare_jid] = delete_devices.get(bare_jid, [])\n delete_devices[bare_jid].append(device_id)\n\n # Now, delete the devices\n for bare_jid, devices in delete_devices.items():\n yield self.__deleteInactiveDevices(bare_jid, devices)", "response": "Delete inactive devices by setting a quota."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef deleteInactiveDevicesByAge(self, age_days):\n\n if age_days < 1:\n return\n\n now = time.time()\n\n bare_jids = yield self._storage.listJIDs()\n\n for bare_jid in bare_jids:\n devices = yield self.__loadInactiveDevices(bare_jid)\n\n delete_devices = []\n for device, timestamp in list(devices.items()):\n elapsed_s = now - timestamp\n elapsed_m = elapsed_s / 60\n elapsed_h = elapsed_m / 60\n elapsed_d = elapsed_h / 24\n\n if elapsed_d >= age_days:\n delete_devices.append(device)\n\n if len(delete_devices) > 0:\n yield self.__deleteInactiveDevices(bare_jid, delete_devices)", "response": "Delete all inactive devices from the device list storage and cache that are older than age_days days."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nrunning both the deleteInactiveDevicesByAge and the deleteInactiveDevicesByQuota methods with the configuration that was set when calling create.", "response": "def runInactiveDeviceCleanup(self):\n \"\"\"\n Runs both the deleteInactiveDevicesByAge and the deleteInactiveDevicesByQuota\n methods with the configuration that was set when calling create.\n \"\"\"\n\n yield self.deleteInactiveDevicesByQuota(\n self.__inactive_per_jid_max,\n self.__inactive_global_max\n )\n\n yield self.deleteInactiveDevicesByAge(self.__inactive_max_age)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef getTrustForJID(self, bare_jid):\n\n result = {\n \"active\" : {},\n \"inactive\" : {}\n }\n\n devices = yield self.__loadActiveDevices(bare_jid)\n\n for device in devices:\n result[\"active\"][device] = yield self.getTrustForDevice(bare_jid, device)\n\n devices = yield self.__loadInactiveDevices(bare_jid)\n\n for device in devices:\n result[\"inactive\"][device] = yield self.getTrustForDevice(bare_jid, device)\n\n promise.returnValue(result)", "response": "Get all - in - one trust information for all devices of a bare jid."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndeleting all data associated with a JID.", "response": "def deleteJID(self, bare_jid):\n \"\"\"\n Delete all data associated with a JID. This includes the list of active/inactive\n devices, all sessions with that JID and all information about trusted keys.\n \"\"\"\n\n yield self.runInactiveDeviceCleanup()\n\n self.__sessions_cache.pop(bare_jid, None)\n self.__devices_cache.pop(bare_jid, None)\n self.__trust_cache.pop(bare_jid, None)\n\n yield self._storage.deleteJID(bare_jid)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_setup(Cnt = {}):\n '''Return a dictionary of GPU, mu-map hardware and third party set-up.'''\n \n # the name of the folder for NiftyPET tools\n Cnt['DIRTOOLS'] = DIRTOOLS\n\n # additional paramteres for compiling tools with cmake\n Cnt['CMAKE_TLS_PAR'] = CMAKE_TLS_PAR\n\n # hardware mu-maps\n Cnt['HMULIST'] = hrdwr_mu\n \n # Microsoft Visual Studio Compiler version\n Cnt['MSVC_VRSN'] = MSVC_VRSN\n\n # GPU related setup\n Cnt = get_gpu_constants(Cnt)\n \n if 'PATHTOOLS' in globals() and PATHTOOLS!='': Cnt['PATHTOOLS'] = PATHTOOLS\n # image processing setup\n if 'RESPATH' in globals() and RESPATH!='': Cnt['RESPATH'] = RESPATH\n if 'REGPATH' in globals() and REGPATH!='': Cnt['REGPATH'] = REGPATH\n if 'DCM2NIIX' in globals() and DCM2NIIX!='': Cnt['DCM2NIIX'] = DCM2NIIX\n # hardware mu-maps\n if 'HMUDIR' in globals() and HMUDIR!='': Cnt['HMUDIR'] = HMUDIR\n if 'VINCIPATH' in globals() and VINCIPATH!='': Cnt['VINCIPATH'] = VINCIPATH\n\n Cnt['ENBLXNAT'] = ENBLXNAT\n Cnt['ENBLAGG'] = ENBLAGG\n Cnt['CMPL_DCM2NIIX'] = CMPL_DCM2NIIX\n\n return Cnt", "response": "Return a dictionary of GPU mu - map hardware and third - party set - up."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting all the constants together in a dictionary", "response": "def get_mmr_constants():\n '''\n Put all the constants together in a dictionary\n '''\n\n Cnt = {\n 'ISOTOPE':'F18',\n 'DCYCRR':DCYCRR,\n 'ALPHA':ALPHA,\n 'NRNG':NRNG,\n 'NSRNG':NSRNG,\n 'NCRS':NCRS,\n 'NCRSR':NCRSR,\n 'NBCKT':224,\n 'NSANGLES':A,\n 'NSBINS':W,\n 'Naw':-1, # number of total active bins per 2D sino\n 'NSN11': NSN11, # number of sinos in span-11\n 'NSN1': NSN1, # number of sinos in span-1\n 'NSN64': NSN64, # number of sinos in span-1 with no MRD limit\n 'MRD': MRD, # maximum ring difference RD\n 'SPN':SPAN, # span-1 (1), span-11 (11), ssrb (0)\n 'TFOV2':TFOV2, # squared radius of TFOV\n\n 'RNG_STRT':RNG_STRT, # limit axial extension by defining start and end ring\n 'RNG_END' :RNG_END, # this feature only works with span-1 processing (Cnt['SPN']=1)\n\n 'SS_IMZ':SS_IMZ, #Scatter mu-map iamge size \n 'SS_IMY':SS_IMY,\n 'SS_IMX':SS_IMX,\n 'SS_VXZ':SS_VXZ,\n 'SS_VXY':SS_VXY,\n 'IS_VXZ':IS_VXZ,\n 'SSE_IMZ':SSE_IMZ, #Scatter emission image size\n 'SSE_IMY':SSE_IMY,\n 'SSE_IMX':SSE_IMX,\n 'SSE_VXZ':SSE_VXZ,\n 'SSE_VXY':SSE_VXY,\n 'SZ_IMZ':SZ_IMZ, #GPU optimised image size\n 'SZ_IMY':SZ_IMY,\n 'SZ_IMX':SZ_IMX,\n 'SZ_VOXZ':SZ_VOXZ,\n 'SZ_VOXY':SZ_VOXY,\n 'SZ_VOXZi':SZ_VOXZi,\n 'SO_IMZ':SO_IMZ, #Original image size (from Siemens)\n 'SO_IMY':SO_IMY,\n 'SO_IMX':SO_IMX,\n 'SO_VXZ':SO_VXZ,\n 'SO_VXY':SO_VXY,\n 'SO_VXX':SO_VXX,\n 'NSEG0':SEG0,\n 'RE':RE, #effective ring radius\n 'R':R,\n 'SEG':seg,\n 'MNRD':minrd,\n 'MXRD':maxrd,\n 'SCTRNG':sct_irng,\n 'TGAP':TGAP,\n 'OFFGAP':OFFGAP,\n 'AXR':AXR,\n 'R02':R02, #squared electron radius\n 'LLD':LLD, #lower energy threashold\n 'E511':E511,\n 'ER':ER, #energy resolution\n 'COSUPSMX':COSUPSMX, #cosine of max allowed scatter angle\n 'NCOS':NCOS, #number of cos samples for LUT\n 'COSSTP':COSSTP, #cosine step\n 'ICOSSTP':ICOSSTP, #inverse of cosine step\n 'ETHRLD':ETHRLD, #intensity emission image threshold (used in scatter modelling)\n 'CLGHT':CLGHT, #speed of light [cm/s]\n 'CWND':CWND, #coincidence time window [ps]\n 'TOFBINN':TOFBINN, #number of TOF bins\n 'TOFBINS':TOFBINS, #TOF bin width [ps]\n 'TOFBIND':TOFBIND,\n 'ITOFBIND':ITOFBIND,\n # affine and image size for the reconstructed image, assuming the centre of voxels in mm\n 'AFFINE':np.array([ [-10*SO_VXX, 0., 0., 5.*SO_IMX*SO_VXX ], #+5.*SO_VXX\n [0., 10*SO_VXY, 0., -5.*SO_IMY*SO_VXY ], #+5.*SO_VXY\n [0., 0., 10*SO_VXZ, -5.*SO_IMZ*SO_VXZ ], #-5.*SO_VXZ\n [0., 0., 0., 1.]]),\n 'IMSIZE':np.array([SO_IMZ, SO_IMY, SO_IMX]),\n\n\n 'BTP':0, #1:non parametric bootstrap, 2: parametric bootstrap (recommended)\n 'BTPRT':1.0, # Ratio of bootstrapped/original events (enables downsampling)\n\n 'VERBOSE':False,\n\n 'SCTSCLEM':SCTSCLEM,\n 'SCTSCLMU':SCTSCLMU,\n }\n\n # get the setup for GPU and third party apps\n Cnt = get_setup(Cnt=Cnt)\n\n return Cnt"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef check_version(Cnt, chcklst=['RESPATH','REGPATH','DCM2NIIX','HMUDIR']):\n ''' Check version and existence of all third-party software and input data.\n Output a dictionary with bool type of the requested bits in 'chcklst'\n '''\n\n # at start, assume that nothing is present yet\n output = {}\n for itm in chcklst:\n output[itm] = False\n\n # niftyreg reg_resample first\n if 'RESPATH' in chcklst and 'RESPATH' in Cnt:\n try:\n proc = Popen([Cnt['RESPATH'], '--version'], stdout=PIPE)\n out = proc.stdout.read()\n if reg_ver in out:\n output['RESPATH'] = True\n except OSError:\n print 'e> NiftyReg (reg_resample) either is NOT installed or is corrupt.'\n \n # niftyreg reg_aladin\n if 'REGPATH' in chcklst and 'REGPATH' in Cnt:\n try:\n proc = Popen([Cnt['REGPATH'], '--version'], stdout=PIPE)\n out = proc.stdout.read()\n if reg_ver in out:\n output['REGPATH'] = True\n except OSError:\n print 'e> NiftyReg (reg_aladin) either is NOT installed or is corrupt.'\n\n # dcm2niix\n if 'DCM2NIIX' in chcklst and 'DCM2NIIX' in Cnt:\n try:\n proc = Popen([Cnt['DCM2NIIX'], '-h'], stdout=PIPE)\n out = proc.stdout.read()\n ver_str = re.search('(?<=dcm2niiX version v)\\d{1,2}.\\d{1,2}.\\d*', out)\n if ver_str and dcm_ver in ver_str.group(0):\n output['DCM2NIIX'] = True\n except OSError:\n print 'e> dcm2niix either is NOT installed or is corrupt.'\n\n # hdw mu-map list\n if 'HMUDIR' in chcklst and 'HMUDIR' in Cnt:\n for hi in Cnt['HMULIST']:\n if os.path.isfile(os.path.join(Cnt['HMUDIR'],hi)):\n output['HMUDIR'] = True\n else:\n output['HMUDIR'] = False\n break\n\n return output", "response": "Check version and existence of all third - party software and input data."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ninstalls the requested software from the git repo and check out the version given by sha1.", "response": "def install_tool(app, Cnt):\n ''' Install the requested software from the git 'repo'\n and check out the version given by 'sha1'.\n '''\n\n # get the current working directory\n cwd = os.getcwd()\n\n # pick the target installation folder for tools\n if 'PATHTOOLS' in Cnt and Cnt['PATHTOOLS']!='':\n path_tools = Cnt['PATHTOOLS']\n elif ('PATHTOOLS' not in Cnt or Cnt['PATHTOOLS']!=''):\n if 'DISPLAY' in os.environ and platform.system() in ['Linux', 'Windows']:\n print '>>>>> DISPLAY', os.environ['DISPLAY']\n Tk().withdraw()\n dircore = askdirectory(title='choose a place for NiftyPET tools', initialdir=os.path.expanduser('~'))\n Tk().destroy()\n # get the full (combined path)\n path_tools = os.path.join(dircore, Cnt['DIRTOOLS'])\n else:\n try:\n path_tools = input_path('Enter path for NiftyPET tools (registration, etc):')\n except:\n print 'enter the intended PATHTOOLS in resources.py located in ~/.niftypet/'\n raise ValueError('\\n e> could not get the path for NiftyPET_tools \\n')\n Cnt['PATHTOOLS'] = path_tools\n\n else:\n if platform.system() == 'Linux' :\n path_tools = os.path.join( os.path.expanduser('~'), Cnt['DIRTOOLS'] )\n elif platform.system() == 'Windows' :\n path_tools = os.path.join( os.getenv('LOCALAPPDATA'), Cnt['DIRTOOLS'] )\n else:\n print '\\n=========================================================='\n print 'e> only Linux and Windows operating systems are supported!'\n print '==========================================================\\n'\n raise SystemError('OS not supported!') \n Cnt['PATHTOOLS'] = path_tools\n\n #create the main tools folder\n if not os.path.isdir(path_tools):\n os.mkdir(path_tools)\n # identify the specific path for the requested app\n if app=='niftyreg':\n repo = repo_reg\n sha1 = sha1_reg\n path = os.path.join(path_tools, 'niftyreg')\n elif app=='dcm2niix':\n repo = repo_dcm\n sha1 = sha1_dcm\n path = os.path.join(path_tools, 'dcm2niix')\n\n if not Cnt['CMPL_DCM2NIIX']:\n # avoid installing from source, instead download the full version:\n Cnt = download_dcm2niix(Cnt, path)\n return Cnt\n\n # Check if the source folder exists and delete it, if it does\n if os.path.isdir(path): shutil.rmtree(path)\n # Create an empty folder and enter it\n os.mkdir(path)\n os.chdir(path)\n\n # clone the git repository\n call(['git', 'clone', repo, dirsrc])\n os.chdir(dirsrc)\n print 'i> checking out the specific git version of the software...'\n call(['git', 'checkout', sha1])\n os.chdir('../')\n\n # create the building folder\n if not os.path.isdir(dirbld):\n os.mkdir(dirbld)\n # go inside the build folder\n os.chdir(dirbld)\n\n # run cmake with arguments\n if platform.system()=='Windows':\n cmd = ['cmake', '../'+dirsrc,\n '-DBUILD_ALL_DEP=ON',\n '-DCMAKE_INSTALL_PREFIX='+path,\n '-G', Cnt['MSVC_VRSN']]\n call(cmd)\n call(['cmake', '--build', './', '--config', 'Release', '--target', 'install'])\n elif platform.system() in ['Linux', 'Darwin']:\n cmd = ['cmake', '../'+dirsrc,\n '-DBUILD_ALL_DEP=ON',\n '-DCMAKE_INSTALL_PREFIX='+path]\n if Cnt['CMAKE_TLS_PAR']!='': cmd.append(Cnt['CMAKE_TLS_PAR'])\n call(cmd)\n call(\n ['cmake', '--build', './',\n '--config', 'Release',\n '--target', 'install',\n '--','-j', str(ncpu)]\n )\n\n # restore the current working directory\n os.chdir(cwd)\n\n if app=='niftyreg':\n try:\n Cnt['RESPATH'] = glob.glob(os.path.join(os.path.join(path,'bin'), 'reg_resample*'))[0]\n Cnt['REGPATH'] = glob.glob(os.path.join(os.path.join(path,'bin'), 'reg_aladin*'))[0]\n except IndexError:\n print 'e> NiftyReg has NOT been successfully installed.'\n raise SystemError('Failed Installation (NiftyReg)')\n # updated the file resources.py\n Cnt = update_resources(Cnt)\n # check the installation:\n chck_niftyreg = check_version(Cnt, chcklst=['RESPATH','REGPATH'])\n if not all([chck_niftyreg[k] for k in chck_niftyreg.keys()]):\n print 'e> NiftyReg has NOT been successfully installed.'\n raise SystemError('Failed Installation (NiftyReg)')\n\n elif app=='dcm2niix':\n try:\n Cnt['DCM2NIIX'] = glob.glob(os.path.join(os.path.join(path,'bin'), 'dcm2niix*'))[0]\n except IndexError:\n print 'e> dcm2niix has NOT been successfully installed.'\n Cnt = download_dcm2niix(Cnt, path)\n # check the installation:\n if not check_version(Cnt, chcklst=['DCM2NIIX']):\n print 'e> dcm2niix has NOT been successfully compiled from github.'\n Cnt = download_dcm2niix(Cnt, path)\n return Cnt"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef update_resources(Cnt):\n '''Update resources.py with the paths to the new installed apps.\n '''\n\n # list of path names which will be saved\n key_list = ['PATHTOOLS', 'RESPATH', 'REGPATH', 'DCM2NIIX', 'HMUDIR']\n\n # get the local path to NiftyPET resources.py\n path_resources = cs.path_niftypet_local()\n resources_file = os.path.join(path_resources,'resources.py')\n\n # update resources.py\n if os.path.isfile(resources_file):\n f = open(resources_file, 'r')\n rsrc = f.read()\n f.close()\n # get the region of keeping in synch with Python\n i0 = rsrc.find('### start NiftyPET tools ###')\n i1 = rsrc.find('### end NiftyPET tools ###')\n pth_list = []\n for k in key_list:\n if k in Cnt:\n pth_list.append('\\'' + Cnt[k].replace(\"\\\\\",\"/\") + '\\'')\n else:\n pth_list.append('\\'\\'')\n\n # modify resources.py with the new paths\n strNew = '### start NiftyPET tools ###\\n'\n for i in range(len(key_list)):\n if pth_list[i] != '\\'\\'':\n strNew += key_list[i]+' = '+pth_list[i] + '\\n'\n rsrcNew = rsrc[:i0] + strNew + rsrc[i1:]\n f = open(resources_file, 'w')\n f.write(rsrcNew)\n f.close()\n\n return Cnt", "response": "Update resources. py with the paths to the new installed apps."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nloading the session for each device id.", "response": "def loadSessions(self, callback, bare_jid, device_ids):\n \"\"\"\n Return a dict containing the session for each device id. By default, this method\n calls loadSession for each device id.\n \"\"\"\n\n if self.is_async:\n self.__loadSessionsAsync(callback, bare_jid, device_ids, {})\n else:\n return self.__loadSessionsSync(bare_jid, device_ids)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nload trust status for each device id.", "response": "def loadTrusts(self, callback, bare_jid, device_ids):\n \"\"\"\n Return a dict containing the trust status for each device id. By default, this\n method calls loadTrust for each device id.\n \"\"\"\n\n if self.is_async:\n self.__loadTrustsAsync(callback, bare_jid, device_ids, {})\n else:\n return self.__loadTrustsSync(bare_jid, device_ids)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef parse(cls, backend, ik, spk, spk_signature, otpks):\n\n ik = backend.decodePublicKey(ik)[0]\n\n spk[\"key\"] = backend.decodePublicKey(spk[\"key\"])[0]\n\n otpks = list(map(lambda otpk: {\n \"key\" : backend.decodePublicKey(otpk[\"key\"])[0],\n \"id\" : otpk[\"id\"]\n }, otpks))\n\n return cls(ik, spk, spk_signature, otpks)", "response": "Create a new instance of the class from the data you ve retrieved from the PEP node."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nusing this method to prepare the data to be uploaded directly to some PEP node. This method applies an additional encoding step to the public keys in the bundle. The result is a dictionary with the keys ik, spk, spk_signature and otpks. The values are structured the same way as the inputs of the constructor.", "response": "def serialize(self, backend):\n \"\"\"\n Use this method to prepare the data to be uploaded directly to some PEP node. This\n method applies an additional encoding step to the public keys in the bundle. The\n result is a dictionary with the keys ik, spk, spk_signature and otpks. The values\n are structured the same way as the inputs of the constructor.\n \"\"\"\n\n return {\n \"ik\": backend.encodePublicKey(self.ik, \"25519\"),\n \"spk\": {\n \"id\" : self.spk[\"id\"],\n \"key\" : backend.encodePublicKey(self.spk[\"key\"], \"25519\"),\n },\n \"spk_signature\": self.spk_signature,\n \"otpks\": list(map(lambda otpk: {\n \"id\" : otpk[\"id\"],\n \"key\" : backend.encodePublicKey(otpk[\"key\"], \"25519\")\n }, self.otpks))\n }"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef imfill(immsk):\n '''fill the empty patches of image mask 'immsk' '''\n\n for iz in range(immsk.shape[0]):\n for iy in range(immsk.shape[1]):\n ix0 = np.argmax(immsk[iz,iy,:]>0)\n ix1 = immsk.shape[2] - np.argmax(immsk[iz,iy,::-1]>0)\n if (ix1-ix0) > immsk.shape[2]-10: continue\n immsk[iz,iy,ix0:ix1] = 1\n return immsk", "response": "fill the empty patches of image mask immsk"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate mask over the whole image or over the threshold area", "response": "def create_mask(\n fnii,\n fimout = '',\n outpath = '',\n fill = 1,\n dtype_fill = np.uint8,\n \n thrsh = 0.,\n fwhm = 0.,):\n\n ''' create mask over the whole image or over the threshold area'''\n \n\n #> output path\n if outpath=='' and fimout!='':\n opth = os.path.dirname(fimout)\n if opth=='':\n opth = os.path.dirname(fnii)\n fimout = os.path.join(opth, fimout)\n\n elif outpath=='':\n opth = os.path.dirname(fnii)\n\n else:\n opth = outpath\n\n #> output file name if not given\n if fimout=='':\n fniis = os.path.split(fnii)\n fimout = os.path.join(opth, fniis[1].split('.nii')[0]+'_mask.nii.gz')\n\n niidct = imio.getnii(fnii, output='all')\n im = niidct['im']\n hdr = niidct['hdr']\n\n if im.ndim>3:\n raise ValueError('The masking function only accepts 3-D images.')\n\n #> generate output image\n if thrsh>0.:\n smoim = ndi.filters.gaussian_filter(\n im,\n imio.fwhm2sig(fwhm, voxsize=abs(hdr['pixdim'][1])), \n mode='mirror')\n thrsh = thrsh*smoim.max()\n immsk = np.int8(smoim>thrsh)\n immsk = imfill(immsk)\n\n #> output image\n imo = fill * immsk.astype(dtype_fill)\n\n else:\n\n imo = fill * np.ones(im.shape, dtype = dtype_fill)\n\n #> save output image\n imio.array2nii( \n imo,\n niidct['affine'],\n fimout,\n trnsp = (niidct['transpose'].index(0),\n niidct['transpose'].index(1),\n niidct['transpose'].index(2)),\n flip = niidct['flip'])\n\n return {'fim':fimout, 'im':imo}"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dice_coeff(im1, im2, val=1):\n ''' Calculate Dice score for parcellation images and and ROI value .\n Input images can be given as:\n 1. paths to NIfTI image files or as\n 2. Numpy arrays.\n The ROI value can be given as:\n 1. a single integer representing one ROI out of many in the parcellation\n images ( and ) or as\n 2. a list of integers to form a composite ROI used for the association test.\n Outputs a float number representing the Dice score.\n '''\n\n if isinstance(im1, basestring) and isinstance(im2, basestring) \\\n and os.path.isfile(im1) and os.path.basename(im1).endswith(('nii', 'nii.gz')) \\\n and os.path.isfile(im2) and os.path.basename(im2).endswith(('nii', 'nii.gz')):\n imn1 = imio.getnii(im1, output='image')\n imn2 = imio.getnii(im2, output='image')\n elif isinstance(im1, (np.ndarray, np.generic)) and isinstance(im1, (np.ndarray, np.generic)):\n imn1 = im1\n imn2 = im2\n else:\n raise TypeError('Unrecognised or Mismatched Images.')\n\n # a single value corresponding to one ROI\n if isinstance(val, (int, long)):\n imv1 = (imn1 == val)\n imv2 = (imn2 == val)\n # multiple values in list corresponding to a composite ROI\n elif isinstance(val, list) and all([isinstance(v, (int, long)) for v in val]):\n imv1 = (imn1==val[0])\n imv2 = (imn2==val[0])\n for v in val[1:]:\n # boolean addition to form a composite ROI\n imv1 += (imn1==v)\n imv2 += (imn2==v)\n else:\n raise TypeError('ROI Values have to be integer (single or in a list).')\n \n\n if imv1.shape != imv2.shape:\n raise ValueError('Shape Mismatch: Input images must have the same shape.')\n\n #-compute Dice coefficient\n intrsctn = np.logical_and(imv1, imv2)\n\n return 2. * intrsctn.sum() / (imv1.sum() + imv2.sum())", "response": "Calculate the Dice coefficient for the parcellation images im1 and im2 and ROI value val."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncalculate the Dice coefficient for parcellation images im1 and im2 and ROI value roi2ind.", "response": "def dice_coeff_multiclass(im1, im2, roi2ind):\n ''' Calculate Dice score for parcellation images and and ROI value .\n Input images can be given as:\n 1. paths to NIfTI image files or as\n 2. Numpy arrays.\n The ROI value must be given as a dictionary of lists of indexes for each ROI\n Outputs a float number representing the Dice score.\n '''\n\n if isinstance(im1, basestring) and isinstance(im2, basestring) \\\n and os.path.isfile(im1) and os.path.basename(im1).endswith(('nii', 'nii.gz')) \\\n and os.path.isfile(im2) and os.path.basename(im2).endswith(('nii', 'nii.gz')):\n imn1 = imio.getnii(im1, output='image')\n imn2 = imio.getnii(im2, output='image')\n elif isinstance(im1, (np.ndarray, np.generic)) and isinstance(im1, (np.ndarray, np.generic)):\n imn1 = im1\n imn2 = im2\n else:\n raise TypeError('Unrecognised or Mismatched Images.')\n\n if imn1.shape != imn2.shape:\n raise ValueError('Shape Mismatch: Input images must have the same shape.')\n\n out = {}\n for k in roi2ind.keys():\n\n \t# multiple values in list corresponding to a composite ROI\n imv1 = (imn1==roi2ind[k][0])\n imv2 = (imn2==roi2ind[k][0])\n for v in roi2ind[k][1:]:\n # boolean addition to form a composite ROI\n imv1 += (imn1==v)\n imv2 += (imn2==v)\n\n\t #-compute Dice coefficient\n \tintrsctn = np.logical_and(imv1, imv2)\n \tout[k] = 2. * intrsctn.sum() / (imv1.sum() + imv2.sum())\n\n return out"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngenerates a new Skybellpy token.", "response": "def gen_token():\n \"\"\"Generate a new Skybellpy token.\"\"\"\n return ''.join(\n random.choice(\n string.ascii_uppercase + string.ascii_lowercase + string.digits)\n for _ in range(32))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _validate_setting(setting, value):\n if setting not in CONST.ALL_SETTINGS:\n raise SkybellException(ERROR.INVALID_SETTING, setting)\n\n if setting == CONST.SETTINGS_DO_NOT_DISTURB:\n if value not in CONST.SETTINGS_DO_NOT_DISTURB_VALUES:\n raise SkybellException(ERROR.INVALID_SETTING_VALUE,\n (setting, value))\n\n if setting == CONST.SETTINGS_OUTDOOR_CHIME:\n if value not in CONST.SETTINGS_OUTDOOR_CHIME_VALUES:\n raise SkybellException(ERROR.INVALID_SETTING_VALUE,\n (setting, value))\n\n if setting == CONST.SETTINGS_MOTION_POLICY:\n if value not in CONST.SETTINGS_MOTION_POLICY_VALUES:\n raise SkybellException(ERROR.INVALID_SETTING_VALUE,\n (setting, value))\n\n if setting == CONST.SETTINGS_MOTION_THRESHOLD:\n if value not in CONST.SETTINGS_MOTION_THRESHOLD_VALUES:\n raise SkybellException(ERROR.INVALID_SETTING_VALUE,\n (setting, value))\n\n if setting == CONST.SETTINGS_VIDEO_PROFILE:\n if value not in CONST.SETTINGS_VIDEO_PROFILE_VALUES:\n raise SkybellException(ERROR.INVALID_SETTING_VALUE,\n (setting, value))\n\n if setting in CONST.SETTINGS_LED_COLOR:\n if (value < CONST.SETTINGS_LED_VALUES[0] or\n value > CONST.SETTINGS_LED_VALUES[1]):\n raise SkybellException(ERROR.INVALID_SETTING_VALUE,\n (setting, value))\n\n if setting == CONST.SETTINGS_LED_INTENSITY:\n if not isinstance(value, int):\n raise SkybellException(ERROR.COLOR_INTENSITY_NOT_VALID, value)\n\n if (value < CONST.SETTINGS_LED_INTENSITY_VALUES[0] or\n value > CONST.SETTINGS_LED_INTENSITY_VALUES[1]):\n raise SkybellException(ERROR.INVALID_SETTING_VALUE,\n (setting, value))", "response": "Validate the setting and value."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef refresh(self):\n # Update core device data\n new_device_json = self._device_request()\n _LOGGER.debug(\"Device Refresh Response: %s\", new_device_json)\n\n # Update avatar url\n new_avatar_json = self._avatar_request()\n _LOGGER.debug(\"Avatar Refresh Response: %s\", new_avatar_json)\n\n # Update device detail info\n new_info_json = self._info_request()\n _LOGGER.debug(\"Device Info Refresh Response: %s\", new_info_json)\n\n # Update device setting details\n new_settings_json = self._settings_request()\n _LOGGER.debug(\"Device Settings Refresh Response: %s\",\n new_settings_json)\n\n # Update the stored data\n self.update(new_device_json, new_info_json, new_settings_json,\n new_avatar_json)\n\n # Update the activities\n self._update_activities()", "response": "Refresh the devices json object data."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef update(self, device_json=None, info_json=None, settings_json=None,\n avatar_json=None):\n \"\"\"Update the internal device json data.\"\"\"\n if device_json:\n UTILS.update(self._device_json, device_json)\n\n if avatar_json:\n UTILS.update(self._avatar_json, avatar_json)\n\n if info_json:\n UTILS.update(self._info_json, info_json)\n\n if settings_json:\n UTILS.update(self._settings_json, settings_json)", "response": "Update the internal device json data."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nupdates stored activities and update caches as required.", "response": "def _update_activities(self):\n \"\"\"Update stored activities and update caches as required.\"\"\"\n self._activities = self._activities_request()\n _LOGGER.debug(\"Device Activities Response: %s\", self._activities)\n\n if not self._activities:\n self._activities = []\n elif not isinstance(self._activities, (list, tuple)):\n self._activities = [self._activities]\n\n self._update_events()"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _update_events(self):\n events = self._skybell.dev_cache(self, CONST.EVENT) or {}\n\n for activity in self._activities:\n event = activity.get(CONST.EVENT)\n created_at = activity.get(CONST.CREATED_AT)\n\n old_event = events.get(event)\n\n if old_event and created_at < old_event.get(CONST.CREATED_AT):\n continue\n else:\n events[event] = activity\n\n self._skybell.update_dev_cache(\n self,\n {\n CONST.EVENT: events\n })", "response": "Update our cached list of latest activity events."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the requested number of activity information.", "response": "def activities(self, limit=1, event=None):\n \"\"\"Return device activity information.\"\"\"\n activities = self._activities or []\n\n # Filter our activity array if requested\n if event:\n activities = list(\n filter(\n lambda activity:\n activity[CONST.EVENT] == event, activities))\n\n # Return the requested number\n return activities[:limit]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nvalidate the settings and then send the PATCH request.", "response": "def _set_setting(self, settings):\n \"\"\"Validate the settings and then send the PATCH request.\"\"\"\n for key, value in settings.items():\n _validate_setting(key, value)\n\n try:\n self._settings_request(method=\"patch\", json_data=settings)\n\n self.update(settings_json=settings)\n except SkybellException as exc:\n _LOGGER.warning(\"Exception changing settings: %s\", settings)\n _LOGGER.warning(exc)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef location(self):\n location = self._device_json.get(CONST.LOCATION, {})\n\n return (location.get(CONST.LOCATION_LAT, 0),\n location.get(CONST.LOCATION_LNG, 0))", "response": "Return lat and lng tuple."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets the wifi status.", "response": "def wifi_status(self):\n \"\"\"Get the wifi status.\"\"\"\n return self._info_json.get(CONST.STATUS, {}).get(CONST.WIFI_LINK)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef do_not_disturb(self):\n return bool(strtobool(str(self._settings_json.get(\n CONST.SETTINGS_DO_NOT_DISTURB))))", "response": "Get if do not disturb is enabled."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsets do not disturb.", "response": "def do_not_disturb(self, enabled):\n \"\"\"Set do not disturb.\"\"\"\n self._set_setting(\n {\n CONST.SETTINGS_DO_NOT_DISTURB: str(enabled).lower()\n })"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset the motion sensor state.", "response": "def motion_sensor(self, enabled):\n \"\"\"Set the motion sensor state.\"\"\"\n if enabled is True:\n value = CONST.SETTINGS_MOTION_POLICY_ON\n elif enabled is False:\n value = CONST.SETTINGS_MOTION_POLICY_OFF\n else:\n raise SkybellException(ERROR.INVALID_SETTING_VALUE,\n (CONST.SETTINGS_MOTION_POLICY, enabled))\n\n self._set_setting({CONST.SETTINGS_MOTION_POLICY: value})"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget devices LED color.", "response": "def led_rgb(self):\n \"\"\"Get devices LED color.\"\"\"\n return (int(self._settings_json.get(CONST.SETTINGS_LED_R)),\n int(self._settings_json.get(CONST.SETTINGS_LED_G)),\n int(self._settings_json.get(CONST.SETTINGS_LED_B)))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef led_rgb(self, color):\n if (not isinstance(color, (list, tuple)) or\n not all(isinstance(item, int) for item in color)):\n raise SkybellException(ERROR.COLOR_VALUE_NOT_VALID, color)\n\n self._set_setting(\n {\n CONST.SETTINGS_LED_R: color[0],\n CONST.SETTINGS_LED_G: color[1],\n CONST.SETTINGS_LED_B: color[2]\n })", "response": "Set devices LED color."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconfigure remote access to the local repository", "response": "def remote(self, remote_base=None, username=None, password=None):\n \"\"\"\n Configures remote access\n\n Parameters\n ----------\n remote_base : str\n base URL path for remote repository\n username : str\n user name for remote repository\n password : str\n password for local repository\n \"\"\"\n if remote_base is not None:\n self.remote_base = remote_base\n self._remote = True\n self.set_auth(username=username, password=password)\n if self.auth.ready():\n passman = HTTPPasswordMgrWithDefaultRealm()\n passman.add_password(None, self.remote_base, self.auth.username, self.auth.password)\n authhandler = HTTPBasicAuthHandler(passman)\n opener = build_opener(authhandler)\n install_opener(opener)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns file name downloading if remote access configured.", "response": "def get(self, filetype, **kwargs):\n \"\"\"Returns file name, downloading if remote access configured.\n\n Parameters\n ----------\n filetype : str\n type of file\n\n keyword arguments :\n keywords to fully specify path\n\n Notes\n -----\n Path templates are defined in $DIMAGE_DIR/data/dimage_paths.ini\n \"\"\"\n\n path = self.full(filetype, **kwargs)\n\n if path:\n if self._remote:\n self.download_url_to_path(self.url(filetype, **kwargs), path)\n else:\n print(\"There is no file with filetype=%r to access in the tree module loaded\" % filetype)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef download_url_to_path(self, url, path, force=False):\n\n path_exists = isfile(path)\n if not path_exists or force:\n\n dir = dirname(path)\n if not exists(dir):\n if self.verbose:\n print(\"CREATE %s\" % dir)\n makedirs(dir)\n\n try:\n u = urlopen(url)\n except HTTPError as e:\n u = None\n print(\"HTTP error code %r. Please check you ~/.netrc has the correct authorization\" % e.code)\n\n if u:\n with open(path, 'wb') as file:\n meta = u.info()\n meta_func = meta.getheaders \\\n if hasattr(meta, 'getheaders') else meta.get_all\n meta_length = meta_func(\"Content-Length\")\n file_size = None\n if meta_length:\n file_size = int(meta_length[0])\n if self.verbose:\n print(\"Downloading: {0} Bytes: {1}\".format(url, file_size))\n\n file_size_dl = 0\n block_sz = 8192\n while True:\n buffer = u.read(block_sz)\n if not buffer:\n break\n file_size_dl += len(buffer)\n file.write(buffer)\n\n if self.verbose:\n if path_exists:\n print(\"OVERWRITING %s\" % path)\n else:\n print(\"CREATE %s\" % path)\n\n elif self.verbose:\n print(\"FOUND %s (already downloaded)\" % path)", "response": "Download a file from url to path and put it at path."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef reset(self):\n ''' Reset all streams '''\n\n # reset the main stream\n if self.stream:\n self.stream.reset()\n\n # reset the initial stream (otherwise old 'adds' remain in the new stream)\n if self.initial_stream:\n self.initial_stream.reset()", "response": "Reset all the streams and initial streams"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nadding a filepath into the list of tasks to download", "response": "def add(self, filetype, **kwargs):\n \"\"\" Adds a filepath into the list of tasks to download\"\"\"\n\n location = self.location(filetype, **kwargs)\n source = self.url(filetype, sasdir='sas' if not self.public else '', **kwargs)\n if 'full' not in kwargs:\n destination = self.full(filetype, **kwargs)\n else:\n destination = kwargs.get('full')\n\n if location and source and destination:\n self.initial_stream.append_task(location=location, source=source, destination=destination)\n else:\n print(\"There is no file with filetype=%r to access in the tree module loaded\" % filetype)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset the download streams", "response": "def set_stream(self):\n \"\"\" Sets the download streams \"\"\"\n\n if not self.auth:\n raise AccessError(\"Please use the remote() method to set rsync authorization or use remote(public=True) for public data\")\n elif not self.initial_stream.task:\n raise AccessError(\"No files to download.\")\n else:\n self.stream = self.get_stream()\n self.stream.source = join(self.remote_base, 'sas') if self.remote_base and not self.public else join(self.remote_base, self.release) if self.release else self.remote_base\n self.stream.destination = join(self.base_dir, self.release) if self.public and self.release else self.base_dir\n self.stream.cli.env = {'RSYNC_PASSWORD': self.auth.password} if self.auth.ready() else None\n if self.stream.source and self.stream.destination:\n for task in self.initial_stream.task:\n self.set_stream_task(task)\n ntask = len(self.stream.task)\n if self.stream.stream_count > ntask:\n if self.verbose:\n print(\"SDSS_ACCESS> Reducing the number of streams from %r to %r, the number of download tasks.\" % (self.stream.stream_count, ntask))\n self.stream.stream_count = ntask\n self.stream.streamlet = self.stream.streamlet[:ntask]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef commit(self, offset=None, limit=None, dryrun=False):\n\n self.stream.command = \"rsync -avRK --files-from={path} {source} {destination}\"\n self.stream.append_tasks_to_streamlets(offset=offset, limit=limit)\n self.stream.commit_streamlets()\n self.stream.run_streamlets()\n self.stream.reset_streamlet()", "response": "Commit the rsync download"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef balance(self) -> Decimal:\n return sum_queryset(AccountEntry.objects.filter(account=self.account, timestamp__lte=self.timestamp).exclude(timestamp=self.timestamp, id__gt=self.id))", "response": "Returns the balance of this entry after this entry."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_balance(self, t: datetime):\n return sum_queryset(self.accountentry_set.all().filter(timestamp__lt=t))", "response": "Returns the balance of all entries in the account before the specified datetime."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef needs_settling(self, e: AccountEntry) -> bool:\n return e.type and e.type.is_settlement and e.account.id == self.id and e.settled_invoice and AccountEntry.objects.filter(parent=e).count() == 0", "response": "Returns True if the account entry needs settling"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef receivables_account(self) -> Account:\n row = AccountEntry.objects.filter(source_invoice=self).order_by('id').first()\n return row.account if row else None", "response": "Returns the account that this invoice was recorded in."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_entries(self, acc: Account, cls=AccountEntry) -> QuerySet:\n return cls.objects.filter(Q(account=acc) & (Q(source_invoice=self) | Q(settled_invoice=self))) if acc else cls.objects.none()", "response": "Returns all entries related to this invoice on specified account."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn balances of items of the invoice.", "response": "def get_item_balances(self, acc: Account) -> list:\n \"\"\"\n Returns balances of items of the invoice.\n :param acc: Account\n :return: list (AccountEntry, Decimal) in item id order\n \"\"\"\n items = []\n entries = self.get_entries(acc)\n for item in entries.filter(source_invoice=self).order_by('id'):\n assert isinstance(item, AccountEntry)\n settlements = sum_queryset(entries.filter(settled_item=item))\n bal = item.amount + settlements if item.amount is not None else settlements\n items.append((item, bal))\n return items"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_unpaid_items(self, acc: Account) -> list:\n unpaid_items = []\n for item, bal in self.get_item_balances(acc):\n assert isinstance(item, AccountEntry)\n if self.type == INVOICE_DEFAULT:\n if bal > Decimal(0):\n unpaid_items.append((item.type.payback_priority, item, bal))\n elif self.type == INVOICE_CREDIT_NOTE:\n if bal < Decimal(0):\n unpaid_items.append((item.type.payback_priority, item, bal))\n else:\n raise Exception('jacc.models.Invoice.get_unpaid_items() unimplemented for invoice type {}'.format(self.type))\n return [i[1:] for i in sorted(unpaid_items, key=lambda x: x[0])]", "response": "Returns the list of unpaid items of the invoice in payback priority order."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nturns a list like 1 2 3 7 8 9 into a range list", "response": "def MakeRanges(codes):\n \"\"\"Turn a list like [1,2,3,7,8,9] into a range list [[1,3], [7,9]]\"\"\"\n ranges = []\n last = -100\n for c in codes:\n if c == last+1:\n ranges[-1][1] = c\n else:\n ranges.append([c, c])\n last = c\n return ranges"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nprint the ranges as an array of type named name.", "response": "def PrintRanges(type, name, ranges):\n \"\"\"Print the ranges as an array of type named name.\"\"\"\n print \"static const %s %s[] = {\" % (type, name,)\n for lo, hi in ranges:\n print \"\\t{ %d, %d },\" % (lo, hi)\n print \"};\""} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef PrintGroup(name, codes):\n\n # See unicode_groups.h for a description of the data structure.\n\n # Split codes into 16-bit ranges and 32-bit ranges.\n range16 = MakeRanges([c for c in codes if c < 65536])\n range32 = MakeRanges([c for c in codes if c >= 65536])\n\n # Pull singleton ranges out of range16.\n # code16 = [lo for lo, hi in range16 if lo == hi]\n # range16 = [[lo, hi] for lo, hi in range16 if lo != hi]\n\n global n16\n global n32\n n16 += len(range16)\n n32 += len(range32)\n\n ugroup = \"{ \\\"%s\\\", +1\" % (name,)\n # if len(code16) > 0:\n # PrintCodes(\"uint16\", name+\"_code16\", code16)\n # ugroup += \", %s_code16, %d\" % (name, len(code16))\n # else:\n # ugroup += \", 0, 0\"\n if len(range16) > 0:\n PrintRanges(\"URange16\", name+\"_range16\", range16)\n ugroup += \", %s_range16, %d\" % (name, len(range16))\n else:\n ugroup += \", 0, 0\"\n if len(range32) > 0:\n PrintRanges(\"URange32\", name+\"_range32\", range32)\n ugroup += \", %s_range32, %d\" % (name, len(range32))\n else:\n ugroup += \", 0, 0\"\n ugroup += \" }\"\n return ugroup", "response": "Print the data structures for the group of codes."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of unpaid items in the invoice which are assigned to the given account.", "response": "def settle_assigned_invoice(receivables_account: Account, settlement: AccountEntry, cls, **kwargs) -> list:\n \"\"\"\n Finds unpaid items in the invoice and generates entries to receivables account.\n Settlement is matched to invoice items based on entry types payback order.\n Generated payment entries have 'parent' field pointing to settlement, so that\n if settlement is (ever) deleted the payment entries will get deleted as well.\n In case of overpayment method generates entry to receivables account without matching invoice settled_item\n (only matching settled_invoice).\n :param receivables_account: Account which receives settled entries of the invoice\n :param settlement: Settlement to target to unpaid invoice items\n :param cls: Class for generated account entries, e.g. AccountEntry\n :param kwargs: Extra attributes for created for generated account entries\n :return: list (generated receivables account entries)\n \"\"\"\n invoice = settlement.settled_invoice\n if settlement.amount < Decimal(0) and settlement.settled_invoice.type != INVOICE_CREDIT_NOTE:\n raise ValidationError('Cannot target negative settlement {} to invoice {}'.format(settlement, settlement.settled_invoice))\n if settlement.amount > Decimal(0) and settlement.settled_invoice.type == INVOICE_CREDIT_NOTE:\n raise ValidationError('Cannot target positive settlement {} to credit note {}'.format(settlement, settlement.settled_invoice))\n if not invoice:\n raise ValidationError('Cannot target settlement {} without settled invoice'.format(settlement))\n assert isinstance(invoice, Invoice)\n if not settlement.type.is_settlement:\n raise ValidationError('Cannot settle account entry {} which is not settlement'.format(settlement))\n if AccountEntry.objects.filter(parent=settlement, account=receivables_account).count() > 0:\n raise ValidationError('Settlement is parent of old account entries already, maybe duplicate settlement?')\n if not receivables_account:\n raise ValidationError('Receivables account missing. Invoice with no rows?')\n\n new_payments = []\n remaining = Decimal(settlement.amount)\n invoice = settlement.settled_invoice\n timestamp = kwargs.pop('timestamp', settlement.timestamp)\n assert isinstance(invoice, Invoice)\n for item, bal in invoice.get_unpaid_items(receivables_account):\n if invoice.type == INVOICE_DEFAULT:\n if bal > Decimal(0):\n amt = min(remaining, bal)\n ae = cls.objects.create(account=receivables_account, amount=-amt, type=item.type, settled_item=item, settled_invoice=invoice, timestamp=timestamp, description=settlement.description, parent=settlement, **kwargs)\n new_payments.append(ae)\n remaining -= amt\n if remaining <= Decimal(0):\n break\n elif invoice.type == INVOICE_CREDIT_NOTE:\n if bal < Decimal(0):\n amt = max(remaining, bal)\n ae = cls.objects.create(account=receivables_account, amount=-amt, type=item.type, settled_item=item, settled_invoice=invoice, timestamp=timestamp, description=settlement.description, parent=settlement, **kwargs)\n new_payments.append(ae)\n remaining -= amt\n if remaining >= Decimal(0):\n break\n else:\n raise Exception('jacc.settle.settle_assigned_invoice() unimplemented for invoice type {}'.format(invoice.type))\n\n invoice.update_cached_fields()\n return new_payments"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a list of payments for the given credit note.", "response": "def settle_credit_note(credit_note: Invoice, debit_note: Invoice, cls, account: Account, **kwargs) -> list:\n \"\"\"\n Settles credit note. Records settling account entries for both original invoice and the credit note\n (negative entries for the credit note).\n Default timestamp for account entries is 'created' time of the credit note, can be overriden by kwargs.\n :param credit_note: Credit note to settle\n :param debit_note: Invoice to settle\n :param cls: AccountEntry (derived) class to use for new entries\n :param account: Settlement account\n :param kwargs: Variable arguments to cls() instance creation\n :return: list of new payments\n \"\"\"\n assert isinstance(credit_note, Invoice)\n assert credit_note.type == INVOICE_CREDIT_NOTE\n assert debit_note\n assert debit_note.type == INVOICE_DEFAULT\n\n credit = -credit_note.get_unpaid_amount()\n balance = debit_note.get_unpaid_amount()\n\n amt = min(balance, credit)\n amount = kwargs.pop('amount', None)\n if amount is not None:\n if amount > amt:\n raise ValidationError(_('Cannot settle credit note amount which is larger than remaining unpaid balance'))\n amt = amount\n\n entry_type = kwargs.pop('entry_type', None)\n if entry_type is None:\n if not hasattr(settings, 'E_CREDIT_NOTE_RECONCILIATION'):\n raise Exception('settle_credit_note() requires settings.E_CREDIT_NOTE_RECONCILIATION (account entry type code) or entry_type to be pass in kwargs')\n entry_type = EntryType.objects.get(code=settings.E_CREDIT_NOTE_RECONCILIATION)\n description = kwargs.pop('description', _('credit.note.reconciliation'))\n\n pmts = []\n if amt > Decimal(0):\n timestamp = kwargs.pop('timestamp', credit_note.created or now())\n # record entry to debit note settlement account\n pmt1 = cls.objects.create(account=account, amount=amt, type=entry_type, settled_invoice=debit_note, description=description + ' #{}'.format(credit_note.number), timestamp=timestamp, **kwargs)\n pmts.append(pmt1)\n # record entry to credit note settlement account\n pmt2 = cls.objects.create(account=account, parent=pmt1, amount=-amt, type=entry_type, settled_invoice=credit_note, description=description + ' #{}'.format(debit_note.number), timestamp=timestamp, **kwargs)\n pmts.append(pmt2)\n\n return pmts"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _input_templates(self):\n foo = self._config.read([self._pathfile])\n if len(foo) == 1:\n for k, v in self._config.items('paths'):\n self.templates[k] = v\n else:\n raise ValueError(\"Could not read {0}!\".format(self._pathfile))\n return", "response": "Read the path template file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef lookup_keys(self, name):\n ''' Lookup the keyword arguments needed for a given path name\n\n Parameters:\n name (str):\n The name of the path\n\n Returns:\n A list of keywords needed for filepath generation\n\n '''\n\n assert name, 'Must specify a path name'\n assert name in self.templates.keys(), '{0} must be defined in the path templates'.format(name)\n # find all words inside brackets\n keys = list(set(re.findall(r'{(.*?)}', self.templates[name])))\n # lookup any keys referenced inside special functions\n skeys = self._check_special_kwargs(name)\n keys.extend(skeys)\n # remove any duplicates\n keys = list(set(keys))\n # remove the type : descriptor\n keys = [k.split(':')[0] for k in keys]\n return keys", "response": "Lookup the keyword arguments needed for a given path name."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck special functions for kwargs", "response": "def _check_special_kwargs(self, name):\n ''' check special functions for kwargs\n \n Checks the content of the special functions (%methodname) for\n any keyword arguments referenced within\n\n Parameters:\n name (str):\n A path key name\n\n Returns:\n A list of keyword arguments found in any special functions \n '''\n keys = []\n # find any %method names in the template string\n functions = re.findall(r\"\\%\\w+\", self.templates[name])\n if not functions:\n return keys\n\n # loop over special method names and extract keywords\n for function in functions:\n method = getattr(self, function[1:])\n # get source code of special method\n source = self._find_source(method)\n fkeys = re.findall(r'kwargs\\[(.*?)\\]', source)\n if fkeys:\n # evaluate to proper string\n fkeys = [ast.literal_eval(k) for k in fkeys]\n keys.extend(fkeys)\n return keys"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfind source code of a given method in a module", "response": "def _find_source(method):\n ''' find source code of a given method\n \n Find and extract the source code of a given method in a module.\n Uses inspect.findsource to get all source code and performs some\n selection magic to identify method source code. Doing it this way\n because inspect.getsource returns wrong method.\n\n Parameters:\n method (obj):\n A method object\n \n Returns:\n A string containing the source code of a given method\n \n Example:\n >>> from sdss_access.path import Path\n >>> path = Path()\n >>> path._find_source(path.full)\n '''\n \n # get source code lines of entire module method is in\n source = inspect.findsource(method)\n is_method = inspect.ismethod(method)\n # create single source code string\n source_str = '\\n'.join(source[0])\n # define search pattern\n if is_method:\n pattern = r'def\\s{0}\\(self'.format(method.__name__)\n # search for pattern within the string\n start = re.search(pattern, source_str)\n if start:\n # find start and end positions of source code\n startpos = start.start()\n endpos = source_str.find('def ', startpos + 1)\n code = source_str[startpos:endpos]\n else:\n code = None\n return code"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef extract(self, name, example):\n ''' Extract keywords from an example path '''\n\n # if pathlib not available do nothing\n if not pathlib:\n return None\n\n # ensure example is a string\n if isinstance(example, pathlib.Path):\n example = str(example)\n assert isinstance(example, six.string_types), 'example file must be a string'\n\n # get the template\n assert name in self.lookup_names(), '{0} must be a valid template name'.format(name)\n template = self.templates[name]\n\n # expand the environment variable\n template = os.path.expandvars(template)\n\n # handle special functions; perform a drop in replacement\n if re.match('%spectrodir', template):\n template = re.sub('%spectrodir', os.environ['BOSS_SPECTRO_REDUX'], template)\n elif re.search('%platedir', template):\n template = re.sub('%platedir', '(.*)/{plateid:0>6}', template)\n elif re.search('%definitiondir', template):\n template = re.sub('%definitiondir', '{designid:0>6}', template)\n if re.search('%plateid6', template):\n template = re.sub('%plateid6', '{plateid:0>6}', template)\n\n # check if template has any brackets\n haskwargs = re.search('[{}]', template)\n if not haskwargs:\n return None\n\n # escape the envvar $ and any dots\n subtemp = template.replace('$', '\\\\$').replace('.', '\\\\.')\n # define search pattern; replace all template keywords with regex \"(.*)\" group\n research = re.sub('{(.*?)}', '(.*)', subtemp)\n # look for matches in template and example\n pmatch = re.search(research, template)\n tmatch = re.search(research, example)\n\n path_dict = {}\n # if example match extract keys and values from the match groups\n if tmatch:\n values = tmatch.groups(0)\n keys = pmatch.groups(0)\n assert len(keys) == len(values), 'pattern and template matches must have same length'\n parts = zip(keys, values)\n # parse into dictionary\n for part in parts:\n value = part[1]\n if re.findall('{(.*?)}', part[0]):\n # get the key name inside the brackets\n keys = re.findall('{(.*?)}', part[0])\n # remove the type : designation\n keys = [k.split(':')[0] for k in keys]\n # handle double bracket edge cases; remove this when better solution found\n if len(keys) > 1:\n if keys[0] == 'dr':\n # for {dr}{version}\n drval = re.match('^DR[1-9][0-9]', value).group(0)\n otherval = value.split(drval)[-1]\n pdict = {keys[0]: drval, keys[1]: otherval}\n elif keys[0] in ['rc', 'br', 'filter', 'camrow']:\n # for {camrow}{camcol}, {filter}{camcol}, {br}{id}, etc\n pdict = {keys[0]: value[0], keys[1]: value[1:]}\n else:\n raise ValueError('This case has not yet been accounted for.')\n path_dict.update(pdict)\n else:\n path_dict[keys[0]] = value\n return path_dict", "response": "Extract keywords from an example path"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the directory containing a file of a given type.", "response": "def dir(self, filetype, **kwargs):\n \"\"\"Return the directory containing a file of a given type.\n\n Parameters\n ----------\n filetype : str\n File type parameter.\n\n Returns\n -------\n dir : str\n Directory containing the file.\n \"\"\"\n\n full = kwargs.get('full', None)\n if not full:\n full = self.full(filetype, **kwargs)\n\n return os.path.dirname(full)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef name(self, filetype, **kwargs):\n\n full = kwargs.get('full', None)\n if not full:\n full = self.full(filetype, **kwargs)\n\n return os.path.basename(full)", "response": "Return the name of a file of a given type."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchecks if the given type of file exists locally.", "response": "def exists(self, filetype, remote=None, **kwargs):\n '''Checks if the given type of file exists locally\n\n Parameters\n ----------\n filetype : str\n File type parameter.\n\n remote : bool\n If True, checks for remote existence of the file\n\n Returns\n -------\n exists : bool\n Boolean indicating if the file exists.\n\n '''\n\n full = kwargs.get('full', None)\n if not full:\n full = self.full(filetype, **kwargs)\n\n if remote:\n # check for remote existence using a HEAD request\n url = self.url('', full=full)\n try:\n resp = requests.head(url)\n except Exception as e:\n raise AccessError('Cannot check for remote file existence for {0}: {1}'.format(url, e))\n else:\n return resp.ok\n else:\n return os.path.isfile(full)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nexpand a wildcard path locally and return a list of SAS urls or a list of files by the given type.", "response": "def expand(self, filetype, **kwargs):\n ''' Expand a wildcard path locally\n\n Parameters\n ----------\n filetype : str\n File type parameter.\n\n as_url: bool\n Boolean to return SAS urls\n\n refine: str\n Regular expression string to filter the list of files by\n before random selection\n\n Returns\n -------\n expand : list\n List of expanded full paths of the given type.\n\n '''\n\n full = kwargs.get('full', None)\n if not full:\n full = self.full(filetype, **kwargs)\n\n # assert '*' in full, 'Wildcard must be present in full path'\n files = glob(full)\n\n # return as urls?\n as_url = kwargs.get('as_url', None)\n newfiles = [self.url('', full=full) for full in files] if as_url else files\n\n # optionally refine the results\n refine = kwargs.get('refine', None)\n if refine:\n newfiles = self.refine(newfiles, refine, **kwargs)\n\n return newfiles"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if the local directory contains any of the type of file", "response": "def any(self, filetype, **kwargs):\n ''' Checks if the local directory contains any of the type of file\n\n Parameters\n ----------\n filetype : str\n File type parameter.\n\n Returns\n -------\n any : bool\n Boolean indicating if the any files exist in the expanded path on disk.\n\n '''\n expanded_files = self.expand(filetype, **kwargs)\n return any(expanded_files)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a random one of the given type of file", "response": "def one(self, filetype, **kwargs):\n ''' Returns random one of the given type of file\n\n Parameters\n ----------\n filetype : str\n File type parameter.\n\n as_url: bool\n Boolean to return SAS urls\n\n refine: str\n Regular expression string to filter the list of files by\n before random selection\n\n Returns\n -------\n one : str\n Random file selected from the expanded list of full paths on disk.\n\n '''\n expanded_files = self.expand(filetype, **kwargs)\n isany = self.any(filetype, **kwargs)\n return choice(expanded_files) if isany else None"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a random number of the given type of file.", "response": "def random(self, filetype, **kwargs):\n ''' Returns random number of the given type of file\n\n Parameters\n ----------\n filetype : str\n File type parameter.\n\n num : int\n The number of files to return\n\n as_url: bool\n Boolean to return SAS urls\n\n refine: str\n Regular expression string to filter the list of files by\n before random selection\n\n Returns\n -------\n random : list\n Random file selected from the expanded list of full paths on disk.\n\n '''\n expanded_files = self.expand(filetype, **kwargs)\n isany = self.any(filetype, **kwargs)\n if isany:\n # get the desired number\n num = kwargs.get('num', 1)\n assert num <= len(expanded_files), 'Requested number must be larger the sample. Reduce your number.'\n return sample(expanded_files, num)\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a list of files that satisfy a regular expression.", "response": "def refine(self, filelist, regex, filterdir='out', **kwargs):\n ''' Returns a list of files filterd by a regular expression\n\n Parameters\n ----------\n filelist : list\n A list of files to filter on.\n\n regex : str\n The regular expression string to filter your list\n\n filterdir: {'in', 'out'}\n Indicates the filter to be inclusive or exclusive\n 'out' removes the items satisfying the regular expression\n 'in' keeps the items satisfying the regular expression\n\n Returns\n -------\n refine : list\n A file list refined by an input regular expression.\n\n '''\n assert filelist, 'Must provide a list of filenames to refine on'\n assert regex, 'Must provide a regular expression to refine the file list'\n r = re.compile(regex)\n\n # icheck filter direction; default is out\n assert filterdir in ['in', 'out'], 'Filter direction must be either \"in\" or \"out\"'\n if filterdir == 'out':\n subset = list(filter(lambda i: r.search(i), filelist))\n elif filterdir == 'in':\n subset = list(filter(lambda i: not r.search(i), filelist))\n return subset"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the full name of a given type of file.", "response": "def full(self, filetype, **kwargs):\n \"\"\"Return the full name of a given type of file.\n\n Parameters\n ----------\n filetype : str\n File type parameter.\n\n Returns\n -------\n full : str\n The full path to the file.\n \"\"\"\n\n # check if full already in kwargs\n if 'full' in kwargs:\n return kwargs.get('full')\n\n # check for filetype in template\n assert filetype in self.templates, ('No entry {0} found. Filetype must '\n 'be one of the designated templates '\n 'in the currently loaded tree'.format(filetype))\n template = self.templates[filetype]\n\n # Now replace {} items\n if template:\n # check for missing keyword arguments\n keys = self.lookup_keys(filetype)\n # split keys to remove :format from any \"key:format\"\n keys = [k.split(':')[0] for k in keys]\n missing_keys = set(keys) - set(kwargs.keys())\n if missing_keys:\n raise KeyError('Missing required keyword arguments: {0}'.format(list(missing_keys)))\n else:\n template = template.format(**kwargs)\n\n if template:\n # Now replace environmental variables\n template = os.path.expandvars(template)\n\n # Now call special functions as appropriate\n template = self._call_special_functions(filetype, template, **kwargs)\n\n return template"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _call_special_functions(self, filetype, template, **kwargs):\n ''' Call the special functions found in a template path\n\n Calls special functions indicated by %methodname found in the\n sdss_paths.ini template file, and replaces the %location in the path\n with the returned content.\n\n Parameters:\n filetype (str):\n template name of file\n template (str):\n the template path\n kwargs (dict):\n Any kwargs needed to pass into the methods\n\n Returns:\n The expanded template path \n '''\n # Now call special functions as appropriate\n functions = re.findall(r\"\\%\\w+\", template)\n if not functions:\n return template\n\n for function in functions:\n try:\n method = getattr(self, function[1:])\n except AttributeError:\n return None\n else:\n value = method(filetype, **kwargs)\n template = re.sub(function, value, template)\n return template", "response": "Calls the special functions found in the path \n and returns the expanded template path \n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the location of the relative SAS path of a given type of file.", "response": "def location(self, filetype, base_dir=None, **kwargs):\n \"\"\"Return the location of the relative sas path of a given type of file.\n\n Parameters\n ----------\n filetype : str\n File type parameter.\n\n Returns\n -------\n full : str\n The relative sas path to the file.\n \"\"\"\n\n full = kwargs.get('full', None)\n if not full:\n full = self.full(filetype, **kwargs)\n\n self.set_base_dir(base_dir=base_dir)\n location = full[len(self.base_dir):] if full and full.startswith(self.base_dir) else None\n\n if location and '//' in location:\n location = location.replace('//', '/')\n\n return location"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef url(self, filetype, base_dir=None, sasdir='sas', **kwargs):\n\n location = self.location(filetype, **kwargs)\n return join(self.remote_base, sasdir, location) if self.remote_base and location else None", "response": "Return the url of a given type of file."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef plateid6(self, filetype, **kwargs):\n plateid = int(kwargs['plateid'])\n if plateid < 10000:\n return \"{:0>6d}\".format(plateid)\n else:\n return \"{:d}\".format(plateid)", "response": "Print plate ID accounting for 5 - 6 digit plate IDs."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the platedir of the current object.", "response": "def platedir(self, filetype, **kwargs):\n \"\"\"Returns plate subdirectory in :envvar:`PLATELIST_DIR` of the form: ``NNNNXX/NNNNNN``.\n\n Parameters\n ----------\n filetype : str\n File type parameter.\n plateid : int or str\n Plate ID number. Will be converted to int internally.\n\n Returns\n -------\n platedir : str\n Plate directory in the format ``NNNNXX/NNNNNN``.\n \"\"\"\n plateid = int(kwargs['plateid'])\n plateid100 = plateid // 100\n subdir = \"{:0>4d}\".format(plateid100) + \"XX\"\n return os.path.join(subdir, \"{:0>6d}\".format(plateid))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the value of SPECTRO_REDUX or BOSS_SPECTRO_REDUX depending on the value of run2d.", "response": "def spectrodir(self, filetype, **kwargs):\n \"\"\"Returns :envvar:`SPECTRO_REDUX` or :envvar:`BOSS_SPECTRO_REDUX`\n depending on the value of `run2d`.\n\n Parameters\n ----------\n filetype : str\n File type parameter.\n run2d : int or str\n 2D Reduction ID.\n\n Returns\n -------\n spectrodir : str\n Value of the appropriate environment variable.\n \"\"\"\n if str(kwargs['run2d']) in ('26', '103', '104'):\n return os.environ['SPECTRO_REDUX']\n else:\n return os.environ['BOSS_SPECTRO_REDUX']"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef definitiondir(self, filetype, **kwargs):\n\n designid = int(kwargs['designid'])\n designid100 = designid // 100\n subdir = \"{:0>4d}\".format(designid100) + \"XX\"\n return subdir", "response": "Returns the definition subdirectory in the form NNNNNXX."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef foreground_run(self, command, test=False, logger=None, logall=False, message=None, outname=None, errname=None):\n if logger is not None:\n logger.debug(command)\n status = 0\n out = ''\n err = ''\n if not test:\n if outname is None:\n outfile = TemporaryFile()\n else:\n outfile = open(outname, 'w+')\n if errname is None:\n errfile = TemporaryFile()\n else:\n errfile = open(errname, 'w+')\n proc = Popen(split(str(command)), stdout=outfile, stderr=errfile, env=self.env)\n tstart = time()\n while proc.poll() is None:\n elapsed = time() - tstart\n if elapsed > 500000:\n message = \"Process still running after more than 5 days!\"\n proc.kill()\n break\n tsleep = 10**(int(log10(elapsed)) - 1)\n if tsleep < 1:\n tsleep = 1\n sleep(tsleep)\n # proc.wait()\n status = proc.returncode\n outfile.seek(0)\n out = outfile.read()\n errfile.seek(0)\n err = errfile.read()\n outfile.close()\n errfile.close()\n if logger is not None:\n if status == 0 and logall:\n if len(out) > 0:\n logger.debug('STDOUT = \\n' + out)\n if len(err) > 0:\n logger.debug('STDERR = \\n' + err)\n if status != 0:\n logger.error('status = {0}'.format(status))\n if len(out) > 0:\n logger.error('STDOUT = \\n' + out)\n if len(err) > 0:\n logger.error('STDERR = \\n' + err)\n if message is not None:\n logger.critical(message)\n exit(status)\n return (status, out, err)", "response": "A convenient wrapper to log and perform system calls."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate edit or delete a change list", "response": "def change(ui, repo, *pats, **opts):\n\t\"\"\"create, edit or delete a change list\n\n\tCreate, edit or delete a change list.\n\tA change list is a group of files to be reviewed and submitted together,\n\tplus a textual description of the change.\n\tChange lists are referred to by simple alphanumeric names.\n\n\tChanges must be reviewed before they can be submitted.\n\n\tIn the absence of options, the change command opens the\n\tchange list for editing in the default editor.\n\n\tDeleting a change with the -d or -D flag does not affect\n\tthe contents of the files listed in that change. To revert\n\tthe files listed in a change, use\n\n\t\thg revert @123456\n\n\tbefore running hg change -d 123456.\n\t\"\"\"\n\n\tif codereview_disabled:\n\t\traise hg_util.Abort(codereview_disabled)\n\t\n\tdirty = {}\n\tif len(pats) > 0 and GoodCLName(pats[0]):\n\t\tname = pats[0]\n\t\tif len(pats) != 1:\n\t\t\traise hg_util.Abort(\"cannot specify CL name and file patterns\")\n\t\tpats = pats[1:]\n\t\tcl, err = LoadCL(ui, repo, name, web=True)\n\t\tif err != '':\n\t\t\traise hg_util.Abort(err)\n\t\tif not cl.local and (opts[\"stdin\"] or not opts[\"stdout\"]):\n\t\t\traise hg_util.Abort(\"cannot change non-local CL \" + name)\n\telse:\n\t\tname = \"new\"\n\t\tcl = CL(\"new\")\n\t\tif repo[None].branch() != \"default\":\n\t\t\traise hg_util.Abort(\"cannot create CL outside default branch; switch with 'hg update default'\")\n\t\tdirty[cl] = True\n\t\tfiles = ChangedFiles(ui, repo, pats, taken=Taken(ui, repo))\n\n\tif opts[\"delete\"] or opts[\"deletelocal\"]:\n\t\tif opts[\"delete\"] and opts[\"deletelocal\"]:\n\t\t\traise hg_util.Abort(\"cannot use -d and -D together\")\n\t\tflag = \"-d\"\n\t\tif opts[\"deletelocal\"]:\n\t\t\tflag = \"-D\"\n\t\tif name == \"new\":\n\t\t\traise hg_util.Abort(\"cannot use \"+flag+\" with file patterns\")\n\t\tif opts[\"stdin\"] or opts[\"stdout\"]:\n\t\t\traise hg_util.Abort(\"cannot use \"+flag+\" with -i or -o\")\n\t\tif not cl.local:\n\t\t\traise hg_util.Abort(\"cannot change non-local CL \" + name)\n\t\tif opts[\"delete\"]:\n\t\t\tif cl.copied_from:\n\t\t\t\traise hg_util.Abort(\"original author must delete CL; hg change -D will remove locally\")\n\t\t\tPostMessage(ui, cl.name, \"*** Abandoned ***\", send_mail=cl.mailed)\n\t\t\tEditDesc(cl.name, closed=True, private=cl.private)\n\t\tcl.Delete(ui, repo)\n\t\treturn\n\n\tif opts[\"stdin\"]:\n\t\ts = sys.stdin.read()\n\t\tclx, line, err = ParseCL(s, name)\n\t\tif err != '':\n\t\t\traise hg_util.Abort(\"error parsing change list: line %d: %s\" % (line, err))\n\t\tif clx.desc is not None:\n\t\t\tcl.desc = clx.desc;\n\t\t\tdirty[cl] = True\n\t\tif clx.reviewer is not None:\n\t\t\tcl.reviewer = clx.reviewer\n\t\t\tdirty[cl] = True\n\t\tif clx.cc is not None:\n\t\t\tcl.cc = clx.cc\n\t\t\tdirty[cl] = True\n\t\tif clx.files is not None:\n\t\t\tcl.files = clx.files\n\t\t\tdirty[cl] = True\n\t\tif clx.private != cl.private:\n\t\t\tcl.private = clx.private\n\t\t\tdirty[cl] = True\n\n\tif not opts[\"stdin\"] and not opts[\"stdout\"]:\n\t\tif name == \"new\":\n\t\t\tcl.files = files\n\t\terr = EditCL(ui, repo, cl)\n\t\tif err != \"\":\n\t\t\traise hg_util.Abort(err)\n\t\tdirty[cl] = True\n\n\tfor d, _ in dirty.items():\n\t\tname = d.name\n\t\td.Flush(ui, repo)\n\t\tif name == \"new\":\n\t\t\td.Upload(ui, repo, quiet=True)\n\n\tif opts[\"stdout\"]:\n\t\tui.write(cl.EditorText())\n\telif opts[\"pending\"]:\n\t\tui.write(cl.PendingText())\n\telif name == \"new\":\n\t\tif ui.quiet:\n\t\t\tui.write(cl.name)\n\t\telse:\n\t\t\tui.write(\"CL created: \" + cl.url + \"\\n\")\n\treturn"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nundoing the effect of a CL", "response": "def undo(ui, repo, clname, **opts):\n\t\"\"\"undo the effect of a CL\n\t\n\tCreates a new CL that undoes an earlier CL.\n\tAfter creating the CL, opens the CL text for editing so that\n\tyou can add the reason for the undo to the description.\n\t\"\"\"\n\tif repo[None].branch() != \"default\":\n\t\traise hg_util.Abort(\"cannot run hg undo outside default branch\")\n\terr = clpatch_or_undo(ui, repo, clname, opts, mode=\"undo\")\n\tif err:\n\t\traise hg_util.Abort(err)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef release_apply(ui, repo, clname, **opts):\n\tc = repo[None]\n\tif not releaseBranch:\n\t\traise hg_util.Abort(\"no active release branches\")\n\tif c.branch() != releaseBranch:\n\t\tif c.modified() or c.added() or c.removed():\n\t\t\traise hg_util.Abort(\"uncommitted local changes - cannot switch branches\")\n\t\terr = hg_clean(repo, releaseBranch)\n\t\tif err:\n\t\t\traise hg_util.Abort(err)\n\ttry:\n\t\terr = clpatch_or_undo(ui, repo, clname, opts, mode=\"backport\")\n\t\tif err:\n\t\t\traise hg_util.Abort(err)\n\texcept Exception, e:\n\t\thg_clean(repo, \"default\")\n\t\traise e", "response": "Apply a CL to the release branch."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef download(ui, repo, clname, **opts):\n\tif codereview_disabled:\n\t\traise hg_util.Abort(codereview_disabled)\n\n\tcl, vers, patch, err = DownloadCL(ui, repo, clname)\n\tif err != \"\":\n\t\treturn err\n\tui.write(cl.EditorText() + \"\\n\")\n\tui.write(patch + \"\\n\")\n\treturn", "response": "download a change from the code review server"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef file(ui, repo, clname, pat, *pats, **opts):\n\tif codereview_disabled:\n\t\traise hg_util.Abort(codereview_disabled)\n\n\tpats = tuple([pat] + list(pats))\n\tif not GoodCLName(clname):\n\t\treturn \"invalid CL name \" + clname\n\n\tdirty = {}\n\tcl, err = LoadCL(ui, repo, clname, web=False)\n\tif err != '':\n\t\treturn err\n\tif not cl.local:\n\t\treturn \"cannot change non-local CL \" + clname\n\n\tfiles = ChangedFiles(ui, repo, pats)\n\n\tif opts[\"delete\"]:\n\t\toldfiles = Intersect(files, cl.files)\n\t\tif oldfiles:\n\t\t\tif not ui.quiet:\n\t\t\t\tui.status(\"# Removing files from CL. To undo:\\n\")\n\t\t\t\tui.status(\"#\tcd %s\\n\" % (repo.root))\n\t\t\t\tfor f in oldfiles:\n\t\t\t\t\tui.status(\"#\thg file %s %s\\n\" % (cl.name, f))\n\t\t\tcl.files = Sub(cl.files, oldfiles)\n\t\t\tcl.Flush(ui, repo)\n\t\telse:\n\t\t\tui.status(\"no such files in CL\")\n\t\treturn\n\n\tif not files:\n\t\treturn \"no such modified files\"\n\n\tfiles = Sub(files, cl.files)\n\ttaken = Taken(ui, repo)\n\twarned = False\n\tfor f in files:\n\t\tif f in taken:\n\t\t\tif not warned and not ui.quiet:\n\t\t\t\tui.status(\"# Taking files from other CLs. To undo:\\n\")\n\t\t\t\tui.status(\"#\tcd %s\\n\" % (repo.root))\n\t\t\t\twarned = True\n\t\t\tocl = taken[f]\n\t\t\tif not ui.quiet:\n\t\t\t\tui.status(\"#\thg file %s %s\\n\" % (ocl.name, f))\n\t\t\tif ocl not in dirty:\n\t\t\t\tocl.files = Sub(ocl.files, files)\n\t\t\t\tdirty[ocl] = True\n\tcl.files = Add(cl.files, files)\n\tdirty[cl] = True\n\tfor d, _ in dirty.items():\n\t\td.Flush(ui, repo)\n\treturn", "response": "file - change list"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\napplies gofmt to modified files in the repository that match the given patterns.", "response": "def gofmt(ui, repo, *pats, **opts):\n\t\"\"\"apply gofmt to modified files\n\n\tApplies gofmt to the modified files in the repository that match\n\tthe given patterns.\n\t\"\"\"\n\tif codereview_disabled:\n\t\traise hg_util.Abort(codereview_disabled)\n\n\tfiles = ChangedExistingFiles(ui, repo, pats, opts)\n\tfiles = gofmt_required(files)\n\tif not files:\n\t\tui.status(\"no modified go files\\n\")\n\t\treturn\n\tcwd = os.getcwd()\n\tfiles = [RelativePath(repo.root + '/' + f, cwd) for f in files]\n\ttry:\n\t\tcmd = [\"gofmt\", \"-l\"]\n\t\tif not opts[\"list\"]:\n\t\t\tcmd += [\"-w\"]\n\t\tif subprocess.call(cmd + files) != 0:\n\t\t\traise hg_util.Abort(\"gofmt did not exit cleanly\")\n\texcept hg_error.Abort, e:\n\t\traise\n\texcept:\n\t\traise hg_util.Abort(\"gofmt: \" + ExceptionDetail())\n\treturn"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef mail(ui, repo, *pats, **opts):\n\tif codereview_disabled:\n\t\traise hg_util.Abort(codereview_disabled)\n\n\tcl, err = CommandLineCL(ui, repo, pats, opts, op=\"mail\", defaultcc=defaultcc)\n\tif err != \"\":\n\t\traise hg_util.Abort(err)\n\tcl.Upload(ui, repo, gofmt_just_warn=True)\n\tif not cl.reviewer:\n\t\t# If no reviewer is listed, assign the review to defaultcc.\n\t\t# This makes sure that it appears in the \n\t\t# codereview.appspot.com/user/defaultcc\n\t\t# page, so that it doesn't get dropped on the floor.\n\t\tif not defaultcc:\n\t\t\traise hg_util.Abort(\"no reviewers listed in CL\")\n\t\tcl.cc = Sub(cl.cc, defaultcc)\n\t\tcl.reviewer = defaultcc\n\t\tcl.Flush(ui, repo)\n\n\tif cl.files == []:\n\t\t\traise hg_util.Abort(\"no changed files, not sending mail\")\n\n\tcl.Mail(ui, repo)", "response": "mail a change for review\n\tUploads a patch to the code review server and sends mail\n\tto the reviewer and CC list asking for a review."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef ps(ui, repo, *pats, **opts):\n\topts['short'] = True\n\treturn pending(ui, repo, *pats, **opts)", "response": "alias for hg p"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\naliases for hg p", "response": "def pq(ui, repo, *pats, **opts):\n\t\"\"\"alias for hg p --quick\n\t\"\"\"\n\topts['quick'] = True\n\treturn pending(ui, repo, *pats, **opts)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nshows pending changes Lists pending changes followed by a list of unassigned but modified files.", "response": "def pending(ui, repo, *pats, **opts):\n\t\"\"\"show pending changes\n\n\tLists pending changes followed by a list of unassigned but modified files.\n\t\"\"\"\n\tif codereview_disabled:\n\t\traise hg_util.Abort(codereview_disabled)\n\n\tquick = opts.get('quick', False)\n\tshort = opts.get('short', False)\n\tm = LoadAllCL(ui, repo, web=not quick and not short)\n\tnames = m.keys()\n\tnames.sort()\n\tfor name in names:\n\t\tcl = m[name]\n\t\tif short:\n\t\t\tui.write(name + \"\\t\" + line1(cl.desc) + \"\\n\")\n\t\telse:\n\t\t\tui.write(cl.PendingText(quick=quick) + \"\\n\")\n\n\tif short:\n\t\treturn 0\n\tfiles = DefaultFiles(ui, repo, [])\n\tif len(files) > 0:\n\t\ts = \"Changed files not in any CL:\\n\"\n\t\tfor f in files:\n\t\t\ts += \"\\t\" + f + \"\\n\"\n\t\tui.write(s)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsubmit change to remote repository", "response": "def submit(ui, repo, *pats, **opts):\n\t\"\"\"submit change to remote repository\n\n\tSubmits change to remote repository.\n\tBails out if the local repository is not in sync with the remote one.\n\t\"\"\"\n\tif codereview_disabled:\n\t\traise hg_util.Abort(codereview_disabled)\n\n\t# We already called this on startup but sometimes Mercurial forgets.\n\tset_mercurial_encoding_to_utf8()\n\n\tif not opts[\"no_incoming\"] and hg_incoming(ui, repo):\n\t\tneed_sync()\n\n\tcl, err = CommandLineCL(ui, repo, pats, opts, op=\"submit\", defaultcc=defaultcc)\n\tif err != \"\":\n\t\traise hg_util.Abort(err)\n\n\tuser = None\n\tif cl.copied_from:\n\t\tuser = cl.copied_from\n\tuserline = CheckContributor(ui, repo, user)\n\ttypecheck(userline, str)\n\n\tabout = \"\"\n\tif cl.reviewer:\n\t\tabout += \"R=\" + JoinComma([CutDomain(s) for s in cl.reviewer]) + \"\\n\"\n\tif opts.get('tbr'):\n\t\ttbr = SplitCommaSpace(opts.get('tbr'))\n\t\tcl.reviewer = Add(cl.reviewer, tbr)\n\t\tabout += \"TBR=\" + JoinComma([CutDomain(s) for s in tbr]) + \"\\n\"\n\tif cl.cc:\n\t\tabout += \"CC=\" + JoinComma([CutDomain(s) for s in cl.cc]) + \"\\n\"\n\n\tif not cl.reviewer:\n\t\traise hg_util.Abort(\"no reviewers listed in CL\")\n\n\tif not cl.local:\n\t\traise hg_util.Abort(\"cannot submit non-local CL\")\n\n\t# upload, to sync current patch and also get change number if CL is new.\n\tif not cl.copied_from:\n\t\tcl.Upload(ui, repo, gofmt_just_warn=True)\n\n\t# check gofmt for real; allowed upload to warn in order to save CL.\n\tcl.Flush(ui, repo)\n\tCheckFormat(ui, repo, cl.files)\n\n\tabout += \"%s%s\\n\" % (server_url_base, cl.name)\n\n\tif cl.copied_from:\n\t\tabout += \"\\nCommitter: \" + CheckContributor(ui, repo, None) + \"\\n\"\n\ttypecheck(about, str)\n\n\tif not cl.mailed and not cl.copied_from:\t\t# in case this is TBR\n\t\tcl.Mail(ui, repo)\n\n\t# submit changes locally\n\tmessage = cl.desc.rstrip() + \"\\n\\n\" + about\n\ttypecheck(message, str)\n\n\tset_status(\"pushing \" + cl.name + \" to remote server\")\n\n\tif hg_outgoing(ui, repo):\n\t\traise hg_util.Abort(\"local repository corrupt or out-of-phase with remote: found outgoing changes\")\n\t\n\told_heads = len(hg_heads(ui, repo).split())\n\n\tglobal commit_okay\n\tcommit_okay = True\n\tret = hg_commit(ui, repo, *['path:'+f for f in cl.files], message=message, user=userline)\n\tcommit_okay = False\n\tif ret:\n\t\traise hg_util.Abort(\"nothing changed\")\n\tnode = repo[\"-1\"].node()\n\t# push to remote; if it fails for any reason, roll back\n\ttry:\n\t\tnew_heads = len(hg_heads(ui, repo).split())\n\t\tif old_heads != new_heads and not (old_heads == 0 and new_heads == 1):\n\t\t\t# Created new head, so we weren't up to date.\n\t\t\tneed_sync()\n\n\t\t# Push changes to remote. If it works, we're committed. If not, roll back.\n\t\ttry:\n\t\t\tif hg_push(ui, repo):\n\t\t\t\traise hg_util.Abort(\"push error\")\n\t\texcept hg_error.Abort, e:\n\t\t\tif e.message.find(\"push creates new heads\") >= 0:\n\t\t\t\t# Remote repository had changes we missed.\n\t\t\t\tneed_sync()\n\t\t\traise\n\t\texcept urllib2.HTTPError, e:\n\t\t\tprint >>sys.stderr, \"pushing to remote server failed; do you have commit permissions?\"\n\t\t\traise\n\texcept:\n\t\treal_rollback()\n\t\traise\n\n\t# We're committed. Upload final patch, close review, add commit message.\n\tchangeURL = hg_node.short(node)\n\turl = ui.expandpath(\"default\")\n\tm = re.match(\"(^https?://([^@/]+@)?([^.]+)\\.googlecode\\.com/hg/?)\" + \"|\" +\n\t\t\"(^https?://([^@/]+@)?code\\.google\\.com/p/([^/.]+)(\\.[^./]+)?/?)\", url)\n\tif m:\n\t\tif m.group(1): # prj.googlecode.com/hg/ case\n\t\t\tchangeURL = \"https://code.google.com/p/%s/source/detail?r=%s\" % (m.group(3), changeURL)\n\t\telif m.group(4) and m.group(7): # code.google.com/p/prj.subrepo/ case\n\t\t\tchangeURL = \"https://code.google.com/p/%s/source/detail?r=%s&repo=%s\" % (m.group(6), changeURL, m.group(7)[1:])\n\t\telif m.group(4): # code.google.com/p/prj/ case\n\t\t\tchangeURL = \"https://code.google.com/p/%s/source/detail?r=%s\" % (m.group(6), changeURL)\n\t\telse:\n\t\t\tprint >>sys.stderr, \"URL: \", url\n\telse:\n\t\tprint >>sys.stderr, \"URL: \", url\n\tpmsg = \"*** Submitted as \" + changeURL + \" ***\\n\\n\" + message\n\n\t# When posting, move reviewers to CC line,\n\t# so that the issue stops showing up in their \"My Issues\" page.\n\tPostMessage(ui, cl.name, pmsg, reviewers=\"\", cc=JoinComma(cl.reviewer+cl.cc))\n\n\tif not cl.copied_from:\n\t\tEditDesc(cl.name, closed=True, private=cl.private)\n\tcl.Delete(ui, repo)\n\n\tc = repo[None]\n\tif c.branch() == releaseBranch and not c.modified() and not c.added() and not c.removed():\n\t\tui.write(\"switching from %s to default branch.\\n\" % releaseBranch)\n\t\terr = hg_clean(repo, \"default\")\n\t\tif err:\n\t\t\treturn err\n\treturn 0"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef sync(ui, repo, **opts):\n\tif codereview_disabled:\n\t\traise hg_util.Abort(codereview_disabled)\n\n\tif not opts[\"local\"]:\n\t\t# If there are incoming CLs, pull -u will do the update.\n\t\t# If there are no incoming CLs, do hg update to make sure\n\t\t# that an update always happens regardless. This is less\n\t\t# surprising than update depending on incoming CLs.\n\t\t# It is important not to do both hg pull -u and hg update\n\t\t# in the same command, because the hg update will end\n\t\t# up marking resolve conflicts from the hg pull -u as resolved,\n\t\t# causing files with <<< >>> markers to not show up in \n\t\t# hg resolve -l. Yay Mercurial.\n\t\tif hg_incoming(ui, repo):\n\t\t\terr = hg_pull(ui, repo, update=True)\n\t\telse:\n\t\t\terr = hg_update(ui, repo)\n\t\tif err:\n\t\t\treturn err\n\tsync_changes(ui, repo)", "response": "synchronize with remote repository\n\tIncorporates recent changes from the remote repository\n\t into the local repository\n\tif local flag is set to True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef upload(ui, repo, name, **opts):\n\tif codereview_disabled:\n\t\traise hg_util.Abort(codereview_disabled)\n\n\trepo.ui.quiet = True\n\tcl, err = LoadCL(ui, repo, name, web=True)\n\tif err != \"\":\n\t\traise hg_util.Abort(err)\n\tif not cl.local:\n\t\traise hg_util.Abort(\"cannot upload non-local change\")\n\tcl.Upload(ui, repo)\n\tprint \"%s%s\\n\" % (server_url_base, cl.name)\n\treturn 0", "response": "upload diffs to the code review server"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef MySend(request_path, payload=None,\n\t\tcontent_type=\"application/octet-stream\",\n\t\ttimeout=None, force_auth=True,\n\t\t**kwargs):\n\t\"\"\"Run MySend1 maybe twice, because Rietveld is unreliable.\"\"\"\n\ttry:\n\t\treturn MySend1(request_path, payload, content_type, timeout, force_auth, **kwargs)\n\texcept Exception, e:\n\t\tif type(e) != urllib2.HTTPError or e.code != 500:\t# only retry on HTTP 500 error\n\t\t\traise\n\t\tprint >>sys.stderr, \"Loading \"+request_path+\": \"+ExceptionDetail()+\"; trying again in 2 seconds.\"\n\t\ttime.sleep(2)\n\t\treturn MySend1(request_path, payload, content_type, timeout, force_auth, **kwargs)", "response": "Run MySend1 maybe twice because Rietveld is unreliable."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsending a request to the server and return the response body.", "response": "def MySend1(request_path, payload=None,\n\t\t\t\tcontent_type=\"application/octet-stream\",\n\t\t\t\ttimeout=None, force_auth=True,\n\t\t\t\t**kwargs):\n\t\"\"\"Sends an RPC and returns the response.\n\n\tArgs:\n\t\trequest_path: The path to send the request to, eg /api/appversion/create.\n\t\tpayload: The body of the request, or None to send an empty request.\n\t\tcontent_type: The Content-Type header to use.\n\t\ttimeout: timeout in seconds; default None i.e. no timeout.\n\t\t\t(Note: for large requests on OS X, the timeout doesn't work right.)\n\t\tkwargs: Any keyword arguments are converted into query string parameters.\n\n\tReturns:\n\t\tThe response body, as a string.\n\t\"\"\"\n\t# TODO: Don't require authentication. Let the server say\n\t# whether it is necessary.\n\tglobal rpc\n\tif rpc == None:\n\t\trpc = GetRpcServer(upload_options)\n\tself = rpc\n\tif not self.authenticated and force_auth:\n\t\tself._Authenticate()\n\tif request_path is None:\n\t\treturn\n\tif timeout is None:\n\t\ttimeout = 30 # seconds\n\n\told_timeout = socket.getdefaulttimeout()\n\tsocket.setdefaulttimeout(timeout)\n\ttry:\n\t\ttries = 0\n\t\twhile True:\n\t\t\ttries += 1\n\t\t\targs = dict(kwargs)\n\t\t\turl = \"https://%s%s\" % (self.host, request_path)\n\t\t\tif args:\n\t\t\t\turl += \"?\" + urllib.urlencode(args)\n\t\t\treq = self._CreateRequest(url=url, data=payload)\n\t\t\treq.add_header(\"Content-Type\", content_type)\n\t\t\ttry:\n\t\t\t\tf = self.opener.open(req)\n\t\t\t\tresponse = f.read()\n\t\t\t\tf.close()\n\t\t\t\t# Translate \\r\\n into \\n, because Rietveld doesn't.\n\t\t\t\tresponse = response.replace('\\r\\n', '\\n')\n\t\t\t\t# who knows what urllib will give us\n\t\t\t\tif type(response) == unicode:\n\t\t\t\t\tresponse = response.encode(\"utf-8\")\n\t\t\t\ttypecheck(response, str)\n\t\t\t\treturn response\n\t\t\texcept urllib2.HTTPError, e:\n\t\t\t\tif tries > 3:\n\t\t\t\t\traise\n\t\t\t\telif e.code == 401:\n\t\t\t\t\tself._Authenticate()\n\t\t\t\telif e.code == 302:\n\t\t\t\t\tloc = e.info()[\"location\"]\n\t\t\t\t\tif not loc.startswith('https://www.google.com/a') or loc.find('/ServiceLogin') < 0:\n\t\t\t\t\t\treturn ''\n\t\t\t\t\tself._Authenticate()\n\t\t\t\telse:\n\t\t\t\t\traise\n\tfinally:\n\t\tsocket.setdefaulttimeout(old_timeout)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef GetEmail(prompt):\n\tlast_email_file_name = os.path.expanduser(\"~/.last_codereview_email_address\")\n\tlast_email = \"\"\n\tif os.path.exists(last_email_file_name):\n\t\ttry:\n\t\t\tlast_email_file = open(last_email_file_name, \"r\")\n\t\t\tlast_email = last_email_file.readline().strip(\"\\n\")\n\t\t\tlast_email_file.close()\n\t\t\tprompt += \" [%s]\" % last_email\n\t\texcept IOError, e:\n\t\t\tpass\n\temail = raw_input(prompt + \": \").strip()\n\tif email:\n\t\ttry:\n\t\t\tlast_email_file = open(last_email_file_name, \"w\")\n\t\t\tlast_email_file.write(email)\n\t\t\tlast_email_file.close()\n\t\texcept IOError, e:\n\t\t\tpass\n\telse:\n\t\temail = last_email\n\treturn email", "response": "Prompts the user for their email address and returns it."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef GetRpcServer(options):\n\n\trpc_server_class = HttpRpcServer\n\n\tdef GetUserCredentials():\n\t\t\"\"\"Prompts the user for a username and password.\"\"\"\n\t\t# Disable status prints so they don't obscure the password prompt.\n\t\tglobal global_status\n\t\tst = global_status\n\t\tglobal_status = None\n\n\t\temail = options.email\n\t\tif email is None:\n\t\t\temail = GetEmail(\"Email (login for uploading to %s)\" % options.server)\n\t\tpassword = getpass.getpass(\"Password for %s: \" % email)\n\n\t\t# Put status back.\n\t\tglobal_status = st\n\t\treturn (email, password)\n\n\t# If this is the dev_appserver, use fake authentication.\n\thost = (options.host or options.server).lower()\n\tif host == \"localhost\" or host.startswith(\"localhost:\"):\n\t\temail = options.email\n\t\tif email is None:\n\t\t\temail = \"test@example.com\"\n\t\t\tlogging.info(\"Using debug user %s. Override with --email\" % email)\n\t\tserver = rpc_server_class(\n\t\t\t\toptions.server,\n\t\t\t\tlambda: (email, \"password\"),\n\t\t\t\thost_override=options.host,\n\t\t\t\textra_headers={\"Cookie\": 'dev_appserver_login=\"%s:False\"' % email},\n\t\t\t\tsave_cookies=options.save_cookies)\n\t\t# Don't try to talk to ClientLogin.\n\t\tserver.authenticated = True\n\t\treturn server\n\n\treturn rpc_server_class(options.server, GetUserCredentials,\n\t\thost_override=options.host, save_cookies=options.save_cookies)", "response": "Returns an instance of an AbstractRpcServer."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nencode form fields for multipart / form - data.", "response": "def EncodeMultipartFormData(fields, files):\n\t\"\"\"Encode form fields for multipart/form-data.\n\n\tArgs:\n\t\tfields: A sequence of (name, value) elements for regular form fields.\n\t\tfiles: A sequence of (name, filename, value) elements for data to be\n\t\t\t\t\tuploaded as files.\n\tReturns:\n\t\t(content_type, body) ready for httplib.HTTP instance.\n\n\tSource:\n\t\thttp://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/146306\n\t\"\"\"\n\tBOUNDARY = '-M-A-G-I-C---B-O-U-N-D-A-R-Y-'\n\tCRLF = '\\r\\n'\n\tlines = []\n\tfor (key, value) in fields:\n\t\ttypecheck(key, str)\n\t\ttypecheck(value, str)\n\t\tlines.append('--' + BOUNDARY)\n\t\tlines.append('Content-Disposition: form-data; name=\"%s\"' % key)\n\t\tlines.append('')\n\t\tlines.append(value)\n\tfor (key, filename, value) in files:\n\t\ttypecheck(key, str)\n\t\ttypecheck(filename, str)\n\t\ttypecheck(value, str)\n\t\tlines.append('--' + BOUNDARY)\n\t\tlines.append('Content-Disposition: form-data; name=\"%s\"; filename=\"%s\"' % (key, filename))\n\t\tlines.append('Content-Type: %s' % GetContentType(filename))\n\t\tlines.append('')\n\t\tlines.append(value)\n\tlines.append('--' + BOUNDARY + '--')\n\tlines.append('')\n\tbody = CRLF.join(lines)\n\tcontent_type = 'multipart/form-data; boundary=%s' % BOUNDARY\n\treturn content_type, body"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nexecutes a command and returns the output from stdout and the return code.", "response": "def RunShellWithReturnCode(command, print_output=False,\n\t\tuniversal_newlines=True, env=os.environ):\n\t\"\"\"Executes a command and returns the output from stdout and the return code.\n\n\tArgs:\n\t\tcommand: Command to execute.\n\t\tprint_output: If True, the output is printed to stdout.\n\t\t\tIf False, both stdout and stderr are ignored.\n\t\tuniversal_newlines: Use universal_newlines flag (default: True).\n\n\tReturns:\n\t\tTuple (output, return code)\n\t\"\"\"\n\tlogging.info(\"Running %s\", command)\n\tp = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE,\n\t\tshell=use_shell, universal_newlines=universal_newlines, env=env)\n\tif print_output:\n\t\toutput_array = []\n\t\twhile True:\n\t\t\tline = p.stdout.readline()\n\t\t\tif not line:\n\t\t\t\tbreak\n\t\t\tprint line.strip(\"\\n\")\n\t\t\toutput_array.append(line)\n\t\toutput = \"\".join(output_array)\n\telse:\n\t\toutput = p.stdout.read()\n\tp.wait()\n\terrout = p.stderr.read()\n\tif print_output and errout:\n\t\tprint >>sys.stderr, errout\n\tp.stdout.close()\n\tp.stderr.close()\n\treturn output, p.returncode"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef SplitPatch(data):\n\tpatches = []\n\tfilename = None\n\tdiff = []\n\tfor line in data.splitlines(True):\n\t\tnew_filename = None\n\t\tif line.startswith('Index:'):\n\t\t\tunused, new_filename = line.split(':', 1)\n\t\t\tnew_filename = new_filename.strip()\n\t\telif line.startswith('Property changes on:'):\n\t\t\tunused, temp_filename = line.split(':', 1)\n\t\t\t# When a file is modified, paths use '/' between directories, however\n\t\t\t# when a property is modified '\\' is used on Windows. Make them the same\n\t\t\t# otherwise the file shows up twice.\n\t\t\ttemp_filename = to_slash(temp_filename.strip())\n\t\t\tif temp_filename != filename:\n\t\t\t\t# File has property changes but no modifications, create a new diff.\n\t\t\t\tnew_filename = temp_filename\n\t\tif new_filename:\n\t\t\tif filename and diff:\n\t\t\t\tpatches.append((filename, ''.join(diff)))\n\t\t\tfilename = new_filename\n\t\t\tdiff = [line]\n\t\t\tcontinue\n\t\tif diff is not None:\n\t\t\tdiff.append(line)\n\tif filename and diff:\n\t\tpatches.append((filename, ''.join(diff)))\n\treturn patches", "response": "Splits a svn diff into separate pieces for each file."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef UploadSeparatePatches(issue, rpc_server, patchset, data, options):\n\tpatches = SplitPatch(data)\n\trv = []\n\tfor patch in patches:\n\t\tset_status(\"uploading patch for \" + patch[0])\n\t\tif len(patch[1]) > MAX_UPLOAD_SIZE:\n\t\t\tprint (\"Not uploading the patch for \" + patch[0] +\n\t\t\t\t\" because the file is too large.\")\n\t\t\tcontinue\n\t\tform_fields = [(\"filename\", patch[0])]\n\t\tif not options.download_base:\n\t\t\tform_fields.append((\"content_upload\", \"1\"))\n\t\tfiles = [(\"data\", \"data.diff\", patch[1])]\n\t\tctype, body = EncodeMultipartFormData(form_fields, files)\n\t\turl = \"/%d/upload_patch/%d\" % (int(issue), int(patchset))\n\t\tprint \"Uploading patch for \" + patch[0]\n\t\tresponse_body = rpc_server.Send(url, body, content_type=ctype)\n\t\tlines = response_body.splitlines()\n\t\tif not lines or lines[0] != \"OK\":\n\t\t\tStatusUpdate(\" --> %s\" % response_body)\n\t\t\tsys.exit(1)\n\t\trv.append([lines[1], patch[0]])\n\treturn rv", "response": "Uploads a separate patch for each file in the diff output."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate a new urllib request.", "response": "def _CreateRequest(self, url, data=None):\n\t\t\"\"\"Creates a new urllib request.\"\"\"\n\t\tlogging.debug(\"Creating request for: '%s' with payload:\\n%s\", url, data)\n\t\treq = urllib2.Request(url, data=data)\n\t\tif self.host_override:\n\t\t\treq.add_header(\"Host\", self.host_override)\n\t\tfor key, value in self.extra_headers.iteritems():\n\t\t\treq.add_header(key, value)\n\t\treturn req"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _GetAuthToken(self, email, password):\n\t\taccount_type = \"GOOGLE\"\n\t\tif self.host.endswith(\".google.com\") and not force_google_account:\n\t\t\t# Needed for use inside Google.\n\t\t\taccount_type = \"HOSTED\"\n\t\treq = self._CreateRequest(\n\t\t\t\turl=\"https://www.google.com/accounts/ClientLogin\",\n\t\t\t\tdata=urllib.urlencode({\n\t\t\t\t\t\t\"Email\": email,\n\t\t\t\t\t\t\"Passwd\": password,\n\t\t\t\t\t\t\"service\": \"ah\",\n\t\t\t\t\t\t\"source\": \"rietveld-codereview-upload\",\n\t\t\t\t\t\t\"accountType\": account_type,\n\t\t\t\t}),\n\t\t)\n\t\ttry:\n\t\t\tresponse = self.opener.open(req)\n\t\t\tresponse_body = response.read()\n\t\t\tresponse_dict = dict(x.split(\"=\") for x in response_body.split(\"\\n\") if x)\n\t\t\treturn response_dict[\"Auth\"]\n\t\texcept urllib2.HTTPError, e:\n\t\t\tif e.code == 403:\n\t\t\t\tbody = e.read()\n\t\t\t\tresponse_dict = dict(x.split(\"=\", 1) for x in body.split(\"\\n\") if x)\n\t\t\t\traise ClientLoginError(req.get_full_url(), e.code, e.msg, e.headers, response_dict)\n\t\t\telse:\n\t\t\t\traise", "response": "Uses ClientLogin to authenticate the user returning an auth token."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfetch the authentication cookies for an authentication token.", "response": "def _GetAuthCookie(self, auth_token):\n\t\t\"\"\"Fetches authentication cookies for an authentication token.\n\n\t\tArgs:\n\t\t\tauth_token: The authentication token returned by ClientLogin.\n\n\t\tRaises:\n\t\t\tHTTPError: If there was an error fetching the authentication cookies.\n\t\t\"\"\"\n\t\t# This is a dummy value to allow us to identify when we're successful.\n\t\tcontinue_location = \"http://localhost/\"\n\t\targs = {\"continue\": continue_location, \"auth\": auth_token}\n\t\treq = self._CreateRequest(\"https://%s/_ah/login?%s\" % (self.host, urllib.urlencode(args)))\n\t\ttry:\n\t\t\tresponse = self.opener.open(req)\n\t\texcept urllib2.HTTPError, e:\n\t\t\tresponse = e\n\t\tif (response.code != 302 or\n\t\t\t\tresponse.info()[\"location\"] != continue_location):\n\t\t\traise urllib2.HTTPError(req.get_full_url(), response.code, response.msg, response.headers, response.fp)\n\t\tself.authenticated = True"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _Authenticate(self):\n\t\tfor i in range(3):\n\t\t\tcredentials = self.auth_function()\n\t\t\ttry:\n\t\t\t\tauth_token = self._GetAuthToken(credentials[0], credentials[1])\n\t\t\texcept ClientLoginError, e:\n\t\t\t\tif e.msg == \"BadAuthentication\":\n\t\t\t\t\tprint >>sys.stderr, \"Invalid username or password.\"\n\t\t\t\t\tcontinue\n\t\t\t\tif e.msg == \"CaptchaRequired\":\n\t\t\t\t\tprint >>sys.stderr, (\n\t\t\t\t\t\t\"Please go to\\n\"\n\t\t\t\t\t\t\"https://www.google.com/accounts/DisplayUnlockCaptcha\\n\"\n\t\t\t\t\t\t\"and verify you are a human. Then try again.\")\n\t\t\t\t\tbreak\n\t\t\t\tif e.msg == \"NotVerified\":\n\t\t\t\t\tprint >>sys.stderr, \"Account not verified.\"\n\t\t\t\t\tbreak\n\t\t\t\tif e.msg == \"TermsNotAgreed\":\n\t\t\t\t\tprint >>sys.stderr, \"User has not agreed to TOS.\"\n\t\t\t\t\tbreak\n\t\t\t\tif e.msg == \"AccountDeleted\":\n\t\t\t\t\tprint >>sys.stderr, \"The user account has been deleted.\"\n\t\t\t\t\tbreak\n\t\t\t\tif e.msg == \"AccountDisabled\":\n\t\t\t\t\tprint >>sys.stderr, \"The user account has been disabled.\"\n\t\t\t\t\tbreak\n\t\t\t\tif e.msg == \"ServiceDisabled\":\n\t\t\t\t\tprint >>sys.stderr, \"The user's access to the service has been disabled.\"\n\t\t\t\t\tbreak\n\t\t\t\tif e.msg == \"ServiceUnavailable\":\n\t\t\t\t\tprint >>sys.stderr, \"The service is not available; try again later.\"\n\t\t\t\t\tbreak\n\t\t\t\traise\n\t\t\tself._GetAuthCookie(auth_token)\n\t\t\treturn", "response": "Authenticates the user.\n\n\t\tThe authentication process works as follows:\n\t\t1) We get a username and password from the user\n\t\t2) We use ClientLogin to obtain an AUTH token for the user\n\t\t\t\t(see http://code.google.com/apis/accounts/AuthForInstalledApps.html).\n\t\t3) We pass the auth token to /_ah/login on the server to obtain an\n\t\t\t\tauthentication cookie. If login was successful, it tries to redirect\n\t\t\t\tus to the URL we provided.\n\n\t\tIf we attempt to access the upload API without first obtaining an\n\t\tauthentication cookie, it returns a 401 response (or a 302) and\n\t\tdirects us to authenticate ourselves with ClientLogin."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef Send(self, request_path, payload=None,\n\t\t\t\t\tcontent_type=\"application/octet-stream\",\n\t\t\t\t\ttimeout=None,\n\t\t\t\t\t**kwargs):\n\t\t\"\"\"Sends an RPC and returns the response.\n\n\t\tArgs:\n\t\t\trequest_path: The path to send the request to, eg /api/appversion/create.\n\t\t\tpayload: The body of the request, or None to send an empty request.\n\t\t\tcontent_type: The Content-Type header to use.\n\t\t\ttimeout: timeout in seconds; default None i.e. no timeout.\n\t\t\t\t(Note: for large requests on OS X, the timeout doesn't work right.)\n\t\t\tkwargs: Any keyword arguments are converted into query string parameters.\n\n\t\tReturns:\n\t\t\tThe response body, as a string.\n\t\t\"\"\"\n\t\t# TODO: Don't require authentication. Let the server say\n\t\t# whether it is necessary.\n\t\tif not self.authenticated:\n\t\t\tself._Authenticate()\n\n\t\told_timeout = socket.getdefaulttimeout()\n\t\tsocket.setdefaulttimeout(timeout)\n\t\ttry:\n\t\t\ttries = 0\n\t\t\twhile True:\n\t\t\t\ttries += 1\n\t\t\t\targs = dict(kwargs)\n\t\t\t\turl = \"https://%s%s\" % (self.host, request_path)\n\t\t\t\tif args:\n\t\t\t\t\turl += \"?\" + urllib.urlencode(args)\n\t\t\t\treq = self._CreateRequest(url=url, data=payload)\n\t\t\t\treq.add_header(\"Content-Type\", content_type)\n\t\t\t\ttry:\n\t\t\t\t\tf = self.opener.open(req)\n\t\t\t\t\tresponse = f.read()\n\t\t\t\t\tf.close()\n\t\t\t\t\treturn response\n\t\t\t\texcept urllib2.HTTPError, e:\n\t\t\t\t\tif tries > 3:\n\t\t\t\t\t\traise\n\t\t\t\t\telif e.code == 401 or e.code == 302:\n\t\t\t\t\t\tself._Authenticate()\n\t\t\t\t\telse:\n\t\t\t\t\t\traise\n\t\tfinally:\n\t\t\tsocket.setdefaulttimeout(old_timeout)", "response": "Sends an RPC and returns the response body."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _Authenticate(self):\n\t\tsuper(HttpRpcServer, self)._Authenticate()\n\t\tif self.save_cookies:\n\t\t\tStatusUpdate(\"Saving authentication cookies to %s\" % self.cookie_file)\n\t\t\tself.cookie_jar.save()", "response": "Save the cookie jar after authentication."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns an OpenerDirector that supports cookies and ignores redirects.", "response": "def _GetOpener(self):\n\t\t\"\"\"Returns an OpenerDirector that supports cookies and ignores redirects.\n\n\t\tReturns:\n\t\t\tA urllib2.OpenerDirector object.\n\t\t\"\"\"\n\t\topener = urllib2.OpenerDirector()\n\t\topener.add_handler(urllib2.ProxyHandler())\n\t\topener.add_handler(urllib2.UnknownHandler())\n\t\topener.add_handler(urllib2.HTTPHandler())\n\t\topener.add_handler(urllib2.HTTPDefaultErrorHandler())\n\t\topener.add_handler(urllib2.HTTPSHandler())\n\t\topener.add_handler(urllib2.HTTPErrorProcessor())\n\t\tif self.save_cookies:\n\t\t\tself.cookie_file = os.path.expanduser(\"~/.codereview_upload_cookies_\" + server)\n\t\t\tself.cookie_jar = cookielib.MozillaCookieJar(self.cookie_file)\n\t\t\tif os.path.exists(self.cookie_file):\n\t\t\t\ttry:\n\t\t\t\t\tself.cookie_jar.load()\n\t\t\t\t\tself.authenticated = True\n\t\t\t\t\tStatusUpdate(\"Loaded authentication cookies from %s\" % self.cookie_file)\n\t\t\t\texcept (cookielib.LoadError, IOError):\n\t\t\t\t\t# Failed to load cookies - just ignore them.\n\t\t\t\t\tpass\n\t\t\telse:\n\t\t\t\t# Create an empty cookie file with mode 600\n\t\t\t\tfd = os.open(self.cookie_file, os.O_CREAT, 0600)\n\t\t\t\tos.close(fd)\n\t\t\t# Always chmod the cookie file\n\t\t\tos.chmod(self.cookie_file, 0600)\n\t\telse:\n\t\t\t# Don't save cookies across runs of update.py.\n\t\t\tself.cookie_jar = cookielib.CookieJar()\n\t\topener.add_handler(urllib2.HTTPCookieProcessor(self.cookie_jar))\n\t\treturn opener"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nshowing an error if there are unknown files.", "response": "def CheckForUnknownFiles(self):\n\t\t\"\"\"Show an \"are you sure?\" prompt if there are unknown files.\"\"\"\n\t\tunknown_files = self.GetUnknownFiles()\n\t\tif unknown_files:\n\t\t\tprint \"The following files are not added to version control:\"\n\t\t\tfor line in unknown_files:\n\t\t\t\tprint line\n\t\t\tprompt = \"Are you sure to continue?(y/N) \"\n\t\t\tanswer = raw_input(prompt).strip()\n\t\t\tif answer != \"y\":\n\t\t\t\tErrorExit(\"User aborted\")"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef UploadBaseFiles(self, issue, rpc_server, patch_list, patchset, options,\n\t\t\t\t\t\t\t\t\t\t\tfiles):\n\t\t\"\"\"Uploads the base files (and if necessary, the current ones as well).\"\"\"\n\n\t\tdef UploadFile(filename, file_id, content, is_binary, status, is_base):\n\t\t\t\"\"\"Uploads a file to the server.\"\"\"\n\t\t\tset_status(\"uploading \" + filename)\n\t\t\tfile_too_large = False\n\t\t\tif is_base:\n\t\t\t\ttype = \"base\"\n\t\t\telse:\n\t\t\t\ttype = \"current\"\n\t\t\tif len(content) > MAX_UPLOAD_SIZE:\n\t\t\t\tprint (\"Not uploading the %s file for %s because it's too large.\" %\n\t\t\t\t\t\t\t(type, filename))\n\t\t\t\tfile_too_large = True\n\t\t\t\tcontent = \"\"\n\t\t\tchecksum = md5(content).hexdigest()\n\t\t\tif options.verbose > 0 and not file_too_large:\n\t\t\t\tprint \"Uploading %s file for %s\" % (type, filename)\n\t\t\turl = \"/%d/upload_content/%d/%d\" % (int(issue), int(patchset), file_id)\n\t\t\tform_fields = [\n\t\t\t\t(\"filename\", filename),\n\t\t\t\t(\"status\", status),\n\t\t\t\t(\"checksum\", checksum),\n\t\t\t\t(\"is_binary\", str(is_binary)),\n\t\t\t\t(\"is_current\", str(not is_base)),\n\t\t\t]\n\t\t\tif file_too_large:\n\t\t\t\tform_fields.append((\"file_too_large\", \"1\"))\n\t\t\tif options.email:\n\t\t\t\tform_fields.append((\"user\", options.email))\n\t\t\tctype, body = EncodeMultipartFormData(form_fields, [(\"data\", filename, content)])\n\t\t\tresponse_body = rpc_server.Send(url, body, content_type=ctype)\n\t\t\tif not response_body.startswith(\"OK\"):\n\t\t\t\tStatusUpdate(\" --> %s\" % response_body)\n\t\t\t\tsys.exit(1)\n\n\t\t# Don't want to spawn too many threads, nor do we want to\n\t\t# hit Rietveld too hard, or it will start serving 500 errors.\n\t\t# When 8 works, it's no better than 4, and sometimes 8 is\n\t\t# too many for Rietveld to handle.\n\t\tMAX_PARALLEL_UPLOADS = 4\n\n\t\tsema = threading.BoundedSemaphore(MAX_PARALLEL_UPLOADS)\n\t\tupload_threads = []\n\t\tfinished_upload_threads = []\n\t\t\n\t\tclass UploadFileThread(threading.Thread):\n\t\t\tdef __init__(self, args):\n\t\t\t\tthreading.Thread.__init__(self)\n\t\t\t\tself.args = args\n\t\t\tdef run(self):\n\t\t\t\tUploadFile(*self.args)\n\t\t\t\tfinished_upload_threads.append(self)\n\t\t\t\tsema.release()\n\n\t\tdef StartUploadFile(*args):\n\t\t\tsema.acquire()\n\t\t\twhile len(finished_upload_threads) > 0:\n\t\t\t\tt = finished_upload_threads.pop()\n\t\t\t\tupload_threads.remove(t)\n\t\t\t\tt.join()\n\t\t\tt = UploadFileThread(args)\n\t\t\tupload_threads.append(t)\n\t\t\tt.start()\n\n\t\tdef WaitForUploads():\t\t\t\n\t\t\tfor t in upload_threads:\n\t\t\t\tt.join()\n\n\t\tpatches = dict()\n\t\t[patches.setdefault(v, k) for k, v in patch_list]\n\t\tfor filename in patches.keys():\n\t\t\tbase_content, new_content, is_binary, status = files[filename]\n\t\t\tfile_id_str = patches.get(filename)\n\t\t\tif file_id_str.find(\"nobase\") != -1:\n\t\t\t\tbase_content = None\n\t\t\t\tfile_id_str = file_id_str[file_id_str.rfind(\"_\") + 1:]\n\t\t\tfile_id = int(file_id_str)\n\t\t\tif base_content != None:\n\t\t\t\tStartUploadFile(filename, file_id, base_content, is_binary, status, True)\n\t\t\tif new_content != None:\n\t\t\t\tStartUploadFile(filename, file_id, new_content, is_binary, status, False)\n\t\tWaitForUploads()", "response": "This method uploads the base files to the server."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning true if the filename has an image extension.", "response": "def IsImage(self, filename):\n\t\t\"\"\"Returns true if the filename has an image extension.\"\"\"\n\t\tmimetype = mimetypes.guess_type(filename)[0]\n\t\tif not mimetype:\n\t\t\treturn False\n\t\treturn mimetype.startswith(\"image/\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn true if the guessed mimetype is a text group.", "response": "def IsBinary(self, filename):\n\t\t\"\"\"Returns true if the guessed mimetyped isnt't in text group.\"\"\"\n\t\tmimetype = mimetypes.guess_type(filename)[0]\n\t\tif not mimetype:\n\t\t\treturn False # e.g. README, \"real\" binaries usually have an extension\n\t\t# special case for text files which don't start with text/\n\t\tif mimetype in TEXT_MIMETYPES:\n\t\t\treturn False\n\t\treturn not mimetype.startswith(\"text/\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting relative path of a file according to the current directory.", "response": "def _GetRelPath(self, filename):\n\t\t\"\"\"Get relative path of a file according to the current directory,\n\t\tgiven its logical path in the repo.\"\"\"\n\t\tassert filename.startswith(self.subdir), (filename, self.subdir)\n\t\treturn filename[len(self.subdir):].lstrip(r\"\\/\")"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef GetUnknownFiles(self):\n\t\targs = []\n\t\tstatus = RunShell([\"hg\", \"status\", \"--rev\", self.base_rev, \"-u\", \".\"],\n\t\t\t\tsilent_ok=True)\n\t\tunknown_files = []\n\t\tfor line in status.splitlines():\n\t\t\tst, fn = line.split(\" \", 1)\n\t\t\tif st == \"?\":\n\t\t\t\tunknown_files.append(fn)\n\t\treturn unknown_files", "response": "Return a list of files unknown to the VCS."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting a Unicode string to Unicode code point.", "response": "def _UInt(s):\n \"\"\"Converts string to Unicode code point ('263A' => 0x263a).\n\n Args:\n s: string to convert\n\n Returns:\n Unicode code point\n\n Raises:\n InputError: the string is not a valid Unicode value.\n \"\"\"\n\n try:\n v = int(s, 16)\n except ValueError:\n v = -1\n if len(s) < 4 or len(s) > 6 or v < 0 or v > _RUNE_MAX:\n raise InputError(\"invalid Unicode value %s\" % (s,))\n return v"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _URange(s):\n a = s.split(\"..\")\n if len(a) == 1:\n return [_UInt(a[0])]\n if len(a) == 2:\n lo = _UInt(a[0])\n hi = _UInt(a[1])\n if lo < hi:\n return range(lo, hi + 1)\n raise InputError(\"invalid Unicode range %s\" % (s,))", "response": "Converts a string to Unicode range."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nparsing a Unicode continuation field.", "response": "def _ParseContinue(s):\n \"\"\"Parses a Unicode continuation field.\n\n These are of the form '' or ''.\n Instead of giving an explicit range in a single table entry,\n some Unicode tables use two entries, one for the first\n code value in the range and one for the last.\n The first entry's description is '' instead of 'Name'\n and the second is ''.\n\n '' => ('Name', 'First')\n '' => ('Name', 'Last')\n 'Anything else' => ('Anything else', None)\n\n Args:\n s: continuation field string\n\n Returns:\n pair: name and ('First', 'Last', or None)\n \"\"\"\n\n match = re.match(\"<(.*), (First|Last)>\", s)\n if match is not None:\n return match.groups()\n return (s, None)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef CaseGroups(unicode_dir=_UNICODE_DIR):\n\n # Dict mapping lowercase code point to fold-equivalent group.\n togroup = {}\n\n def DoLine(codes, fields):\n \"\"\"Process single CaseFolding.txt line, updating togroup.\"\"\"\n (_, foldtype, lower, _) = fields\n if foldtype not in (\"C\", \"S\"):\n return\n lower = _UInt(lower)\n togroup.setdefault(lower, [lower]).extend(codes)\n\n ReadUnicodeTable(unicode_dir+\"/CaseFolding.txt\", 4, DoLine)\n\n groups = togroup.values()\n for g in groups:\n g.sort()\n groups.sort()\n return togroup, groups", "response": "Returns list of Unicode code groups equivalent under case folding."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning dict mapping script names to code lists.", "response": "def Scripts(unicode_dir=_UNICODE_DIR):\n \"\"\"Returns dict mapping script names to code lists.\n\n Args:\n unicode_dir: Unicode data directory\n\n Returns:\n dict mapping script names to code lists\n \"\"\"\n\n scripts = {}\n\n def DoLine(codes, fields):\n \"\"\"Process single Scripts.txt line, updating scripts.\"\"\"\n (_, name) = fields\n scripts.setdefault(name, []).extend(codes)\n\n ReadUnicodeTable(unicode_dir+\"/Scripts.txt\", 2, DoLine)\n return scripts"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef Categories(unicode_dir=_UNICODE_DIR):\n\n categories = {}\n\n def DoLine(codes, fields):\n \"\"\"Process single UnicodeData.txt line, updating categories.\"\"\"\n category = fields[2]\n categories.setdefault(category, []).extend(codes)\n # Add codes from Lu into L, etc.\n if len(category) > 1:\n short = category[0]\n categories.setdefault(short, []).extend(codes)\n\n ReadUnicodeTable(unicode_dir+\"/UnicodeData.txt\", 15, DoLine)\n return categories", "response": "Returns a dict mapping category names to code lists."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncalculates simple interest of specified entries over time.", "response": "def calculate_simple_interest(entries, rate_pct: Decimal, interest_date: date or None=None, begin: date or None=None) -> Decimal:\n \"\"\"\n Calculates simple interest of specified entries over time.\n Does not accumulate interest to interest.\n :param entries: AccountEntry iterable (e.g. list/QuerySet) ordered by timestamp (ascending)\n :param rate_pct: Interest rate %, e.g. 8.00 for 8%\n :param interest_date: Interest end date. Default is current date.\n :param begin: Optional begin date for the interest. Default is whole range from the timestamp of account entries.\n :return: Decimal accumulated interest\n \"\"\"\n if interest_date is None:\n interest_date = now().date()\n\n bal = None\n cur_date = None\n daily_rate = rate_pct / Decimal(36500)\n accum_interest = Decimal('0.00')\n done = False\n\n entries_list = list(entries)\n nentries = len(entries_list)\n if nentries > 0:\n # make sure we calculate interest over whole range until interest_date\n last = entries_list[nentries-1]\n assert isinstance(last, AccountEntry)\n if last.timestamp.date() < interest_date:\n timestamp = pytz.utc.localize(datetime.combine(interest_date, time(0, 0)))\n e = AccountEntry(timestamp=timestamp, amount=Decimal('0.00'), type=last.type)\n entries_list.append(e)\n\n # initial values from the first account entry\n e = entries_list[0]\n bal = e.amount\n cur_date = e.timestamp.date()\n if begin and begin > cur_date:\n cur_date = begin\n\n for e in entries_list[1:]:\n assert isinstance(e, AccountEntry)\n next_date = e.timestamp.date()\n if begin and begin > next_date:\n next_date = begin\n if next_date > interest_date:\n next_date = interest_date\n done = True\n time_days = (next_date - cur_date).days\n if time_days > 0:\n day_interest = bal * daily_rate\n interval_interest = day_interest * Decimal(time_days)\n accum_interest += interval_interest\n cur_date = next_date\n bal += e.amount\n if done:\n break\n\n return accum_interest"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _format_templates(name, command, templates):\n ''' Creates a list-table directive\n\n for a set of defined environment variables\n\n Parameters:\n name (str):\n The name of the config section\n command (object):\n The sdss_access path instance\n templates (dict):\n A dictionary of the path templates\n\n Yields:\n A string rst-formated list-table directive\n\n '''\n\n yield '.. list-table:: {0}'.format(name)\n yield _indent(':widths: 20 50 70')\n yield _indent(':header-rows: 1')\n yield ''\n yield _indent('* - Name')\n yield _indent(' - Template')\n yield _indent(' - Kwargs')\n for key, var in templates.items():\n kwargs = command.lookup_keys(key)\n yield _indent('* - {0}'.format(key))\n yield _indent(' - {0}'.format(var))\n yield _indent(' - {0}'.format(', '.join(kwargs)))\n yield ''", "response": "Generates a list - table directive that can be used to create a set of environment variables that are defined in the templates dictionary."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _generate_nodes(self, name, command, templates=None):\n\n # the source name\n source_name = name\n\n # Title\n section = nodes.section(\n '',\n nodes.title(text=name),\n ids=[nodes.make_id(name)],\n names=[nodes.fully_normalize_name(name)])\n\n # Summarize\n result = statemachine.ViewList()\n\n if templates:\n lines = _format_templates(name, command, command.templates)\n\n for line in lines:\n result.append(line, source_name)\n\n self.state.nested_parse(result, 0, section)\n\n return [section]", "response": "Generates the relevant Sphinx nodes."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _Delta(a, b):\n if a+1 == b:\n if a%2 == 0:\n return 'EvenOdd'\n else:\n return 'OddEven'\n if a == b+1:\n if a%2 == 0:\n return 'OddEven'\n else:\n return 'EvenOdd'\n return b - a", "response": "Compute the delta for a and b."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a + delta handling EvenOdd and OddEven specially.", "response": "def _AddDelta(a, delta):\n \"\"\"Return a + delta, handling EvenOdd and OddEven specially.\"\"\"\n if type(delta) == int:\n return a+delta\n if delta == 'EvenOdd':\n if a%2 == 0:\n return a+1\n else:\n return a-1\n if delta == 'OddEven':\n if a%2 == 1:\n return a+1\n else:\n return a-1\n print >>sys.stderr, \"Bad Delta: \", delta\n raise \"Bad Delta\""} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _MakeRanges(pairs):\n ranges = []\n last = -100\n\n def evenodd(last, a, b, r):\n if a != last+1 or b != _AddDelta(a, r[2]):\n return False\n r[1] = a\n return True\n\n def evenoddpair(last, a, b, r):\n if a != last+2:\n return False\n delta = r[2]\n d = delta\n if type(delta) is not str:\n return False\n if delta.endswith('Skip'):\n d = delta[:-4]\n else:\n delta = d + 'Skip'\n if b != _AddDelta(a, d):\n return False\n r[1] = a\n r[2] = delta\n return True\n\n for a, b in pairs:\n if ranges and evenodd(last, a, b, ranges[-1]):\n pass\n elif ranges and evenoddpair(last, a, b, ranges[-1]):\n pass\n else:\n ranges.append([a, a, _Delta(a, b)])\n last = a\n return ranges", "response": "Turn a list like [ 65 98... + 32 ) into a list like [ 65 90 + 32 )."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nalign a list of lines so that all rows in single column match. Columns separated by '|' in every line.", "response": "def align_lines(lines: list, column_separator: str='|') -> list:\n \"\"\"\n Pads lines so that all rows in single column match. Columns separated by '|' in every line.\n :param lines: list of lines\n :param column_separator: column separator. default is '|'\n :return: list of lines\n \"\"\"\n rows = []\n col_len = []\n for line in lines:\n line = str(line)\n cols = []\n for col_index, col in enumerate(line.split(column_separator)):\n col = str(col).strip()\n cols.append(col)\n if col_index >= len(col_len):\n col_len.append(0)\n col_len[col_index] = max(col_len[col_index], len(col))\n rows.append(cols)\n\n lines_out = []\n for row in rows:\n cols_out = []\n for col_index, col in enumerate(row):\n if col_index == 0:\n col = col.ljust(col_len[col_index])\n else:\n col = col.rjust(col_len[col_index])\n cols_out.append(col)\n lines_out.append(' '.join(cols_out))\n return lines_out"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nmark invoices with as un - sent.", "response": "def resend_invoices(modeladmin, request: HttpRequest, queryset: QuerySet):\n \"\"\"\n Marks invoices with as un-sent.\n :param modeladmin:\n :param request:\n :param queryset:\n :return:\n \"\"\"\n user = request.user\n assert isinstance(user, User)\n for obj in queryset:\n assert isinstance(obj, Invoice)\n admin_log([obj, user], 'Invoice id={invoice} marked for re-sending'.format(invoice=obj.id), who=user)\n queryset.update(sent=None)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncleans the entries of the derived account.", "response": "def clean_entries(self, source_invoice: Invoice or None, settled_invoice: Invoice or None, account: Account, **kw):\n \"\"\"\n This needs to be called from a derived class clean().\n :param source_invoice:\n :param settled_invoice:\n :param account:\n :return: None\n \"\"\"\n for form in self.forms:\n obj = form.instance\n assert isinstance(obj, AccountEntry)\n obj.account = account\n obj.source_invoice = source_invoice\n obj.settled_invoice = settled_invoice\n if obj.parent:\n if obj.amount is None:\n obj.amount = obj.parent.amount\n if obj.type is None:\n obj.type = obj.parent.type\n if obj.amount > obj.parent.amount > Decimal(0) or obj.amount < obj.parent.amount < Decimal(0):\n raise ValidationError(_('Derived account entry amount cannot be larger than original'))\n for k, v in kw.items():\n setattr(obj, k, v)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _format_date(self, obj) -> str:\n if obj is None:\n return ''\n if isinstance(obj, datetime):\n obj = obj.date()\n return date_format(obj, 'SHORT_DATE_FORMAT')", "response": "Short date format.\n :param obj: date or datetime or None\n :return: str"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef sum_queryset(qs: QuerySet, key: str= 'amount', default=Decimal(0)) -> Decimal:\n res = qs.aggregate(b=Sum(key))['b']\n return default if res is None else res", "response": "Returns aggregate sum of queryset amount field."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nbuild a blueprint that contains the endpoints for callback URLs of . Only call this once per instance.", "response": "def build_blueprint(self, url_prefix=''):\n \"\"\"Build a blueprint that contains the endpoints for callback URLs of\n the current subscriber. Only call this once per instance. Arguments:\n\n - url_prefix; this allows you to prefix the callback URLs in your app.\n\n \"\"\"\n self.blueprint_name, self.blueprint = build_blueprint(self, url_prefix)\n return self.blueprint"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef unsubscribe(self, callback_id):\n request = self.get_active_subscription(callback_id)\n request['mode'] = 'unsubscribe'\n self.subscribe_impl(callback_id, **request)", "response": "Cancel the subscription for the given callback_id and delete the entry from the local database."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef renew_close_to_expiration(self, margin_in_seconds=A_DAY):\n subscriptions = self.storage.close_to_expiration(margin_in_seconds)\n for subscription in subscriptions:\n try:\n self.subscribe_impl(**subscription)\n except SubscriberError as e:\n warn(RENEW_FAILURE % (subscription['topic_url'],\n subscription['callback_id']), e)", "response": "Automatically renews subscriptions that are close to expiring or have already expired."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndiscover the hub url and topic url of a given url.", "response": "def discover(url, timeout=None):\n \"\"\"Discover the hub url and topic url of a given url. Firstly, by inspecting\n the page's headers, secondarily by inspecting the content for link tags.\n\n timeout determines how long to wait for the url to load. It defaults to 3.\n\n \"\"\"\n resp = get_content({'REQUEST_TIMEOUT': timeout}, url)\n\n parser = LinkParser()\n parser.hub_url = (resp.links.get('hub') or {}).get('url')\n parser.topic_url = (resp.links.get('self') or {}).get('url')\n try:\n parser.updated()\n for chunk in resp.iter_content(chunk_size=None, decode_unicode=True):\n parser.feed(chunk)\n parser.close()\n except Finished:\n return {'hub_url': parser.hub_url, 'topic_url': parser.topic_url}\n\n raise DiscoveryError(\"Could not find hub url in topic page\")"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef open_tls_file(file_path, mode, private=True):\n\n containing_dir = os.path.dirname(file_path)\n fh = None\n try:\n if 'w' in mode:\n os.chmod(containing_dir, mode=0o755)\n fh = open(file_path, mode)\n except OSError as e:\n if 'w' in mode:\n os.makedirs(containing_dir, mode=0o755, exist_ok=True)\n os.chmod(containing_dir, mode=0o755)\n fh = open(file_path, 'w')\n else:\n raise\n yield fh\n mode = 0o600 if private else 0o644\n os.chmod(file_path, mode=mode)\n fh.close()", "response": "Context to ensure correct file permissions for certs and directories"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nloads from a file and return an x509 object", "response": "def load(self):\n \"\"\"Load from a file and return an x509 object\"\"\"\n\n private = self.is_private()\n with open_tls_file(self.file_path, 'r', private=private) as fh:\n if private:\n self.x509 = crypto.load_privatekey(self.encoding, fh.read())\n else:\n self.x509 = crypto.load_certificate(self.encoding, fh.read())\n return self.x509"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef save(self, x509):\n\n self.x509 = x509\n with open_tls_file(self.file_path, 'w',\n private=self.is_private()) as fh:\n fh.write(str(self))", "response": "Persist this x509 object to disk"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _setup_tls_files(self, files):\n\n for file_type in TLSFileType:\n if file_type.value in files:\n file_path = files[file_type.value]\n setattr(self, file_type.value,\n TLSFile(file_path, file_type=file_type))", "response": "Initializes the TLSFIle objects with the paths given to this bundle"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef save_x509s(self, x509s):\n\n for file_type in TLSFileType:\n if file_type.value in x509s:\n x509 = x509s[file_type.value]\n if file_type is not TLSFileType.CA:\n # persist this key or cert to disk\n tlsfile = getattr(self, file_type.value)\n if tlsfile:\n tlsfile.save(x509)", "response": "Saves the x509 objects to the paths known by this bundle"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates a CertStore record from this TLSFileBundle", "response": "def to_record(self):\n \"\"\"Create a CertStore record from this TLSFileBundle\"\"\"\n\n tf_list = [getattr(self, k, None) for k in\n [_.value for _ in TLSFileType]]\n # If a cert isn't defined in this bundle, remove it\n tf_list = filter(lambda x: x, tf_list)\n files = {tf.file_type.value: tf.file_path for tf in tf_list}\n self.record['files'] = files\n return self.record"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef from_record(self, record):\n\n self.record = record\n self._setup_tls_files(self.record['files'])\n return self", "response": "Build a bundle from a CertStore record"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef save(self):\n\n with open(self.store_file_path, 'w') as fh:\n fh.write(json.dumps(self.store, indent=4))", "response": "Write the store dict to a file specified by store_file_path"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load(self):\n\n with open(self.store_file_path, 'r') as fh:\n self.store = json.loads(fh.read())", "response": "Read the store dict from file"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_record(self, common_name):\n\n try:\n record = self.store[common_name]\n return record\n except KeyError as e:\n raise CertNotFoundError(\n \"Unable to find record of {name}\"\n .format(name=common_name), errors=e)", "response": "Return the record associated with this common name."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_files(self, common_name):\n\n record = self.get_record(common_name)\n return TLSFileBundle(common_name).from_record(record)", "response": "Return a bundle of TLS files associated with a common name"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nadds a set of files comprising a certificate to Certipy.", "response": "def add_files(self, common_name, x509s, files=None, parent_ca='',\n is_ca=False, signees=None, serial=0, overwrite=False):\n \"\"\"Add a set files comprising a certificate to Certipy\n\n Used with all the defaults, Certipy will manage creation of file paths\n to be used to store these files to disk and automatically calls save\n on all TLSFiles that it creates (and where it makes sense to).\n \"\"\"\n\n if common_name in self.store and not overwrite:\n raise CertExistsError(\n \"Certificate {name} already exists!\"\n \" Set overwrite=True to force add.\"\n .format(name=common_name))\n elif common_name in self.store and overwrite:\n record = self.get_record(common_name)\n serial = int(record['serial'])\n record['serial'] = serial + 1\n TLSFileBundle(common_name).from_record(record).save_x509s(x509s)\n else:\n file_base_tmpl = \"{prefix}/{cn}/{cn}\"\n file_base = file_base_tmpl.format(\n prefix=self.containing_dir, cn=common_name\n )\n try:\n ca_record = self.get_record(parent_ca)\n ca_file = ca_record['files']['cert']\n except CertNotFoundError:\n ca_file = ''\n files = files or {\n 'key': file_base + '.key',\n 'cert': file_base + '.crt',\n 'ca': ca_file,\n }\n bundle = TLSFileBundle(\n common_name, files=files, x509s=x509s, is_ca=is_ca,\n serial=serial, parent_ca=parent_ca, signees=signees)\n self.store[common_name] = bundle.to_record()\n self.save()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nremove a signee_name from the signee list for ca_name", "response": "def remove_sign_link(self, ca_name, signee_name):\n \"\"\"Removes signee_name to the signee list for ca_name\"\"\"\n\n ca_record = self.get_record(ca_name)\n signee_record = self.get_record(signee_name)\n signees = ca_record['signees'] or {}\n signees = Counter(signees)\n if signee_name in signees:\n signees[signee_name] = 0\n ca_record['signees'] = signees\n signee_record['parent_ca'] = ''\n self.save()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update_record(self, common_name, **fields):\n\n record = self.get_record(common_name)\n if fields is not None:\n for field, value in fields:\n record[field] = value\n self.save()\n return record", "response": "Update fields in an existing record"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef remove_record(self, common_name):\n\n bundle = self.get_files(common_name)\n num_signees = len(Counter(bundle.record['signees']))\n if bundle.is_ca() and num_signees > 0:\n raise CertificateAuthorityInUseError(\n \"Authority {name} has signed {x} certificates\"\n .format(name=common_name, x=num_signees)\n )\n try:\n ca_name = bundle.record['parent_ca']\n ca_record = self.get_record(ca_name)\n self.remove_sign_link(ca_name, common_name)\n except CertNotFoundError:\n pass\n record_copy = dict(self.store[common_name])\n del self.store[common_name]\n self.save()\n return record_copy", "response": "Delete the record associated with this common name"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef remove_files(self, common_name, delete_dir=False):\n\n record = self.remove_record(common_name)\n if delete_dir:\n delete_dirs = []\n if 'files' in record:\n key_containing_dir = os.path.dirname(record['files']['key'])\n delete_dirs.append(key_containing_dir)\n cert_containing_dir = os.path.dirname(record['files']['cert'])\n if key_containing_dir != cert_containing_dir:\n delete_dirs.append(cert_containing_dir)\n for d in delete_dirs:\n shutil.rmtree(d)\n return record", "response": "Delete files and record associated with this common name"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_key_pair(self, cert_type, bits):\n\n pkey = crypto.PKey()\n pkey.generate_key(cert_type, bits)\n return pkey", "response": "Create a public or private key pair."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngenerates a certificate given a certificate request", "response": "def sign(self, req, issuer_cert_key, validity_period, digest=\"sha256\",\n extensions=None, serial=0):\n \"\"\"\n Generate a certificate given a certificate request.\n\n Arguments: req - Certificate request to use\n issuer_cert - The certificate of the issuer\n issuer_key - The private key of the issuer\n not_before - Timestamp (relative to now) when the\n certificate starts being valid\n not_after - Timestamp (relative to now) when the\n certificate stops being valid\n digest - Digest method to use for signing,\n default is sha256\n Returns: The signed certificate in an X509 object\n \"\"\"\n\n issuer_cert, issuer_key = issuer_cert_key\n not_before, not_after = validity_period\n cert = crypto.X509()\n cert.set_serial_number(serial)\n cert.gmtime_adj_notBefore(not_before)\n cert.gmtime_adj_notAfter(not_after)\n cert.set_issuer(issuer_cert.get_subject())\n cert.set_subject(req.get_subject())\n cert.set_pubkey(req.get_pubkey())\n\n if extensions:\n for ext in extensions:\n if callable(ext):\n ext = ext(cert)\n cert.add_extensions([ext])\n\n cert.sign(issuer_key, digest)\n\n return cert"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a CA bundle to trust only certs defined in names", "response": "def create_ca_bundle_for_names(self, bundle_name, names):\n \"\"\"Create a CA bundle to trust only certs defined in names\n \"\"\"\n\n records = [rec for name, rec\n in self.store.store.items() if name in names]\n return self.create_bundle(\n bundle_name, names=[r['parent_ca'] for r in records])"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates a bundle of public certs for trust distribution.", "response": "def create_bundle(self, bundle_name, names=None, ca_only=True):\n \"\"\"Create a bundle of public certs for trust distribution\n\n This will create a bundle of both CAs and/or regular certificates.\n Arguments: names - The names of certs to include in the bundle\n bundle_name - The name of the bundle file to output\n Returns: Path to the bundle file\n \"\"\"\n\n if not names:\n if ca_only:\n names = []\n for name, record in self.store.store.items():\n if record['is_ca']:\n names.append(name)\n else:\n names = self.store.store.keys()\n\n out_file_path = os.path.join(self.store.containing_dir, bundle_name)\n with open(out_file_path, 'w') as fh:\n for name in names:\n bundle = self.store.get_files(name)\n bundle.cert.load()\n fh.write(str(bundle.cert))\n return out_file_path"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a set of trust bundles from a relationship graph.", "response": "def trust_from_graph(self, graph):\n \"\"\"Create a set of trust bundles from a relationship graph.\n\n Components in this sense are defined by unique CAs. This method assists\n in setting up complicated trust between various components that need\n to do TLS auth.\n Arguments: graph - dict component:list(components)\n Returns: dict component:trust bundle file path\n \"\"\"\n\n # Ensure there are CAs backing all graph components\n def distinct_components(graph):\n \"\"\"Return a set of components from the provided graph.\"\"\"\n components = set(graph.keys())\n for trusts in graph.values():\n components |= set(trusts)\n return components\n\n # Default to creating a CA (incapable of signing intermediaries) to\n # identify a component not known to Certipy\n for component in distinct_components(graph):\n try:\n self.store.get_record(component)\n except CertNotFoundError:\n self.create_ca(component)\n\n # Build bundles from the graph\n trust_files = {}\n for component, trusts in graph.items():\n file_name = component + '_trust.crt'\n trust_files[component] = self.create_bundle(\n file_name, names=trusts, ca_only=False)\n\n return trust_files"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate a new CA and return a KeyCertPair object.", "response": "def create_ca(self, name, ca_name='', cert_type=crypto.TYPE_RSA, bits=2048,\n alt_names=None, years=5, serial=0, pathlen=0,\n overwrite=False):\n \"\"\"\n Create a certificate authority\n\n Arguments: name - The name of the CA\n cert_type - The type of the cert. TYPE_RSA or TYPE_DSA\n bits - The number of bits to use\n alt_names - An array of alternative names in the format:\n IP:address, DNS:address\n Returns: KeyCertPair for the new CA\n \"\"\"\n\n cakey = self.create_key_pair(cert_type, bits)\n req = self.create_request(cakey, CN=name)\n signing_key = cakey\n signing_cert = req\n\n parent_ca = ''\n if ca_name:\n ca_bundle = self.store.get_files(ca_name)\n signing_key = ca_bundle.key.load()\n signing_cert = ca_bundle.cert.load()\n parent_ca = ca_bundle.cert.file_path\n\n basicConstraints = \"CA:true\"\n # If pathlen is exactly 0, this CA cannot sign intermediaries.\n # A negative value leaves this out entirely and allows arbitrary\n # numbers of intermediates.\n if pathlen >=0:\n basicConstraints += ', pathlen:' + str(pathlen)\n\n extensions = [\n crypto.X509Extension(\n b\"basicConstraints\", True, basicConstraints.encode()),\n crypto.X509Extension(\n b\"keyUsage\", True, b\"keyCertSign, cRLSign\"),\n crypto.X509Extension(\n b\"extendedKeyUsage\", True, b\"serverAuth, clientAuth\"),\n lambda cert: crypto.X509Extension(\n b\"subjectKeyIdentifier\", False, b\"hash\", subject=cert),\n lambda cert: crypto.X509Extension(\n b\"authorityKeyIdentifier\", False, b\"keyid:always\",\n issuer=cert),\n ]\n\n if alt_names:\n extensions.append(\n crypto.X509Extension(b\"subjectAltName\",\n False, \",\".join(alt_names).encode())\n )\n\n # TODO: start time before today for clock skew?\n cacert = self.sign(\n req, (signing_cert, signing_key), (0, 60*60*24*365*years),\n extensions=extensions)\n\n x509s = {'key': cakey, 'cert': cacert, 'ca': cacert}\n self.store.add_files(name, x509s, overwrite=overwrite,\n parent_ca=parent_ca, is_ca=True)\n if ca_name:\n self.store.add_sign_link(ca_name, name)\n return self.store.get_record(name)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate a new key - cert pair and return it", "response": "def create_signed_pair(self, name, ca_name, cert_type=crypto.TYPE_RSA,\n bits=2048, years=5, alt_names=None, serial=0,\n overwrite=False):\n \"\"\"\n Create a key-cert pair\n\n Arguments: name - The name of the key-cert pair\n ca_name - The name of the CA to sign this cert\n cert_type - The type of the cert. TYPE_RSA or TYPE_DSA\n bits - The number of bits to use\n alt_names - An array of alternative names in the format:\n IP:address, DNS:address\n Returns: KeyCertPair for the new signed pair\n \"\"\"\n\n key = self.create_key_pair(cert_type, bits)\n req = self.create_request(key, CN=name)\n extensions = [\n crypto.X509Extension(\n b\"extendedKeyUsage\", True, b\"serverAuth, clientAuth\"),\n ]\n\n if alt_names:\n extensions.append(\n crypto.X509Extension(b\"subjectAltName\",\n False, \",\".join(alt_names).encode())\n )\n\n ca_bundle = self.store.get_files(ca_name)\n cacert = ca_bundle.cert.load()\n cakey = ca_bundle.key.load()\n\n cert = self.sign(req, (cacert, cakey), (0, 60*60*24*365*years),\n extensions=extensions)\n\n x509s = {'key': key, 'cert': cert, 'ca': None}\n self.store.add_files(name, x509s, parent_ca=ca_name,\n overwrite=overwrite)\n\n # Relate these certs as being parent and child\n self.store.add_sign_link(ca_name, name)\n return self.store.get_record(name)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef subscribe(hub, callback_url, topic_url, lease_seconds, secret,\n endpoint_hook_data):\n \"\"\"5.2 Subscription Validation\"\"\"\n\n for validate in hub.validators:\n error = validate(callback_url, topic_url, lease_seconds, secret,\n endpoint_hook_data)\n if error:\n send_denied(hub, callback_url, topic_url, error)\n return\n\n if intent_verified(hub, callback_url, 'subscribe', topic_url,\n lease_seconds):\n hub.storage[topic_url, callback_url] = {\n 'lease_seconds': lease_seconds,\n 'secret': secret,\n }", "response": "5. 2 Subscription Validation"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef init_celery(self, celery):\n\n count = next(self.counter)\n\n def task_with_hub(f, **opts):\n @functools.wraps(f)\n def wrapper(*args, **kwargs):\n return f(self, *args, **kwargs)\n # Make sure newer instances don't overwride older ones.\n wrapper.__name__ = wrapper.__name__ + '_' + str(count)\n return celery.task(**opts)(wrapper)\n\n # tasks for internal use:\n self.subscribe = task_with_hub(subscribe)\n self.unsubscribe = task_with_hub(unsubscribe)\n\n max_attempts = self.config.get('MAX_ATTEMPTS', 10)\n make_req = task_with_hub(make_request_retrying, bind=True,\n max_retries=max_attempts)\n self.make_request_retrying = make_req\n\n # user facing tasks\n\n # wrapped by send_change_notification:\n self.send_change = task_with_hub(send_change_notification)\n\n # wrapped by cleanup_expired_subscriptions\n @task_with_hub\n def cleanup(hub):\n self.storage.cleanup_expired_subscriptions()\n self.cleanup = cleanup\n\n # wrapped by schedule_cleanup\n def schedule(every_x_seconds=A_DAY):\n celery.add_periodic_task(every_x_seconds,\n self.cleanup_expired_subscriptions.s())\n self.schedule = schedule", "response": "Registers the celery tasks on the hub object."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef init_publisher(app):\n @app.context_processor\n def inject_links():\n return {\n 'websub_self_url': stack.top.websub_self_url,\n 'websub_hub_url': stack.top.websub_hub_url,\n 'websub_self_link': stack.top.websub_self_link,\n 'websub_hub_link': stack.top.websub_hub_link,\n }", "response": "This is a decorator that is used to initialize the available publisher."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef publisher(self_url=None, hub_url=None):\n def decorator(topic_view):\n @functools.wraps(topic_view)\n def wrapper(*args, **kwargs):\n nonlocal hub_url, self_url\n\n if not self_url:\n self_url = request.url\n if not hub_url:\n try:\n hub_url = url_for('websub_hub.endpoint', _external=True)\n except BuildError:\n hub_url = current_app.config['HUB_URL']\n\n stack.top.websub_self_url = self_url\n stack.top.websub_hub_url = hub_url\n stack.top.websub_self_link = Markup(SELF_LINK % self_url)\n stack.top.websub_hub_link = Markup(HUB_LINK % hub_url)\n\n resp = make_response(topic_view(*args, **kwargs))\n resp.headers.add('Link', HEADER_VALUE % (self_url, hub_url))\n return resp\n return wrapper\n return decorator", "response": "This decorator makes it easier to implement a websub publisher."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting the count of records matching the query.", "response": "def count_results(cube, q):\n \"\"\" Get the count of records matching the query. \"\"\"\n q = select(columns=[func.count(True)], from_obj=q.alias())\n return cube.engine.execute(q).scalar()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef generate_results(cube, q):\n if q._limit is not None and q._limit < 1:\n return\n rp = cube.engine.execute(q)\n while True:\n row = rp.fetchone()\n if row is None:\n return\n yield dict(row.items())", "response": "Generate the results for this query."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsort on a set of field specifications of the type ref and direction.", "response": "def apply(self, q, bindings, ordering, distinct=None):\n \"\"\" Sort on a set of field specifications of the type (ref, direction)\n in order of the submitted list. \"\"\"\n info = []\n for (ref, direction) in self.parse(ordering):\n info.append((ref, direction))\n table, column = self.cube.model[ref].bind(self.cube)\n if distinct is not None and distinct != ref:\n column = asc(ref) if direction == 'asc' else desc(ref)\n else:\n column = column.label(column.name)\n column = column.asc() if direction == 'asc' else column.desc()\n bindings.append(Binding(table, ref))\n if self.cube.is_postgresql:\n column = column.nullslast()\n q = q.order_by(column)\n\n if not len(self.results):\n for column in q.columns:\n column = column.asc()\n if self.cube.is_postgresql:\n column = column.nullslast()\n q = q.order_by(column)\n return info, q, bindings"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a deep copy of the object with all of the numpy arrays copied", "response": "def copy(self, only=None):\n '''\n Returns a deep copy with all of the numpy arrays copied\n If only is a list of strings, i.e. ['f', 'v'], then only those properties will be copied\n '''\n if only is None:\n return Mesh(self)\n else:\n import copy\n m = Mesh()\n for a in only:\n setattr(m, a, copy.deepcopy(getattr(self, a)))\n return m"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconcatenates an arbitrary number of meshes.", "response": "def concatenate(cls, *args):\n \"\"\"Concatenates an arbitrary number of meshes.\n\n Currently supports vertices, vertex colors, and faces.\n \"\"\"\n nargs = len(args)\n if nargs == 1:\n return args[0]\n\n vs = [a.v for a in args if a.v is not None]\n vcs = [a.vc for a in args if a.vc is not None]\n fs = [a.f for a in args if a.f is not None]\n\n if vs and len(vs) != nargs:\n raise ValueError('Expected `v` for all args or none.')\n if vcs and len(vcs) != nargs:\n raise ValueError('Expected `vc` for all args or none.')\n if fs and len(fs) != nargs:\n raise ValueError('Expected `f` for all args or none.')\n\n # Offset face indices by the cumulative vertex count.\n face_offsets = np.cumsum([v.shape[0] for v in vs[:-1]])\n for offset, f in zip(face_offsets, fs[1:]): # https://bitbucket.org/logilab/pylint/issues/603/operator-generates-false-positive-unused pylint: disable=unused-variable\n f += offset\n\n return Mesh(\n v=np.vstack(vs) if vs else None,\n vc=np.vstack(vcs) if vcs else None,\n f=np.vstack(fs) if fs else None,\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef getNewConnection(*args, **kargs):\n\tkargs = dict(kargs)\n\tif len(args) > 0:\n\t\tif len(args) >= 1:\n\t\t\tkargs['host'] = args[0]\n\t\tif len(args) >= 2:\n\t\t\tkargs['user'] = args[1]\n\t\tif len(args) >= 3:\n\t\t\tkargs['passwd'] = args[2]\n\t\tif len(args) >= 4:\n\t\t\tkargs['db'] = args[3]\n\t\tif len(args) >= 5:\n\t\t\tkargs['port'] = args[4]\n\t\tif len(args) >= 6:\n\t\t\tkargs['commitOnEnd'] = args[5]\n\treturn connection.Connection(*args, **kargs)", "response": "Quickly Create a new connection object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef getNewQuery(connection = None, commitOnEnd=False, *args, **kargs):\n\tif connection is None:\n\t\treturn query.PySQLQuery(getNewConnection(*args, **kargs), commitOnEnd = commitOnEnd)\n\telse:\n\t\t#Updated 7/24/08 to include commitOnEnd here\n\t\t#-Chandler Prall\n\t\treturn query.PySQLQuery(connection, commitOnEnd = commitOnEnd)", "response": "Create a new PySQLQuery Class\n\t"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nencrypting passwords and environment variables for use with Travis CI. Travis Encrypt requires as arguments the user's GitHub username and repository name. Once the arguments are passed, a password prompt will ask for the password that needs to be encrypted. The given password will then be encrypted via the PKCS1v15 padding scheme and printed to standard output. If the path to a .travis.yml file is given as an argument, the encrypted password is added to the .travis.yml file.", "response": "def cli(username, repository, path, password, deploy, env, clipboard, env_file):\n \"\"\"Encrypt passwords and environment variables for use with Travis CI.\n\n Travis Encrypt requires as arguments the user's GitHub username and repository name.\n Once the arguments are passed, a password prompt will ask for the password that needs\n to be encrypted. The given password will then be encrypted via the PKCS1v15 padding\n scheme and printed to standard output. If the path to a .travis.yml file\n is given as an argument, the encrypted password is added to the .travis.yml file.\n \"\"\"\n key = retrieve_public_key('{}/{}' .format(username, repository))\n\n if env_file:\n if path:\n config = load_travis_configuration(path)\n\n for env_var, value in dotenv_values(env_file).items():\n encrypted_env = encrypt_key(key, value.encode())\n config.setdefault('env', {}).setdefault('global', {})[env_var] = {'secure': encrypted_env}\n dump_travis_configuration(config, path)\n print('Encrypted variables from {} added to {}'.format(env_file, path))\n else:\n print('\\nPlease add the following to your .travis.yml:')\n for env_var, value in dotenv_values(env_file).items():\n encrypted_env = encrypt_key(key, value.encode())\n print(\"{}:\\n secure: {}\".format(env_var, encrypted_env))\n else:\n encrypted_password = encrypt_key(key, password.encode())\n\n if path:\n config = load_travis_configuration(path)\n if config is None:\n config = OrderedDict()\n\n if deploy:\n config.setdefault('deploy', {}).setdefault('password', {})['secure'] = encrypted_password\n\n elif env:\n try:\n config.setdefault('env', {}).setdefault('global', {})['secure'] = encrypted_password\n except TypeError:\n for item in config['env']['global']:\n if isinstance(item, dict) and 'secure' in item:\n item['secure'] = encrypted_password\n\n else:\n config.setdefault('password', {})['secure'] = encrypted_password\n\n dump_travis_configuration(config, path)\n\n print('Encrypted password added to {}' .format(path))\n elif clipboard:\n pyperclip.copy(encrypted_password)\n print('\\nThe encrypted password has been copied to your clipboard.')\n else:\n print('\\nPlease add the following to your .travis.yml:\\nsecure: {}' .format(encrypted_password))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a list of fields that can be used to aggregate the query.", "response": "def apply(self, q, bindings, fields, distinct=False):\n \"\"\" Define a set of fields to return for a non-aggregated query. \"\"\"\n info = []\n\n group_by = None\n\n for field in self.parse(fields):\n for concept in self.cube.model.match(field):\n info.append(concept.ref)\n table, column = concept.bind(self.cube)\n bindings.append(Binding(table, concept.ref))\n if distinct:\n if group_by is None:\n q = q.group_by(column)\n group_by = column\n else:\n min_column = func.max(column)\n min_column = min_column.label(column.name)\n column = min_column\n q = q.column(column)\n\n if not len(self.results):\n # If no fields are requested, return all available fields.\n for concept in list(self.cube.model.attributes) + \\\n list(self.cube.model.measures):\n info.append(concept.ref)\n table, column = concept.bind(self.cube)\n bindings.append(Binding(table, concept.ref))\n q = q.column(column)\n\n return info, q, bindings"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndrags - drag the mouse to the new point", "response": "def drag(self, NewPt):\n # //Mouse drag, calculate rotation (Point2fT Quat4fT)\n \"\"\" drag (Point2fT mouse_coord) -> new_quaternion_rotation_vec\n \"\"\"\n X = 0\n Y = 1\n Z = 2\n W = 3\n\n self.m_EnVec = self._mapToSphere(NewPt)\n\n # //Compute the vector perpendicular to the begin and end vectors\n # Perp = Vector3fT()\n Perp = Vector3fCross(self.m_StVec, self.m_EnVec)\n\n NewRot = Quat4fT()\n # //Compute the length of the perpendicular vector\n if Vector3fLength(Perp) > Epsilon: # //if its non-zero\n # //We're ok, so return the perpendicular vector as the transform after all\n NewRot[X] = Perp[X]\n NewRot[Y] = Perp[Y]\n NewRot[Z] = Perp[Z]\n # //In the quaternion values, w is cosine(theta / 2), where theta is rotation angle\n NewRot[W] = Vector3fDot(self.m_StVec, self.m_EnVec)\n else: # //if its zero\n # //The begin and end vectors coincide, so return a quaternion of zero matrix (no rotation)\n NewRot[X] = NewRot[Y] = NewRot[Z] = NewRot[W] = 0.0\n\n return NewRot"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_cell_volumes(self):\n self.cell_volumes = numpy.array(\n [\n abs(\n self.node_coords[cell[\"nodes\"]][1]\n - self.node_coords[cell[\"nodes\"]][0]\n )\n for cell in self.cells\n ]\n )\n return", "response": "Computes the volumes of the cells in the mesh."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef create_control_volumes(self):\n self.control_volumes = numpy.zeros(len(self.node_coords), dtype=float)\n for k, cell in enumerate(self.cells):\n node_ids = cell[\"nodes\"]\n self.control_volumes[node_ids] += 0.5 * self.cell_volumes[k]\n\n # Sanity checks.\n sum_cv = sum(self.control_volumes)\n sum_cells = sum(self.cell_volumes)\n alpha = sum_cv - sum_cells\n assert abs(alpha) < 1.0e-6\n return", "response": "Compute the control volumes of all nodes in the mesh."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef create_rectangular_prism(origin, size):\n '''\n Return a Mesh which is an axis-aligned rectangular prism. One vertex is\n `origin`; the diametrically opposite vertex is `origin + size`.\n\n size: 3x1 array.\n\n '''\n from lace.topology import quads_to_tris\n\n lower_base_plane = np.array([\n # Lower base plane\n origin,\n origin + np.array([size[0], 0, 0]),\n origin + np.array([size[0], 0, size[2]]),\n origin + np.array([0, 0, size[2]]),\n ])\n upper_base_plane = lower_base_plane + np.array([0, size[1], 0])\n\n vertices = np.vstack([lower_base_plane, upper_base_plane])\n\n faces = quads_to_tris(np.array([\n [0, 1, 2, 3], # lower base (-y)\n [7, 6, 5, 4], # upper base (+y)\n [4, 5, 1, 0], # +z face\n [5, 6, 2, 1], # +x face\n [6, 7, 3, 2], # -z face\n [3, 7, 4, 0], # -x face\n ]))\n\n return Mesh(v=vertices, f=faces)", "response": "Create a rectangular prism."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create_triangular_prism(p1, p2, p3, height):\n '''\n Return a Mesh which is a triangular prism whose base is the triangle\n p1, p2, p3. If the vertices are oriented in a counterclockwise\n direction, the prism extends from behind them.\n\n '''\n from blmath.geometry import Plane\n\n base_plane = Plane.from_points(p1, p2, p3)\n lower_base_to_upper_base = height * -base_plane.normal # pylint: disable=invalid-unary-operand-type\n vertices = np.vstack(([p1, p2, p3], [p1, p2, p3] + lower_base_to_upper_base))\n\n faces = np.array([\n [0, 1, 2], # base\n [0, 3, 4], [0, 4, 1], # side 0, 3, 4, 1\n [1, 4, 5], [1, 5, 2], # side 1, 4, 5, 2\n [2, 5, 3], [2, 3, 0], # side 2, 5, 3, 0\n [5, 4, 3], # base\n ])\n\n return Mesh(v=vertices, f=faces)", "response": "Create a triangular prism from two points."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create_horizontal_plane():\n '''\n Creates a horizontal plane.\n '''\n v = np.array([\n [1., 0., 0.],\n [-1., 0., 0.],\n [0., 0., 1.],\n [0., 0., -1.]\n ])\n f = [[0, 1, 2], [3, 1, 0]]\n return Mesh(v=v, f=f)", "response": "Creates a horizontal plane."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef match_ref(self, ref):\n if ref in self.refs:\n self._matched_ref = ref\n return True\n return False", "response": "Check if the ref matches one of the concept s aliases."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the SQLAlchemy Table object matching a given possibly - named column name.", "response": "def _physical_column(self, cube, column_name):\n \"\"\" Return the SQLAlchemy Column object matching a given, possibly\n qualified, column name (i.e.: 'table.column'). If no table is named,\n the fact table is assumed. \"\"\"\n table_name = self.model.fact_table_name\n if '.' in column_name:\n table_name, column_name = column_name.split('.', 1)\n table = cube._load_table(table_name)\n if column_name not in table.columns:\n raise BindingException('Column %r does not exist on table %r' % (\n column_name, table_name), table=table_name,\n column=column_name)\n return table, table.columns[column_name]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nmap a model reference to a physical column in the database.", "response": "def bind(self, cube):\n \"\"\" Map a model reference to an physical column in the database. \"\"\"\n table, column = self._physical_column(cube, self.column_name)\n column = column.label(self.matched_ref)\n column.quote = True\n return table, column"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nadd a new source token to the customer.", "response": "def add_new_source(self, source_token, stripe_js_response=None):\n \"\"\"Add new source (for example: a new card) to the customer\n\n The new source will be automatically set as customer's default payment source.\n Passing stripe_js_response is optional. If set, StripeCustomer.stripe_js_response will be updated.\n \"\"\"\n customer = self.retrieve_from_stripe()\n customer.source = source_token\n customer.save()\n if stripe_js_response:\n self.stripe_js_response = stripe_js_response\n self._update_from_stripe_object(customer)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef update_from_stripe_data(self, stripe_coupon, exclude_fields=None, commit=True):\n fields_to_update = self.STRIPE_FIELDS - set(exclude_fields or [])\n update_data = {key: stripe_coupon[key] for key in fields_to_update}\n for field in [\"created\", \"redeem_by\"]:\n if update_data.get(field):\n update_data[field] = timestamp_to_timezone_aware_date(update_data[field])\n\n if update_data.get(\"amount_off\"):\n update_data[\"amount_off\"] = Decimal(update_data[\"amount_off\"]) / 100\n\n # also make sure the object is up to date (without the need to call database)\n for key, value in six.iteritems(update_data):\n setattr(self, key, value)\n\n if commit:\n return StripeCoupon.objects.filter(pk=self.pk).update(**update_data)", "response": "Update the object with data from stripe. Coupon without calling StripeCoupon. retrieve."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef save(self, force_retrieve=False, *args, **kwargs):\n stripe.api_key = stripe_settings.API_KEY\n if self._previous_is_deleted != self.is_deleted and self.is_deleted:\n try:\n stripe_coupon = stripe.Coupon.retrieve(self.coupon_id)\n # make sure to delete correct coupon\n if self.created == timestamp_to_timezone_aware_date(stripe_coupon[\"created\"]):\n stripe_coupon.delete()\n except stripe.error.InvalidRequestError:\n # means that the coupon has already been removed from stripe\n pass\n\n return super(StripeCoupon, self).save(*args, **kwargs)\n\n if self.pk or force_retrieve:\n try:\n stripe_coupon = stripe.Coupon.retrieve(self.coupon_id)\n if not force_retrieve:\n stripe_coupon.metadata = self.metadata\n stripe_coupon.save()\n\n if force_retrieve:\n # make sure we are not creating a duplicate\n coupon_qs = StripeCoupon.objects.filter(coupon_id=self.coupon_id)\n if coupon_qs.filter(created=timestamp_to_timezone_aware_date(stripe_coupon[\"created\"])).exists():\n raise StripeCouponAlreadyExists\n\n # all old coupons should be deleted\n for coupon in coupon_qs:\n coupon.is_deleted = True\n super(StripeCoupon, coupon).save() # use super save() to call pre/post save signals\n # update all fields in the local object in case someone tried to change them\n self.update_from_stripe_data(stripe_coupon, exclude_fields=[\"metadata\"] if not force_retrieve else [])\n self.stripe_response = stripe_coupon\n except stripe.error.InvalidRequestError:\n if force_retrieve:\n raise\n\n self.is_deleted = True\n else:\n self.stripe_response = stripe.Coupon.create(\n id=self.coupon_id,\n duration=self.duration,\n amount_off=int(self.amount_off * 100) if self.amount_off else None,\n currency=self.currency,\n duration_in_months=self.duration_in_months,\n max_redemptions=self.max_redemptions,\n metadata=self.metadata,\n percent_off=self.percent_off,\n redeem_by=int(dateformat.format(self.redeem_by, \"U\")) if self.redeem_by else None\n )\n # stripe will generate coupon_id if none was specified in the request\n if not self.coupon_id:\n self.coupon_id = self.stripe_response[\"id\"]\n\n self.created = timestamp_to_timezone_aware_date(self.stripe_response[\"created\"])\n # for future\n self.is_created_at_stripe = True\n return super(StripeCoupon, self).save(*args, **kwargs)", "response": "Save the coupon to Stripe."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_signed_simplex_volumes(cells, pts):\n n = pts.shape[1]\n assert cells.shape[1] == n + 1\n\n p = pts[cells]\n p = numpy.concatenate([p, numpy.ones(list(p.shape[:2]) + [1])], axis=-1)\n return numpy.linalg.det(p) / math.factorial(n)", "response": "Get the signed volume of a simplex in nD."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ngive a sorted 1D input array `a`, e.g., [0 0, 1, 2, 3, 4, 4, 4], this routine returns the indices where the blocks of equal integers start and how long the blocks are.", "response": "def grp_start_len(a):\n \"\"\"Given a sorted 1D input array `a`, e.g., [0 0, 1, 2, 3, 4, 4, 4], this\n routine returns the indices where the blocks of equal integers start and\n how long the blocks are.\n \"\"\"\n # https://stackoverflow.com/a/50394587/353337\n m = numpy.concatenate([[True], a[:-1] != a[1:], [True]])\n idx = numpy.flatnonzero(m)\n return idx[:-1], numpy.diff(idx)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncomputes the circumcenters of all given triangles.", "response": "def compute_triangle_circumcenters(X, ei_dot_ei, ei_dot_ej):\n \"\"\"Computes the circumcenters of all given triangles.\n \"\"\"\n # The input argument are the dot products\n #\n # \n # \n # \n #\n # of the edges\n #\n # e0: x1->x2,\n # e1: x2->x0,\n # e2: x0->x1.\n #\n # Note that edge e_i is opposite of node i and the edges add up to 0.\n\n # The trilinear coordinates of the circumcenter are\n #\n # cos(alpha0) : cos(alpha1) : cos(alpha2)\n #\n # where alpha_k is the angle at point k, opposite of edge k. The Cartesian\n # coordinates are (see\n # )\n #\n # C = sum_i ||e_i|| * cos(alpha_i)/beta * P_i\n #\n # with\n #\n # beta = sum ||e_i||*cos(alpha_i)\n #\n # Incidentally, the cosines are\n #\n # cos(alpha0) = / ||e1|| / ||e2||,\n #\n # so in total\n #\n # C = / sum_i ( ) P0\n # + ... P1\n # + ... P2.\n #\n # An even nicer formula is given on\n # : The\n # barycentric coordinates of the circumcenter are\n #\n # a^2 (b^2 + c^2 - a^2) : b^2 (c^2 + a^2 - b^2) : c^2 (a^2 + b^2 - c^2).\n #\n # This is only using the squared edge lengths, too!\n #\n alpha = ei_dot_ei * ei_dot_ej\n alpha_sum = alpha[0] + alpha[1] + alpha[2]\n beta = alpha / alpha_sum[None]\n a = X * beta[..., None]\n cc = a[0] + a[1] + a[2]\n\n # alpha = numpy.array([\n # ei_dot_ei[0] * (ei_dot_ei[1] + ei_dot_ei[2] - ei_dot_ei[0]),\n # ei_dot_ei[1] * (ei_dot_ei[2] + ei_dot_ei[0] - ei_dot_ei[1]),\n # ei_dot_ei[2] * (ei_dot_ei[0] + ei_dot_ei[1] - ei_dot_ei[2]),\n # ])\n # alpha /= numpy.sum(alpha, axis=0)\n # cc = (X[0].T * alpha[0] + X[1].T * alpha[1] + X[2].T * alpha[2]).T\n return cc"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets edge mask of the faces which are fully in subdomain.", "response": "def get_edge_mask(self, subdomain=None):\n \"\"\"Get faces which are fully in subdomain.\n \"\"\"\n if subdomain is None:\n # https://stackoverflow.com/a/42392791/353337\n return numpy.s_[:]\n\n if subdomain not in self.subdomains:\n self._mark_vertices(subdomain)\n\n # A face is inside if all its edges are in.\n # An edge is inside if all its nodes are in.\n is_in = self.subdomains[subdomain][\"vertices\"][self.idx_hierarchy]\n # Take `all()` over the first index\n is_inside = numpy.all(is_in, axis=tuple(range(1)))\n\n if subdomain.is_boundary_only:\n # Filter for boundary\n is_inside = is_inside & self.is_boundary_edge\n\n return is_inside"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_face_mask(self, subdomain):\n if subdomain is None:\n # https://stackoverflow.com/a/42392791/353337\n return numpy.s_[:]\n\n if subdomain not in self.subdomains:\n self._mark_vertices(subdomain)\n\n # A face is inside if all its edges are in.\n # An edge is inside if all its nodes are in.\n is_in = self.subdomains[subdomain][\"vertices\"][self.idx_hierarchy]\n # Take `all()` over all axes except the last two (face_ids, cell_ids).\n n = len(is_in.shape)\n is_inside = numpy.all(is_in, axis=tuple(range(n - 2)))\n\n if subdomain.is_boundary_only:\n # Filter for boundary\n is_inside = is_inside & self.is_boundary_facet\n\n return is_inside", "response": "Get faces which are fully in subdomain."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nmarks faces and edges which are fully in subdomain.", "response": "def _mark_vertices(self, subdomain):\n \"\"\"Mark faces/edges which are fully in subdomain.\n \"\"\"\n if subdomain is None:\n is_inside = numpy.ones(len(self.node_coords), dtype=bool)\n else:\n is_inside = subdomain.is_inside(self.node_coords.T).T\n\n if subdomain.is_boundary_only:\n # Filter boundary\n self.mark_boundary()\n is_inside = is_inside & self.is_boundary_node\n\n self.subdomains[subdomain] = {\"vertices\": is_inside}\n return"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndumps a yaml configuration as an OrderedDict.", "response": "def ordered_dump(data, stream=None, Dumper=yaml.SafeDumper, **kwds):\n \"\"\"Dump a yaml configuration as an OrderedDict.\"\"\"\n class OrderedDumper(Dumper):\n pass\n\n def dict_representer(dumper, data):\n return dumper.represent_mapping(yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, data.items())\n\n OrderedDumper.add_representer(OrderedDict, dict_representer)\n\n return yaml.dump(data, stream, OrderedDumper, **kwds)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck whether the type of the cut value matches the type of the resource in the Cube model.", "response": "def _check_type(self, ref, value):\n \"\"\"\n Checks whether the type of the cut value matches the type of the\n concept being cut, and raises a QueryException if it doesn't match\n \"\"\"\n if isinstance(value, list):\n return [self._check_type(ref, val) for val in value]\n\n model_type = self.cube.model[ref].datatype\n if model_type is None:\n return\n query_type = self._api_type(value)\n if query_type == model_type:\n return\n else:\n raise QueryException(\"Invalid value %r parsed as type '%s' \"\n \"for cut %s of type '%s'\"\n % (value, query_type, ref, model_type))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the API type of the given value based on its python type.", "response": "def _api_type(self, value):\n \"\"\"\n Returns the API type of the given value based on its python type.\n\n \"\"\"\n if isinstance(value, six.string_types):\n return 'string'\n elif isinstance(value, six.integer_types):\n return 'integer'\n elif type(value) is datetime.datetime:\n return 'date'"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef apply(self, q, bindings, cuts):\n info = []\n for (ref, operator, value) in self.parse(cuts):\n if map_is_class and isinstance(value, map):\n value = list(value)\n self._check_type(ref, value)\n info.append({'ref': ref, 'operator': operator, 'value': value})\n table, column = self.cube.model[ref].bind(self.cube)\n bindings.append(Binding(table, ref))\n q = q.where(column.in_(value))\n return info, q, bindings", "response": "Apply a set of filters to a query."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef colors_like(color, arr, colormap=DEFAULT_COLORMAP):\n '''\n Given an array of size NxM (usually Nx3), we accept color in the following ways:\n - A string color name. The accepted names are roughly what's in X11's rgb.txt\n - An explicit rgb triple, in (3, ), (3, 1), or (1, 3) shape\n - A list of values (N, ), (N, 1), or (1, N) that are put through a colormap to get per vertex color\n - An array of colors (N, 3) or (3, N)\n\n There is a potential for conflict here if N == 3. In that case we assume a value is an rgb triple,\n not a colormap index. This is a sort of degenerate case, as a mesh with three verticies is just a single\n triangle and not something we ever actually use in practice.\n '''\n import numpy as np\n from blmath.numerics import is_empty_arraylike\n if is_empty_arraylike(color):\n return None\n if isinstance(color, basestring):\n from lace.color_names import name_to_rgb\n color = name_to_rgb[color]\n elif isinstance(color, list):\n color = np.array(color)\n color = np.squeeze(color)\n num_verts = arr.shape[0]\n if color.ndim == 1:\n if color.shape[0] == 3: # rgb triple\n return np.ones((num_verts, 3)) * np.array([color])\n else:\n from matplotlib import cm\n return np.ones((num_verts, 3)) * cm.get_cmap(colormap)(color.flatten())[:, :3]\n elif color.ndim == 2:\n if color.shape[1] == num_verts:\n color = color.T\n return np.ones((num_verts, 3)) * color\n else:\n raise ValueError(\"Colors must be specified as one or two dimensions\")", "response": "Given an array of size NxM we accept color in the following ways."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef color_by_height(self, axis=1, threshold=None, color=DEFAULT_COLORMAP): # pylint: disable=unused-argument\n '''\n Color each vertex by its height above the floor point.\n\n axis: The axis to use. 0 means x, 1 means y, 2 means z.\n threshold: The top of the range. Points at and above this height will\n be the same color.\n\n '''\n import numpy as np\n\n heights = self.v[:, axis] - self.floor_point[axis]\n\n if threshold:\n # height == 0 -> saturation = 0.\n # height == threshold -> saturation = 1.\n color_weights = np.minimum(heights / threshold, 1.)\n color_weights = color_weights * color_weights\n self.set_vertex_colors_from_weights(color_weights, scale_to_range_1=False)\n else:\n self.set_vertex_colors_from_weights(heights)", "response": "Color each vertex by its height above the floor point."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef color_as_heatmap(self, weights, max_weight=1.0, color=DEFAULT_COLORMAP):\n '''\n Truncate weights to the range [0, max_weight] and rescale, as you would\n want for a heatmap with known scale.\n '''\n import numpy as np\n adjusted_weights = np.clip(weights, 0., max_weight) / max_weight\n self.vc = colors_like(adjusted_weights, self.v, colormap=color)", "response": "Color the data as a heatmap."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef quads_to_tris(quads):\n '''\n Convert quad faces to triangular faces.\n\n quads: An nx4 array.\n\n Return a 2nx3 array.\n\n '''\n import numpy as np\n tris = np.empty((2 * len(quads), 3))\n tris[0::2, :] = quads[:, [0, 1, 2]]\n tris[1::2, :] = quads[:, [0, 2, 3]]\n return tris", "response": "Convert quad faces to triangular faces."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning all of the faces that contain at least one of the vertices in v_indices.", "response": "def all_faces_with_verts(self, v_indices, as_boolean=False):\n '''\n returns all of the faces that contain at least one of the vertices in v_indices\n '''\n import numpy as np\n included_vertices = np.zeros(self.v.shape[0], dtype=bool)\n included_vertices[np.array(v_indices, dtype=np.uint32)] = True\n faces_with_verts = included_vertices[self.f].all(axis=1)\n if as_boolean:\n return faces_with_verts\n return np.nonzero(faces_with_verts)[0]"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngives a list of segment names return an array of vertex indices for all the vertices in those faces.", "response": "def vertex_indices_in_segments(self, segments, ret_face_indices=False):\n '''\n Given a list of segment names, return an array of vertex indices for\n all the vertices in those faces.\n\n Args:\n segments: a list of segment names,\n ret_face_indices: if it is `True`, returns face indices\n '''\n import numpy as np\n import warnings\n\n face_indices = np.array([])\n vertex_indices = np.array([])\n if self.segm is not None:\n try:\n segments = [self.segm[name] for name in segments]\n except KeyError as e:\n raise ValueError('Unknown segments {}. Consier using Mesh.clean_segments on segments'.format(e.args[0]))\n face_indices = np.unique(np.concatenate(segments))\n vertex_indices = np.unique(np.ravel(self.f[face_indices]))\n else:\n warnings.warn('self.segm is None, will return empty array')\n\n if ret_face_indices:\n return vertex_indices, face_indices\n else:\n return vertex_indices"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nkeep the faces and vertices for given segments discarding all others.", "response": "def keep_segments(self, segments_to_keep, preserve_segmentation=True):\n '''\n Keep the faces and vertices for given segments, discarding all others.\n When preserve_segmentation is false self.segm is discarded for speed.\n '''\n v_ind, f_ind = self.vertex_indices_in_segments(segments_to_keep, ret_face_indices=True)\n self.segm = {name: self.segm[name] for name in segments_to_keep}\n\n if not preserve_segmentation:\n self.segm = None\n self.f = self.f[f_ind]\n if self.ft is not None:\n self.ft = self.ft[f_ind]\n\n self.keep_vertices(v_ind)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nremoves the faces and vertices for given segments keeping all others.", "response": "def remove_segments(self, segments_to_remove):\n ''' Remove the faces and vertices for given segments, keeping all others.\n\n Args:\n segments_to_remove: a list of segnments whose vertices will be removed\n '''\n v_ind = self.vertex_indices_in_segments(segments_to_remove)\n self.segm = {name: faces for name, faces in self.segm.iteritems() if name not in segments_to_remove}\n self.remove_vertices(v_ind)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef verts_in_common(self, segments):\n verts_by_segm = self.verts_by_segm\n return sorted(reduce(lambda s0, s1: s0.intersection(s1),\n [set(verts_by_segm[segm]) for segm in segments]))", "response": "returns array of all vertex indices common to each segment in segments"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a downsampled copy of this mesh.", "response": "def downsampled_mesh(self, step):\n \"\"\"Returns a downsampled copy of this mesh.\n\n Args:\n step: the step size for the sampling\n\n Returns:\n a new, downsampled Mesh object.\n\n Raises:\n ValueError if this Mesh has faces.\n \"\"\"\n from lace.mesh import Mesh\n\n if self.f is not None:\n raise ValueError(\n 'Function `downsampled_mesh` does not support faces.')\n\n low = Mesh()\n if self.v is not None:\n low.v = self.v[::step]\n if self.vc is not None:\n low.vc = self.vc[::step]\n return low"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nkeeping the given vertices and discard the others and any faces to which they may belong.", "response": "def keep_vertices(self, indices_to_keep, ret_kept_faces=False):\n '''\n Keep the given vertices and discard the others, and any faces to which\n they may belong.\n\n\n If `ret_kept_faces` is `True`, return the original indices of the kept\n faces. Otherwise return `self` for chaining.\n\n '''\n import numpy as np\n\n if self.v is None:\n return\n\n indices_to_keep = np.array(indices_to_keep, dtype=np.uint32)\n\n initial_num_verts = self.v.shape[0]\n if self.f is not None:\n initial_num_faces = self.f.shape[0]\n f_indices_to_keep = self.all_faces_with_verts(indices_to_keep, as_boolean=True)\n\n # Why do we test this? Don't know. But we do need to test it before we\n # mutate self.v.\n vn_should_update = self.vn is not None and self.vn.shape[0] == initial_num_verts\n vc_should_update = self.vc is not None and self.vc.shape[0] == initial_num_verts\n\n self.v = self.v[indices_to_keep]\n\n if vn_should_update:\n self.vn = self.vn[indices_to_keep]\n if vc_should_update:\n self.vc = self.vc[indices_to_keep]\n\n if self.f is not None:\n v_old_to_new = np.zeros(initial_num_verts, dtype=int)\n f_old_to_new = np.zeros(initial_num_faces, dtype=int)\n\n v_old_to_new[indices_to_keep] = np.arange(len(indices_to_keep), dtype=int)\n self.f = v_old_to_new[self.f[f_indices_to_keep]]\n f_old_to_new[f_indices_to_keep] = np.arange(self.f.shape[0], dtype=int)\n\n else:\n # Make the code below work, in case there is somehow degenerate\n # segm even though there are no faces.\n f_indices_to_keep = []\n\n if self.segm is not None:\n new_segm = {}\n for segm_name, segm_faces in self.segm.items():\n faces = np.array(segm_faces, dtype=int)\n valid_faces = faces[f_indices_to_keep[faces]]\n if len(valid_faces):\n new_segm[segm_name] = f_old_to_new[valid_faces]\n self.segm = new_segm if new_segm else None\n\n if hasattr(self, '_raw_landmarks') and self._raw_landmarks is not None:\n self.recompute_landmarks()\n\n return np.nonzero(f_indices_to_keep)[0] if ret_kept_faces else self"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_from_mesh_and_lines(cls, mesh, lines):\n '''\n Return a copy of mesh with line vertices and edges added.\n\n mesh: A Mesh\n lines: A list of Polyline or Lines objects.\n\n '''\n mesh_with_lines = mesh.copy()\n mesh_with_lines.add_lines(lines)\n return mesh_with_lines", "response": "Create a copy of a mesh with line vertices and edges added."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nadding line vertices and edges to the mesh.", "response": "def add_lines(self, lines):\n '''\n Add line vertices and edges to the mesh.\n\n lines: A list of Polyline or Lines objects.\n\n '''\n import numpy as np\n if not lines:\n return\n\n v_lines = np.vstack([l.v for l in lines])\n v_index_offset = np.cumsum([0] + [len(l.v) for l in lines])\n e_lines = np.vstack([l.e + v_index_offset[i] for i, l in enumerate(lines)])\n num_body_verts = self.v.shape[0]\n self.v = np.vstack([self.v, v_lines])\n self.e = e_lines + num_body_verts"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a sparse matrix where each nonzero entry is connected to vertex 12.", "response": "def vert_connectivity(self):\n \"\"\"Returns a sparse matrix (of size #verts x #verts) where each nonzero\n element indicates a neighborhood relation. For example, if there is a\n nonzero element in position (15, 12), that means vertex 15 is connected\n by an edge to vertex 12.\"\"\"\n import numpy as np\n import scipy.sparse as sp\n from blmath.numerics.matlab import row\n vpv = sp.csc_matrix((len(self.v), len(self.v)))\n # for each column in the faces...\n for i in range(3):\n IS = self.f[:, i]\n JS = self.f[:, (i+1)%3]\n data = np.ones(len(IS))\n ij = np.vstack((row(IS.flatten()), row(JS.flatten())))\n mtx = sp.csc_matrix((data, ij), shape=vpv.shape)\n vpv = vpv + mtx + mtx.T\n return vpv"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a dictionary from vertidx - pairs to opposites.", "response": "def vert_opposites_per_edge(self):\n \"\"\"Returns a dictionary from vertidx-pairs to opposites.\n For example, a key consist of [4, 5)] meaning the edge between\n vertices 4 and 5, and a value might be [10, 11] which are the indices\n of the vertices opposing this edge.\"\"\"\n result = {}\n for f in self.f:\n for i in range(3):\n key = [f[i], f[(i+1)%3]]\n key.sort()\n key = tuple(key)\n val = f[(i+2)%3]\n\n if key in result:\n result[key].append(val)\n else:\n result[key] = [val]\n return result"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef faces_per_edge(self):\n import numpy as np\n import scipy.sparse as sp\n from blmath.numerics.matlab import col\n IS = np.repeat(np.arange(len(self.f)), 3)\n JS = self.f.ravel()\n data = np.ones(IS.size)\n f2v = sp.csc_matrix((data, (IS, JS)), shape=(len(self.f), np.max(self.f.ravel())+1))\n f2f = f2v.dot(f2v.T)\n f2f = f2f.tocoo()\n f2f = np.hstack((col(f2f.row), col(f2f.col), col(f2f.data)))\n which = (f2f[:, 0] < f2f[:, 1]) & (f2f[:, 2] >= 2)\n return np.asarray(f2f[which, :2], np.uint32)", "response": "Returns an Ex2 array of adjacencies between faces where each element in the array is a face index and each element in the array is a face index. Edges that are not shared by 2 faces are not included."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef vertices_per_edge(self):\n import numpy as np\n return np.asarray([vertices_in_common(e[0], e[1]) for e in self.f[self.faces_per_edge]])", "response": "Returns an Ex2 array of adjacencies between vertices where\n each element in the array is a vertex index. Each element in the array is a vertex index. Edges that are not shared by 2 faces are not included."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a matrix M which if multiplied by vertices gives back edges.", "response": "def get_vertices_to_edges_matrix(self, want_xyz=True):\n \"\"\"Returns a matrix M, which if multiplied by vertices,\n gives back edges (so \"e = M.dot(v)\"). Note that this generates\n one edge per edge, *not* two edges per triangle.\n\n Args:\n want_xyz: if true, takes and returns xyz coordinates, otherwise\n takes and returns x *or* y *or* z coordinates\n \"\"\"\n import numpy as np\n import scipy.sparse as sp\n\n vpe = np.asarray(self.vertices_per_edge, dtype=np.int32)\n IS = np.repeat(np.arange(len(vpe)), 2)\n JS = vpe.flatten()\n data = np.ones_like(vpe)\n data[:, 1] = -1\n data = data.flatten()\n\n if want_xyz:\n IS = np.concatenate((IS*3, IS*3+1, IS*3+2))\n JS = np.concatenate((JS*3, JS*3+1, JS*3+2))\n data = np.concatenate((data, data, data))\n\n ij = np.vstack((IS.flatten(), JS.flatten()))\n return sp.csc_matrix((data, ij))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngives verts and faces this remove colocated vertices", "response": "def remove_redundant_verts(self, eps=1e-10):\n \"\"\"Given verts and faces, this remove colocated vertices\"\"\"\n import numpy as np\n from scipy.spatial import cKDTree # FIXME pylint: disable=no-name-in-module\n fshape = self.f.shape\n tree = cKDTree(self.v)\n close_pairs = list(tree.query_pairs(eps))\n if close_pairs:\n close_pairs = np.sort(close_pairs, axis=1)\n # update faces to not refer to redundant vertices\n equivalent_verts = np.arange(self.v.shape[0])\n for v1, v2 in close_pairs:\n if equivalent_verts[v2] > v1:\n equivalent_verts[v2] = v1\n self.f = equivalent_verts[self.f.flatten()].reshape((-1, 3))\n # get rid of unused verts, and update faces accordingly\n vertidxs_left = np.unique(self.f)\n repl = np.arange(np.max(self.f)+1)\n repl[vertidxs_left] = np.arange(len(vertidxs_left))\n self.v = self.v[vertidxs_left]\n self.f = repl[self.f].reshape((-1, fshape[1]))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef cardinality_class(self):\n if self.cardinality:\n if self.cardinality > 1000:\n return 'high'\n if self.cardinality > 50:\n return 'medium'\n if self.cardinality > 7:\n return 'low'\n return 'tiny'", "response": "Return the cardinality class of the resource."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndumps a Baiji object to a file.", "response": "def _dump(f, obj, flip_faces=False, ungroup=False, comments=None, split_normals=False, write_mtl=True): # pylint: disable=redefined-outer-name\n '''\n write_mtl: When True and mesh has a texture, includes a mtllib\n reference in the .obj and writes a .mtl alongside.\n\n '''\n import os\n import numpy as np\n from baiji import s3\n\n ff = -1 if flip_faces else 1\n def write_face_to_obj_file(obj, faces, face_index, obj_file):\n vertex_indices = faces[face_index][::ff] + 1\n\n write_normals = obj.fn is not None or (obj.vn is not None and obj.vn.shape == obj.v.shape)\n write_texture = obj.ft is not None and obj.vt is not None\n\n if write_normals and obj.fn is not None:\n normal_indices = obj.fn[face_index][::ff] + 1\n assert len(normal_indices) == len(vertex_indices)\n elif write_normals: # unspecified fn but per-vertex normals, assume ordering is same as for v\n normal_indices = faces[face_index][::ff] + 1\n\n if write_texture:\n texture_indices = obj.ft[face_index][::ff] + 1\n assert len(texture_indices) == len(vertex_indices)\n\n # Valid obj face lines are: v, v/vt, v//vn, v/vt/vn\n if write_normals and write_texture:\n pattern = '%d/%d/%d'\n value = tuple(np.array([vertex_indices, texture_indices, normal_indices]).T.flatten())\n elif write_normals:\n pattern = '%d//%d'\n value = tuple(np.array([vertex_indices, normal_indices]).T.flatten())\n elif write_texture:\n pattern = '%d/%d'\n value = tuple(np.array([vertex_indices, texture_indices]).T.flatten())\n else:\n pattern = '%d'\n value = tuple(vertex_indices)\n obj_file.write(('f ' + ' '.join([pattern]*len(vertex_indices)) + '\\n') % value)\n\n if comments != None:\n if isinstance(comments, basestring):\n comments = [comments]\n for comment in comments:\n for line in comment.split(\"\\n\"):\n f.write(\"# %s\\n\" % line)\n\n if write_mtl and hasattr(obj, 'texture_filepath') and obj.texture_filepath is not None:\n save_to = s3.path.dirname(f.name)\n mtl_name = os.path.splitext(s3.path.basename(f.name))[0]\n mtl_filename = mtl_name + '.mtl'\n f.write('mtllib %s\\n' % mtl_filename)\n f.write('usemtl %s\\n' % mtl_name)\n texture_filename = mtl_name + os.path.splitext(obj.texture_filepath)[1]\n if not s3.exists(s3.path.join(save_to, texture_filename)):\n s3.cp(obj.texture_filepath, s3.path.join(save_to, texture_filename))\n obj.write_mtl(s3.path.join(save_to, mtl_filename), mtl_name, texture_filename)\n\n if obj.vc is not None:\n for r, c in zip(obj.v, obj.vc):\n f.write('v %f %f %f %f %f %f\\n' % (r[0], r[1], r[2], c[0], c[1], c[2]))\n elif obj.v is not None:\n for r in obj.v:\n f.write('v %f %f %f\\n' % (r[0], r[1], r[2]))\n\n if obj.vn is not None:\n if split_normals:\n for vn_idx in obj.fn:\n r = obj.vn[vn_idx[0]]\n f.write('vn %f %f %f\\n' % (r[0], r[1], r[2]))\n r = obj.vn[vn_idx[1]]\n f.write('vn %f %f %f\\n' % (r[0], r[1], r[2]))\n r = obj.vn[vn_idx[2]]\n f.write('vn %f %f %f\\n' % (r[0], r[1], r[2]))\n else:\n for r in obj.vn:\n f.write('vn %f %f %f\\n' % (r[0], r[1], r[2]))\n\n if obj.ft is not None and obj.vt is not None:\n for r in obj.vt:\n if len(r) == 3:\n f.write('vt %f %f %f\\n' % (r[0], r[1], r[2]))\n else:\n f.write('vt %f %f\\n' % (r[0], r[1]))\n if obj.f4 is not None:\n faces = obj.f4\n elif obj.f is not None:\n faces = obj.f\n else:\n faces = None\n if obj.segm is not None and not ungroup:\n if faces is not None:\n # An array of strings.\n group_names = np.array(obj.segm.keys())\n\n # A 2d array of booleans indicating which face is in which group.\n group_mask = np.zeros((len(group_names), len(faces)), dtype=bool)\n for i, segm_faces in enumerate(obj.segm.itervalues()):\n group_mask[i][segm_faces] = True\n\n # In an OBJ file, \"g\" changes the current state. This is a slice of\n # group_mask that represents the current state.\n current_group_mask = np.zeros((len(group_names),), dtype=bool)\n\n for face_index in range(len(faces)):\n # If the group has changed from the previous face, write the\n # group entry.\n this_group_mask = group_mask[:, face_index]\n if any(current_group_mask != this_group_mask):\n current_group_mask = this_group_mask\n f.write('g %s\\n' % ' '.join(group_names[current_group_mask]))\n\n write_face_to_obj_file(obj, faces, face_index, f)\n else:\n if faces is not None:\n for face_index in range(len(faces)):\n write_face_to_obj_file(obj, faces, face_index, f)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the signed area of a triangle in 2D.", "response": "def signed_cell_areas(self):\n \"\"\"Signed area of a triangle in 2D.\n \"\"\"\n # http://mathworld.wolfram.com/TriangleArea.html\n assert (\n self.node_coords.shape[1] == 2\n ), \"Signed areas only make sense for triangles in 2D.\"\n\n if self._signed_cell_areas is None:\n # One could make p contiguous by adding a copy(), but that's not\n # really worth it here.\n p = self.node_coords[self.cells[\"nodes\"]].T\n # \n self._signed_cell_areas = (\n +p[0][2] * (p[1][0] - p[1][1])\n + p[0][0] * (p[1][1] - p[1][2])\n + p[0][1] * (p[1][2] - p[1][0])\n ) / 2\n return self._signed_cell_areas"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates edge - node and edge - cell relations.", "response": "def create_edges(self):\n \"\"\"Set up edge-node and edge-cell relations.\n \"\"\"\n # Reshape into individual edges.\n # Sort the columns to make it possible for `unique()` to identify\n # individual edges.\n s = self.idx_hierarchy.shape\n a = numpy.sort(self.idx_hierarchy.reshape(s[0], -1).T)\n a_unique, inv, cts = unique_rows(a)\n\n assert numpy.all(\n cts < 3\n ), \"No edge has more than 2 cells. Are cells listed twice?\"\n\n self.is_boundary_edge = (cts[inv] == 1).reshape(s[1:])\n\n self.is_boundary_edge_individual = cts == 1\n\n self.edges = {\"nodes\": a_unique}\n\n # cell->edges relationship\n self.cells[\"edges\"] = inv.reshape(3, -1).T\n\n self._edges_cells = None\n self._edge_gid_to_edge_list = None\n\n # Store an index {boundary,interior}_edge -> edge_gid\n self._edge_to_edge_gid = [\n [],\n numpy.where(self.is_boundary_edge_individual)[0],\n numpy.where(~self.is_boundary_edge_individual)[0],\n ]\n return"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncomputing the integral of x over all atomic \"triangles i. e. areas cornered by a node edge midpoint and a circumcenter.", "response": "def _compute_integral_x(self):\n \"\"\"Computes the integral of x,\n\n \\\\int_V x,\n\n over all atomic \"triangles\", i.e., areas cornered by a node, an edge\n midpoint, and a circumcenter.\n \"\"\"\n # The integral of any linear function over a triangle is the average of\n # the values of the function in each of the three corners, times the\n # area of the triangle.\n right_triangle_vols = self.cell_partitions\n\n node_edges = self.idx_hierarchy\n\n corner = self.node_coords[node_edges]\n edge_midpoints = 0.5 * (corner[0] + corner[1])\n cc = self.cell_circumcenters\n\n average = (corner + edge_midpoints[None] + cc[None, None]) / 3.0\n\n contribs = right_triangle_vols[None, :, :, None] * average\n\n return node_edges, contribs"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _compute_surface_areas(self, cell_ids):\n # Each of the three edges may contribute to the surface areas of all\n # three vertices. Here, only the two adjacent nodes receive a\n # contribution, but other approaches (e.g., the flat cell corrector),\n # may contribute to all three nodes.\n cn = self.cells[\"nodes\"][cell_ids]\n ids = numpy.stack([cn, cn, cn], axis=1)\n\n half_el = 0.5 * self.edge_lengths[..., cell_ids]\n zero = numpy.zeros([half_el.shape[1]])\n vals = numpy.stack(\n [\n numpy.column_stack([zero, half_el[0], half_el[0]]),\n numpy.column_stack([half_el[1], zero, half_el[1]]),\n numpy.column_stack([half_el[2], half_el[2], zero]),\n ],\n axis=1,\n )\n\n return ids, vals", "response": "Compute the surface areas of all the nodes in the cell."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef compute_curl(self, vector_field):\n # Compute the projection of A on the edge at each edge midpoint.\n # Take the average of `vector_field` at the endpoints to get the\n # approximate value at the edge midpoint.\n A = 0.5 * numpy.sum(vector_field[self.idx_hierarchy], axis=0)\n # sum of for all three edges\n sum_edge_dot_A = numpy.einsum(\"ijk, ijk->j\", self.half_edge_coords, A)\n\n # Get normalized vector orthogonal to triangle\n z = numpy.cross(self.half_edge_coords[0], self.half_edge_coords[1])\n\n # Now compute\n #\n # curl = z / ||z|| * sum_edge_dot_A / |A|.\n #\n # Since ||z|| = 2*|A|, one can save a sqrt and do\n #\n # curl = z * sum_edge_dot_A * 0.5 / |A|^2.\n #\n curl = z * (0.5 * sum_edge_dot_A / self.cell_volumes ** 2)[..., None]\n return curl", "response": "Compute the curl of a vector field over the mesh."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef plot(\n self,\n show_coedges=True,\n show_centroids=True,\n mesh_color=\"k\",\n nondelaunay_edge_color=\"#d62728\", # mpl 2.0 default red\n boundary_edge_color=None,\n comesh_color=(0.8, 0.8, 0.8),\n show_axes=True,\n ):\n \"\"\"Show the mesh using matplotlib.\n \"\"\"\n # Importing matplotlib takes a while, so don't do that at the header.\n from matplotlib import pyplot as plt\n from matplotlib.collections import LineCollection\n\n # from mpl_toolkits.mplot3d import Axes3D\n fig = plt.figure()\n ax = fig.gca()\n # plt.axis(\"equal\")\n if not show_axes:\n ax.set_axis_off()\n\n xmin = numpy.amin(self.node_coords[:, 0])\n xmax = numpy.amax(self.node_coords[:, 0])\n ymin = numpy.amin(self.node_coords[:, 1])\n ymax = numpy.amax(self.node_coords[:, 1])\n\n width = xmax - xmin\n xmin -= 0.1 * width\n xmax += 0.1 * width\n\n height = ymax - ymin\n ymin -= 0.1 * height\n ymax += 0.1 * height\n\n # ax.set_xlim(xmin, xmax)\n # ax.set_ylim(ymin, ymax)\n\n if self.edges is None:\n self.create_edges()\n\n # Get edges, cut off z-component.\n e = self.node_coords[self.edges[\"nodes\"]][:, :, :2]\n # Plot regular edges, mark those with negative ce-ratio red.\n ce_ratios = self.ce_ratios_per_interior_edge\n pos = ce_ratios >= 0\n\n is_pos = numpy.zeros(len(self.edges[\"nodes\"]), dtype=bool)\n is_pos[self._edge_to_edge_gid[2][pos]] = True\n\n # Mark Delaunay-conforming boundary edges\n is_pos_boundary = self.ce_ratios[self.is_boundary_edge] >= 0\n is_pos[self._edge_to_edge_gid[1][is_pos_boundary]] = True\n\n line_segments0 = LineCollection(e[is_pos], color=mesh_color)\n ax.add_collection(line_segments0)\n #\n line_segments1 = LineCollection(e[~is_pos], color=nondelaunay_edge_color)\n ax.add_collection(line_segments1)\n\n if show_coedges:\n # Connect all cell circumcenters with the edge midpoints\n cc = self.cell_circumcenters\n\n edge_midpoints = 0.5 * (\n self.node_coords[self.edges[\"nodes\"][:, 0]]\n + self.node_coords[self.edges[\"nodes\"][:, 1]]\n )\n\n # Plot connection of the circumcenter to the midpoint of all three\n # axes.\n a = numpy.stack(\n [cc[:, :2], edge_midpoints[self.cells[\"edges\"][:, 0], :2]], axis=1\n )\n b = numpy.stack(\n [cc[:, :2], edge_midpoints[self.cells[\"edges\"][:, 1], :2]], axis=1\n )\n c = numpy.stack(\n [cc[:, :2], edge_midpoints[self.cells[\"edges\"][:, 2], :2]], axis=1\n )\n\n line_segments = LineCollection(\n numpy.concatenate([a, b, c]), color=comesh_color\n )\n ax.add_collection(line_segments)\n\n if boundary_edge_color:\n e = self.node_coords[self.edges[\"nodes\"][self.is_boundary_edge_individual]][\n :, :, :2\n ]\n line_segments1 = LineCollection(e, color=boundary_edge_color)\n ax.add_collection(line_segments1)\n\n if show_centroids:\n centroids = self.control_volume_centroids\n ax.plot(\n centroids[:, 0],\n centroids[:, 1],\n linestyle=\"\",\n marker=\".\",\n color=\"#d62728\",\n )\n\n return fig", "response": "Plot the species in the 3D mesh."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nplot the vicinity of a node and its ce_ratio.", "response": "def show_vertex(self, node_id, show_ce_ratio=True):\n \"\"\"Plot the vicinity of a node and its ce_ratio.\n\n :param node_id: Node ID of the node to be shown.\n :type node_id: int\n\n :param show_ce_ratio: If true, shows the ce_ratio of the node, too.\n :type show_ce_ratio: bool, optional\n \"\"\"\n # Importing matplotlib takes a while, so don't do that at the header.\n from matplotlib import pyplot as plt\n\n fig = plt.figure()\n ax = fig.gca()\n plt.axis(\"equal\")\n\n # Find the edges that contain the vertex\n edge_gids = numpy.where((self.edges[\"nodes\"] == node_id).any(axis=1))[0]\n # ... and plot them\n for node_ids in self.edges[\"nodes\"][edge_gids]:\n x = self.node_coords[node_ids]\n ax.plot(x[:, 0], x[:, 1], \"k\")\n\n # Highlight ce_ratios.\n if show_ce_ratio:\n if self.cell_circumcenters is None:\n X = self.node_coords[self.cells[\"nodes\"]]\n self.cell_circumcenters = self.compute_triangle_circumcenters(\n X, self.ei_dot_ei, self.ei_dot_ej\n )\n\n # Find the cells that contain the vertex\n cell_ids = numpy.where((self.cells[\"nodes\"] == node_id).any(axis=1))[0]\n\n for cell_id in cell_ids:\n for edge_gid in self.cells[\"edges\"][cell_id]:\n if node_id not in self.edges[\"nodes\"][edge_gid]:\n continue\n node_ids = self.edges[\"nodes\"][edge_gid]\n edge_midpoint = 0.5 * (\n self.node_coords[node_ids[0]] + self.node_coords[node_ids[1]]\n )\n p = _column_stack(self.cell_circumcenters[cell_id], edge_midpoint)\n q = numpy.column_stack(\n [\n self.cell_circumcenters[cell_id],\n edge_midpoint,\n self.node_coords[node_id],\n ]\n )\n ax.fill(q[0], q[1], color=\"0.5\")\n ax.plot(p[0], p[1], color=\"0.7\")\n return"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _update_cell_values(self, cell_ids, interior_edge_ids):\n # update idx_hierarchy\n nds = self.cells[\"nodes\"][cell_ids].T\n self.idx_hierarchy[..., cell_ids] = nds[self.local_idx]\n\n # update self.half_edge_coords\n self.half_edge_coords[:, cell_ids, :] = numpy.moveaxis(\n self.node_coords[self.idx_hierarchy[1, ..., cell_ids]]\n - self.node_coords[self.idx_hierarchy[0, ..., cell_ids]],\n 0,\n 1,\n )\n\n # update self.ei_dot_ej\n self.ei_dot_ej[:, cell_ids] = numpy.einsum(\n \"ijk, ijk->ij\",\n self.half_edge_coords[[1, 2, 0]][:, cell_ids],\n self.half_edge_coords[[2, 0, 1]][:, cell_ids],\n )\n\n # update self.ei_dot_ei\n e = self.half_edge_coords[:, cell_ids]\n self.ei_dot_ei[:, cell_ids] = numpy.einsum(\"ijk, ijk->ij\", e, e)\n\n # update cell_volumes, ce_ratios_per_half_edge\n cv = compute_tri_areas(self.ei_dot_ej[:, cell_ids])\n ce = compute_ce_ratios(self.ei_dot_ej[:, cell_ids], cv)\n self.cell_volumes[cell_ids] = cv\n self.ce_ratios[:, cell_ids] = ce\n\n if self._interior_ce_ratios is not None:\n self._interior_ce_ratios[interior_edge_ids] = 0.0\n edge_gids = self._edge_to_edge_gid[2][interior_edge_ids]\n adj_cells = self._edges_cells[2][interior_edge_ids]\n\n is0 = self.cells[\"edges\"][adj_cells[:, 0]][:, 0] == edge_gids\n is1 = self.cells[\"edges\"][adj_cells[:, 0]][:, 1] == edge_gids\n is2 = self.cells[\"edges\"][adj_cells[:, 0]][:, 2] == edge_gids\n assert numpy.all(\n numpy.sum(numpy.column_stack([is0, is1, is2]), axis=1) == 1\n )\n #\n self._interior_ce_ratios[interior_edge_ids[is0]] += self.ce_ratios[\n 0, adj_cells[is0, 0]\n ]\n self._interior_ce_ratios[interior_edge_ids[is1]] += self.ce_ratios[\n 1, adj_cells[is1, 0]\n ]\n self._interior_ce_ratios[interior_edge_ids[is2]] += self.ce_ratios[\n 2, adj_cells[is2, 0]\n ]\n\n is0 = self.cells[\"edges\"][adj_cells[:, 1]][:, 0] == edge_gids\n is1 = self.cells[\"edges\"][adj_cells[:, 1]][:, 1] == edge_gids\n is2 = self.cells[\"edges\"][adj_cells[:, 1]][:, 2] == edge_gids\n assert numpy.all(\n numpy.sum(numpy.column_stack([is0, is1, is2]), axis=1) == 1\n )\n #\n self._interior_ce_ratios[interior_edge_ids[is0]] += self.ce_ratios[\n 0, adj_cells[is0, 1]\n ]\n self._interior_ce_ratios[interior_edge_ids[is1]] += self.ce_ratios[\n 1, adj_cells[is1, 1]\n ]\n self._interior_ce_ratios[interior_edge_ids[is2]] += self.ce_ratios[\n 2, adj_cells[is2, 1]\n ]\n\n if self._signed_cell_areas is not None:\n # One could make p contiguous by adding a copy(), but that's not\n # really worth it here.\n p = self.node_coords[self.cells[\"nodes\"][cell_ids]].T\n # \n self._signed_cell_areas[cell_ids] = (\n +p[0][2] * (p[1][0] - p[1][1])\n + p[0][0] * (p[1][1] - p[1][2])\n + p[0][1] * (p[1][2] - p[1][0])\n ) / 2\n\n # TODO update those values\n self._cell_centroids = None\n self._edge_lengths = None\n self._cell_circumcenters = None\n self._control_volumes = None\n self._cell_partitions = None\n self._cv_centroids = None\n self._surface_areas = None\n self.subdomains = {}\n return", "response": "Updates all sorts of cell information for the given cell IDs."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nwrite a mesh to collada file format.", "response": "def _dump(f, mesh):\n '''\n Writes a mesh to collada file format.\n '''\n dae = mesh_to_collada(mesh)\n dae.write(f.name)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef dumps(mesh):\n '''\n Generates a UTF-8 XML string containing the mesh, in collada format.\n '''\n from lxml import etree\n\n dae = mesh_to_collada(mesh)\n\n # Update the xmlnode.\n dae.save()\n\n return etree.tostring(dae.xmlnode, encoding='UTF-8')", "response": "Generates a UTF - 8 XML string containing the mesh in collada format."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting a mesh to a collada object.", "response": "def mesh_to_collada(mesh):\n '''\n Supports per-vertex color, but nothing else.\n '''\n import numpy as np\n try:\n from collada import Collada, scene\n except ImportError:\n raise ImportError(\"lace.serialization.dae.mesh_to_collade requires package pycollada.\")\n\n\n def create_material(dae):\n from collada import material, scene\n effect = material.Effect(\"effect0\", [], \"phong\", diffuse=(1, 1, 1), specular=(0, 0, 0), double_sided=True)\n mat = material.Material(\"material0\", \"mymaterial\", effect)\n dae.effects.append(effect)\n dae.materials.append(mat)\n return scene.MaterialNode(\"materialref\", mat, inputs=[])\n\n def geometry_from_mesh(dae, mesh):\n from collada import source, geometry\n srcs = []\n # v\n srcs.append(source.FloatSource(\"verts-array\", mesh.v, ('X', 'Y', 'Z')))\n input_list = source.InputList()\n input_list.addInput(0, 'VERTEX', \"#verts-array\")\n # vc\n if mesh.vc is not None:\n input_list.addInput(len(srcs), 'COLOR', \"#color-array\")\n srcs.append(source.FloatSource(\"color-array\", mesh.vc[mesh.f.ravel()], ('X', 'Y', 'Z')))\n # f\n geom = geometry.Geometry(str(mesh), \"geometry0\", \"mymesh\", srcs)\n indices = np.dstack([mesh.f for _ in srcs]).ravel()\n triset = geom.createTriangleSet(indices, input_list, \"materialref\")\n geom.primitives.append(triset)\n # e\n if mesh.e is not None:\n indices = np.dstack([mesh.e for _ in srcs]).ravel()\n lineset = geom.createLineSet(indices, input_list, \"materialref\")\n geom.primitives.append(lineset)\n dae.geometries.append(geom)\n return geom\n\n dae = Collada()\n geom = geometry_from_mesh(dae, mesh)\n node = scene.Node(\"node0\", children=[scene.GeometryNode(geom, [create_material(dae)])])\n myscene = scene.Scene(\"myscene\", [node])\n dae.scenes.append(myscene)\n dae.scene = myscene\n return dae"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef convert_units(self, from_units, to_units):\n '''\n Convert the mesh from one set of units to another.\n\n These calls are equivalent:\n\n - mesh.convert_units(from_units='cm', to_units='m')\n - mesh.scale(.01)\n\n '''\n from blmath import units\n factor = units.factor(\n from_units=from_units,\n to_units=to_units,\n units_class='length'\n )\n self.scale(factor)", "response": "Convert the mesh from one set of units to another."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\npredicts the unit system of the united states in the body.", "response": "def predict_body_units(self):\n '''\n There is no prediction for united states unit system.\n This may fail when a mesh is not axis-aligned\n '''\n longest_dist = np.max(np.max(self.v, axis=0) - np.min(self.v, axis=0))\n if round(longest_dist / 1000) > 0:\n return 'mm'\n if round(longest_dist / 100) > 0:\n return 'cm'\n if round(longest_dist / 10) > 0:\n return 'dm'\n return 'm'"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef reorient(self, up, look):\n '''\n Reorient the mesh by specifying two vectors.\n\n up: The foot-to-head direction.\n look: The direction the body is facing.\n\n In the result, the up will end up along +y, and look along +z\n (i.e. facing towards a default OpenGL camera).\n\n '''\n from blmath.geometry.transform import rotation_from_up_and_look\n from blmath.numerics import as_numeric_array\n\n up = as_numeric_array(up, (3,))\n look = as_numeric_array(look, (3,))\n\n if self.v is not None:\n self.v = np.dot(rotation_from_up_and_look(up, look), self.v.T).T", "response": "Reorient the mesh by specifying two vectors."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef flip(self, axis=0, preserve_centroid=False):\n '''\n Flip the mesh across the given axis: 0 for x, 1 for y, 2 for z.\n\n When `preserve_centroid` is True, translate after flipping to\n preserve the location of the centroid.\n\n '''\n self.v[:, axis] *= -1\n\n if preserve_centroid:\n self.v[:, axis] -= 2 * self.centroid[0]\n\n self.flip_faces()", "response": "Flip the mesh across the given axis."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef centroid(self):\n '''\n Return the geometric center.\n\n '''\n if self.v is None:\n raise ValueError('Mesh has no vertices; centroid is not defined')\n\n return np.mean(self.v, axis=0)", "response": "Return the geometric center of the mesh."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the point on the floor that lies below the centroid.", "response": "def floor_point(self):\n '''\n Return the point on the floor that lies below the centroid.\n\n '''\n floor_point = self.centroid\n # y to floor\n floor_point[1] = self.v[:, 1].min()\n return floor_point"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef apex(self, axis):\n '''\n Find the most extreme vertex in the direction of the axis provided.\n\n axis: A vector, which is an 3x1 np.array.\n\n '''\n from blmath.geometry.apex import apex\n return apex(self.v, axis)", "response": "Return the most extreme vertex in the direction of the axis provided."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the first blip of a set of nodes in the mesh.", "response": "def first_blip(self, squash_axis, origin, initial_direction):\n '''\n Flatten the mesh to the plane, dropping the dimension identified by\n `squash_axis`: 0 for x, 1 for y, 2 for z. Cast a ray from `origin`,\n pointing along `initial_direction`. Sweep the ray, like a radar, until\n encountering the mesh, and return that vertex: like the first blip of\n the radar. The radar works as if on a point cloud: it sees sees only\n vertices, not edges.\n\n The ray sweeps clockwise and counterclockwise at the same time, and\n returns the first point it hits.\n\n If no intersection occurs within 90 degrees, return None.\n\n `initial_direction` need not be normalized.\n\n '''\n from blmath.numerics import as_numeric_array\n\n origin = vx.reject_axis(as_numeric_array(origin, (3,)), axis=squash_axis, squash=True)\n initial_direction = vx.reject_axis(as_numeric_array(initial_direction, (3,)), axis=squash_axis, squash=True)\n vertices = vx.reject_axis(self.v, axis=squash_axis, squash=True)\n\n origin_to_mesh = vx.normalize(vertices - origin)\n cosines = vx.normalize(initial_direction).dot(origin_to_mesh.T).T\n index_of_first_blip = np.argmax(cosines)\n\n return self.v[index_of_first_blip]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef cut_across_axis(self, dim, minval=None, maxval=None):\n '''\n Cut the mesh by a plane, discarding vertices that lie behind that\n plane. Or cut the mesh by two parallel planes, discarding vertices\n that lie outside them.\n\n The region to keep is defined by an axis of perpendicularity,\n specified by `dim`: 0 means x, 1 means y, 2 means z. `minval`\n and `maxval` indicate the portion of that axis to keep.\n\n Return the original indices of the kept vertices.\n\n '''\n # vertex_mask keeps track of the vertices we want to keep.\n vertex_mask = np.ones((len(self.v),), dtype=bool)\n\n if minval is not None:\n predicate = self.v[:, dim] >= minval\n vertex_mask = np.logical_and(vertex_mask, predicate)\n\n if maxval is not None:\n predicate = self.v[:, dim] <= maxval\n vertex_mask = np.logical_and(vertex_mask, predicate)\n\n vertex_indices = np.flatnonzero(vertex_mask)\n self.keep_vertices(vertex_indices)\n\n return vertex_indices", "response": "Cut the mesh by a plane discarding vertices that lie behind that plane or discarding vertices that lie outside that plane."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nliking cut_across_axis except the subset of vertices isisolate by percentile of the data along a given axis.", "response": "def cut_across_axis_by_percentile(self, dim, minpct=0, maxpct=100):\n '''\n Like cut_across_axis, except the subset of vertices is\n constrained by percentile of the data along a given axis\n instead of specific values. (See numpy.percentile.)\n\n For example, if the mesh has 50,000 vertices, `dim` is 2, and\n `minpct` is 10, this method drops the 5,000 vertices which are\n furthest along the +z axis.\n\n See numpy.percentile\n\n Return the original indices of the kept vertices.\n\n '''\n value_range = np.percentile(self.v[:, dim], (minpct, maxpct))\n return self.cut_across_axis(dim, *value_range)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef cut_by_plane(self, plane, inverted=False):\n '''\n Like cut_across_axis, but works with an arbitrary plane. Keeps\n vertices that lie in front of the plane (i.e. in the direction\n of the plane normal).\n\n inverted: When `True`, invert the logic, to keep the vertices\n that lie behind the plane instead.\n\n Return the original indices of the kept vertices.\n\n '''\n vertices_to_keep = plane.points_in_front(self.v, inverted=inverted, ret_indices=True)\n\n self.keep_vertices(vertices_to_keep)\n\n return vertices_to_keep", "response": "Like cut_across_axis but works with an arbitrary plane."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\npurge items from the traceability_all_items environment variable.", "response": "def purge_items(app, env, docname):\n \"\"\"\n Clean, if existing, ``item`` entries in ``traceability_all_items``\n environment variable, for all the source docs being purged.\n\n This function should be triggered upon ``env-purge-doc`` event.\n\n \"\"\"\n keys = list(env.traceability_all_items.keys())\n for key in keys:\n if env.traceability_all_items[key]['docname'] == docname:\n del env.traceability_all_items[key]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef process_item_nodes(app, doctree, fromdocname):\n env = app.builder.env\n\n all_items = sorted(env.traceability_all_items, key=naturalsortkey)\n\n # Item matrix:\n # Create table with related items, printing their target references.\n # Only source and target items matching respective regexp shall be included\n for node in doctree.traverse(item_matrix):\n table = nodes.table()\n tgroup = nodes.tgroup()\n left_colspec = nodes.colspec(colwidth=5)\n right_colspec = nodes.colspec(colwidth=5)\n tgroup += [left_colspec, right_colspec]\n tgroup += nodes.thead('', nodes.row(\n '',\n nodes.entry('', nodes.paragraph('', 'Source')),\n nodes.entry('', nodes.paragraph('', 'Target'))))\n tbody = nodes.tbody()\n tgroup += tbody\n table += tgroup\n\n for source_item in all_items:\n if re.match(node['source'], source_item):\n row = nodes.row()\n left = nodes.entry()\n left += make_item_ref(app, env, fromdocname,\n env.traceability_all_items[source_item])\n right = nodes.entry()\n for target_item in all_items:\n if (re.match(node['target'], target_item) and\n are_related(\n env, source_item, target_item, node['type'])):\n right += make_item_ref(\n app, env, fromdocname,\n env.traceability_all_items[target_item])\n row += left\n row += right\n tbody += row\n\n node.replace_self(table)\n\n # Item list:\n # Create list with target references. Only items matching list regexp\n # shall be included\n for node in doctree.traverse(item_list):\n content = nodes.bullet_list()\n for item in all_items:\n if re.match(node['filter'], item):\n bullet_list_item = nodes.list_item()\n paragraph = nodes.paragraph()\n paragraph.append(\n make_item_ref(app, env, fromdocname,\n env.traceability_all_items[item]))\n bullet_list_item.append(paragraph)\n content.append(bullet_list_item)\n\n node.replace_self(content)\n\n # Resolve item cross references (from ``item`` role)\n for node in doctree.traverse(pending_item_xref):\n # Create a dummy reference to be used if target reference fails\n new_node = make_refnode(app.builder,\n fromdocname,\n fromdocname,\n 'ITEM_NOT_FOUND',\n node[0].deepcopy(),\n node['reftarget'] + '??')\n # If target exists, try to create the reference\n if node['reftarget'] in env.traceability_all_items:\n item_info = env.traceability_all_items[node['reftarget']]\n try:\n new_node = make_refnode(app.builder,\n fromdocname,\n item_info['docname'],\n item_info['target']['refid'],\n node[0].deepcopy(),\n node['reftarget'])\n except NoUri:\n # ignore if no URI can be determined, e.g. for LaTeX output :(\n pass\n\n else:\n env.warn_node(\n 'Traceability: item %s not found' % node['reftarget'], node)\n\n node.replace_self(new_node)", "response": "Process all item nodes in the item_list nodes."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update_available_item_relationships(app):\n env = app.builder.env\n env.relationships = {}\n\n for rel in list(app.config.traceability_relationships.keys()):\n env.relationships[rel] = app.config.traceability_relationships[rel]\n env.relationships[app.config.traceability_relationships[rel]] = rel\n\n for rel in sorted(list(env.relationships.keys())):\n ItemDirective.option_spec[rel] = directives.unchanged", "response": "Update the option_spec with custom relationships defined in the application configuration file."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef initialize_environment(app):\n env = app.builder.env\n\n # Assure ``traceability_all_items`` will always be there.\n if not hasattr(env, 'traceability_all_items'):\n env.traceability_all_items = {}\n\n update_available_item_relationships(app)", "response": "Initialize the environment variables needed before the build process starts."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a reference node for an item.", "response": "def make_item_ref(app, env, fromdocname, item_info):\n \"\"\"\n Creates a reference node for an item, embedded in a\n paragraph. Reference text adds also a caption if it exists.\n\n \"\"\"\n id = item_info['target']['refid']\n\n if item_info['caption'] != '':\n caption = ', ' + item_info['caption']\n else:\n caption = ''\n\n para = nodes.paragraph()\n newnode = nodes.reference('', '')\n innernode = nodes.emphasis(id + caption, id + caption)\n newnode['refdocname'] = item_info['docname']\n try:\n newnode['refuri'] = app.builder.get_relative_uri(fromdocname,\n item_info['docname'])\n newnode['refuri'] += '#' + id\n except NoUri:\n # ignore if no URI can be determined, e.g. for LaTeX output :(\n pass\n newnode.append(innernode)\n para += newnode\n\n return para"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef are_related(env, source, target, relationships):\n if not relationships:\n relationships = list(env.relationships.keys())\n\n for rel in relationships:\n if (target in env.traceability_all_items[source][rel] or\n source in\n env.traceability_all_items[target][env.relationships[rel]]):\n return True\n\n return False", "response": "Returns True if source and target items are related to the given list of relationships."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nallow visual inspection of geometric primitives.", "response": "def MeshViewer(\n titlebar='Mesh Viewer', static_meshes=None, static_lines=None, uid=None,\n autorecenter=True, keepalive=False, window_width=1280, window_height=960,\n snapshot_camera=None\n):\n \"\"\"Allows visual inspection of geometric primitives.\n\n Write-only Attributes:\n titlebar: string printed in the window titlebar\n dynamic_meshes: list of Mesh objects to be displayed\n static_meshes: list of Mesh objects to be displayed\n dynamic_lines: list of Lines objects to be displayed\n static_lines: list of Lines objects to be displayed\n\n Note: static_meshes is meant for Meshes that are\n updated infrequently, and dynamic_meshes is for Meshes\n that are updated frequently (same for dynamic_lines vs\n static_lines). They may be treated differently for\n performance reasons.\n \"\"\"\n if not test_for_opengl():\n return Dummy()\n mv = MeshViewerLocal(\n shape=(1, 1), uid=uid, titlebar=titlebar, keepalive=keepalive,\n window_width=window_width, window_height=window_height\n )\n result = mv.get_subwindows()[0][0]\n result.snapshot_camera = snapshot_camera\n if static_meshes:\n result.static_meshes = static_meshes\n if static_lines:\n result.static_lines = static_lines\n result.autorecenter = autorecenter\n return result"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nallow subplot - style inspection of primitives in multiple subwindows.", "response": "def MeshViewers(\n shape=(1, 1), titlebar=\"Mesh Viewers\", keepalive=False,\n window_width=1280, window_height=960\n):\n \"\"\"Allows subplot-style inspection of primitives in multiple subwindows.\n\n Args:\n shape: a tuple indicating the number of vertical and horizontal windows requested\n\n Returns: a list of lists of MeshViewer objects: one per window requested.\n \"\"\"\n if not test_for_opengl():\n return Dummy()\n mv = MeshViewerLocal(\n shape=shape, titlebar=titlebar, uid=None, keepalive=keepalive,\n window_width=window_width, window_height=window_height\n )\n return mv.get_subwindows()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nhandle mouse drag events for the current object.", "response": "def on_drag(self, cursor_x, cursor_y):\n \"\"\" Mouse cursor is moving\n Glut calls this function (when mouse button is down)\n and pases the mouse cursor postion in window coords as the mouse moves.\n \"\"\"\n from blmath.geometry.transform.rodrigues import as_rotation_matrix\n if self.isdragging:\n mouse_pt = arcball.Point2fT(cursor_x, cursor_y)\n # Update End Vector And Get Rotation As Quaternion\n ThisQuat = self.arcball.drag(mouse_pt)\n # Convert Quaternion Into Matrix3fT\n self.thisrot = arcball.Matrix3fSetRotationFromQuat4f(ThisQuat)\n # Use correct Linear Algebra matrix multiplication C = A * B\n # Accumulate Last Rotation Into This One\n self.thisrot = arcball.Matrix3fMulMatrix3f(self.lastrot, self.thisrot)\n # make sure it is a rotation\n self.thisrot = as_rotation_matrix(self.thisrot)\n # Set Our Final Transform's Rotation From This One\n self.transform = arcball.Matrix4fSetRotationFromMatrix3f(self.transform, self.thisrot)\n glut.glutPostRedisplay()\n return"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef on_click(self, button, button_state, cursor_x, cursor_y):\n self.isdragging = False\n if button == glut.GLUT_LEFT_BUTTON and button_state == glut.GLUT_UP:\n # Left button released\n self.lastrot = copy.copy(self.thisrot) # Set Last Static Rotation To Last Dynamic One\n elif button == glut.GLUT_LEFT_BUTTON and button_state == glut.GLUT_DOWN:\n # Left button clicked down\n self.lastrot = copy.copy(self.thisrot) # Set Last Static Rotation To Last Dynamic One\n self.isdragging = True # Prepare For Dragging\n mouse_pt = arcball.Point2fT(cursor_x, cursor_y)\n self.arcball.click(mouse_pt) # Update Start Vector And Prepare For Dragging\n elif button == glut.GLUT_RIGHT_BUTTON and button_state == glut.GLUT_DOWN:\n # If a mouse click location was requested, return it to caller\n if hasattr(self, 'event_port'):\n self.mouseclick_port = self.event_port\n del self.event_port\n if hasattr(self, 'mouseclick_port'):\n self.send_mouseclick_to_caller(cursor_x, cursor_y)\n elif button == glut.GLUT_MIDDLE_BUTTON and button_state == glut.GLUT_DOWN:\n # If a mouse click location was requested, return it to caller\n if hasattr(self, 'event_port'):\n self.mouseclick_port = self.event_port\n del self.event_port\n if hasattr(self, 'mouseclick_port'):\n self.send_mouseclick_to_caller(cursor_x, cursor_y, button='middle')\n glut.glutPostRedisplay()", "response": "Called by the mouse button click event handler."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncompute the center of the circumsphere of each cell.", "response": "def _compute_cell_circumcenters(self):\n \"\"\"Computes the center of the circumsphere of each cell.\n \"\"\"\n # Just like for triangular cells, tetrahedron circumcenters are most easily\n # computed with the quadrilateral coordinates available.\n # Luckily, we have the circumcenter-face distances (cfd):\n #\n # CC = (\n # + cfd[0] * face_area[0] / sum(cfd*face_area) * X[0]\n # + cfd[1] * face_area[1] / sum(cfd*face_area) * X[1]\n # + cfd[2] * face_area[2] / sum(cfd*face_area) * X[2]\n # + cfd[3] * face_area[3] / sum(cfd*face_area) * X[3]\n # )\n #\n # (Compare with\n # .)\n # Because of\n #\n # cfd = zeta / (24.0 * face_areas) / self.cell_volumes[None]\n #\n # we have\n #\n # CC = sum_k (zeta[k] / sum(zeta) * X[k]).\n #\n alpha = self._zeta / numpy.sum(self._zeta, axis=0)\n\n self._circumcenters = numpy.sum(\n alpha[None].T * self.node_coords[self.cells[\"nodes\"]], axis=1\n )\n return"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncomputing the control volumes of all nodes in the mesh.", "response": "def control_volumes(self):\n \"\"\"Compute the control volumes of all nodes in the mesh.\n \"\"\"\n if self._control_volumes is None:\n # 1/3. * (0.5 * edge_length) * covolume\n # = 1/6 * edge_length**2 * ce_ratio_edge_ratio\n v = self.ei_dot_ei * self.ce_ratios / 6.0\n # Explicitly sum up contributions per cell first. Makes\n # numpy.add.at faster.\n # For every node k (range(4)), check for which edges k appears in\n # local_idx, and sum() up the v's from there.\n idx = self.local_idx\n vals = numpy.array(\n [\n sum([v[i, j] for i, j in zip(*numpy.where(idx == k)[1:])])\n for k in range(4)\n ]\n ).T\n #\n self._control_volumes = numpy.zeros(len(self.node_coords))\n numpy.add.at(self._control_volumes, self.cells[\"nodes\"], vals)\n return self._control_volumes"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndisplays the edge with ce_ratio.", "response": "def show_edge(self, edge_id):\n \"\"\"Displays edge with ce_ratio.\n\n :param edge_id: Edge ID for which to show the ce_ratio.\n :type edge_id: int\n \"\"\"\n # pylint: disable=unused-variable,relative-import\n from mpl_toolkits.mplot3d import Axes3D\n from matplotlib import pyplot as plt\n\n if \"faces\" not in self.cells:\n self.create_cell_face_relationships()\n if \"edges\" not in self.faces:\n self.create_face_edge_relationships()\n\n fig = plt.figure()\n ax = fig.gca(projection=Axes3D.name)\n plt.axis(\"equal\")\n\n # find all faces with this edge\n adj_face_ids = numpy.where((self.faces[\"edges\"] == edge_id).any(axis=1))[0]\n # find all cells with the faces\n # https://stackoverflow.com/a/38481969/353337\n adj_cell_ids = numpy.where(\n numpy.in1d(self.cells[\"faces\"], adj_face_ids)\n .reshape(self.cells[\"faces\"].shape)\n .any(axis=1)\n )[0]\n\n # plot all those adjacent cells; first collect all edges\n adj_edge_ids = numpy.unique(\n [\n adj_edge_id\n for adj_cell_id in adj_cell_ids\n for face_id in self.cells[\"faces\"][adj_cell_id]\n for adj_edge_id in self.faces[\"edges\"][face_id]\n ]\n )\n col = \"k\"\n for adj_edge_id in adj_edge_ids:\n x = self.node_coords[self.edges[\"nodes\"][adj_edge_id]]\n ax.plot(x[:, 0], x[:, 1], x[:, 2], col)\n\n # make clear which is edge_id\n x = self.node_coords[self.edges[\"nodes\"][edge_id]]\n ax.plot(x[:, 0], x[:, 1], x[:, 2], color=col, linewidth=3.0)\n\n # connect the face circumcenters with the corresponding cell\n # circumcenters\n X = self.node_coords\n for cell_id in adj_cell_ids:\n cc = self._circumcenters[cell_id]\n #\n x = X[self.node_face_cells[..., [cell_id]]]\n face_ccs = compute_triangle_circumcenters(x, self.ei_dot_ei, self.ei_dot_ej)\n # draw the face circumcenters\n ax.plot(\n face_ccs[..., 0].flatten(),\n face_ccs[..., 1].flatten(),\n face_ccs[..., 2].flatten(),\n \"go\",\n )\n # draw the connections\n # tet circumcenter---face circumcenter\n for face_cc in face_ccs:\n ax.plot(\n [cc[..., 0], face_cc[..., 0]],\n [cc[..., 1], face_cc[..., 1]],\n [cc[..., 2], face_cc[..., 2]],\n \"b-\",\n )\n\n # draw the cell circumcenters\n cc = self._circumcenters[adj_cell_ids]\n ax.plot(cc[:, 0], cc[:, 1], cc[:, 2], \"ro\")\n return"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef parse_int(text, fallback=None):\n try:\n if isinstance(text, six.integer_types):\n return text\n elif isinstance(text, six.string_types):\n return int(text)\n else:\n return fallback\n except ValueError:\n return fallback", "response": "Try to extract an integer from a string return the fallback if that s not possible."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreflect a given table from the database.", "response": "def _load_table(self, name):\n \"\"\" Reflect a given table from the database. \"\"\"\n table = self._tables.get(name, None)\n if table is not None:\n return table\n if not self.engine.has_table(name):\n raise BindingException('Table does not exist: %r' % name,\n table=name)\n table = Table(name, self.meta, autoload=True)\n self._tables[name] = table\n return table"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the primary key of the fact table for use in fact table counting.", "response": "def fact_pk(self):\n \"\"\" Try to determine the primary key of the fact table for use in\n fact table counting. \n If more than one column exists, return the first column of the pk.\n \"\"\"\n keys = [c for c in self.fact_table.columns if c.primary_key]\n return keys[0]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef members(self, ref, cuts=None, order=None, page=None, page_size=None):\n def prep(cuts, ref, order, columns=None):\n q = select(columns=columns)\n bindings = []\n cuts, q, bindings = Cuts(self).apply(q, bindings, cuts)\n fields, q, bindings = \\\n Fields(self).apply(q, bindings, ref, distinct=True)\n ordering, q, bindings = \\\n Ordering(self).apply(q, bindings, order, distinct=fields[0])\n q = self.restrict_joins(q, bindings)\n return q, bindings, cuts, fields, ordering\n\n # Count\n count = count_results(self, prep(cuts, ref, order, [1])[0])\n\n # Member list\n q, bindings, cuts, fields, ordering = prep(cuts, ref, order)\n page, q = Pagination(self).apply(q, page, page_size)\n q = self.restrict_joins(q, bindings)\n return {\n 'total_member_count': count,\n 'data': list(generate_results(self, q)),\n 'cell': cuts,\n 'fields': fields,\n 'order': ordering,\n 'page': page['page'],\n 'page_size': page['page_size']\n }", "response": "List all the distinct members of the given reference filtered and paginated."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef facts(self, fields=None, cuts=None, order=None, page=None,\n page_size=None, page_max=None):\n \"\"\" List all facts in the cube, returning only the specified references\n if these are specified. \"\"\"\n\n def prep(cuts, columns=None):\n q = select(columns=columns).select_from(self.fact_table)\n bindings = []\n _, q, bindings = Cuts(self).apply(q, bindings, cuts)\n q = self.restrict_joins(q, bindings)\n return q, bindings\n\n # Count\n count = count_results(self, prep(cuts, [1])[0])\n\n # Facts\n q, bindings = prep(cuts)\n fields, q, bindings = Fields(self).apply(q, bindings, fields)\n ordering, q, bindings = Ordering(self).apply(q, bindings, order)\n page, q = Pagination(self).apply(q, page, page_size, page_max)\n q = self.restrict_joins(q, bindings)\n return {\n 'total_fact_count': count,\n 'data': list(generate_results(self, q)),\n 'cell': cuts,\n 'fields': fields,\n 'order': ordering,\n 'page': page['page'],\n 'page_size': page['page_size']\n }", "response": "List all facts in the cube returning only the specified references\n if these are specified."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef compute_cardinalities(self):\n for dimension in self.model.dimensions:\n result = self.members(dimension.ref, page_size=0)\n dimension.spec['cardinality'] = result.get('total_member_count')", "response": "This method will count the number of distinct values for each in\n and add that count to the model."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrestrict joins across all tables referenced in the database to those specified in the model.", "response": "def restrict_joins(self, q, bindings):\n \"\"\"\n Restrict the joins across all tables referenced in the database\n query to those specified in the model for the relevant dimensions.\n If a single table is used for the query, no unnecessary joins are\n performed. If more than one table are referenced, this ensures\n their returned rows are connected via the fact table.\n \"\"\"\n if len(q.froms) == 1:\n return q\n else:\n for binding in bindings:\n if binding.table == self.fact_table:\n continue\n concept = self.model[binding.ref]\n if isinstance(concept, Dimension):\n dimension = concept\n else:\n dimension = concept.dimension\n dimension_table, key_col = dimension.key_attribute.bind(self)\n if binding.table != dimension_table:\n raise BindingException('Attributes must be of same table as '\n 'as their dimension key')\n\n join_column_name = dimension.join_column_name\n if isinstance(join_column_name, string_types):\n try:\n join_column = self.fact_table.columns[join_column_name]\n except KeyError:\n raise BindingException(\"Join column '%s' for %r not in fact table.\"\n % (dimension.join_column_name, dimension))\n else:\n if not isinstance(join_column_name, list) or len(join_column_name) != 2:\n raise BindingException(\"Join column '%s' for %r should be either a string or a 2-tuple.\"\n % (join_column_name, dimension))\n try:\n join_column = self.fact_table.columns[join_column_name[0]]\n except KeyError:\n raise BindingException(\"Join column '%s' for %r not in fact table.\"\n % (dimension.join_column_name[0], dimension))\n try:\n key_col = dimension_table.columns[join_column_name[1]]\n except KeyError:\n raise BindingException(\"Join column '%s' for %r not in dimension table.\"\n % (dimension.join_column_name[1], dimension))\n\n q = q.where(join_column == key_col)\n return q"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\norders sub - commands for display.", "response": "def order_manually(sub_commands):\n \"\"\"Order sub-commands for display\"\"\"\n order = [\n \"start\",\n \"projects\",\n ]\n ordered = []\n commands = dict(zip([cmd for cmd in sub_commands], sub_commands))\n for k in order:\n ordered.append(commands.get(k, \"\"))\n if k in commands:\n del commands[k]\n\n # Add commands not present in `order` above\n for k in commands:\n ordered.append(commands[k])\n\n return ordered"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef format_help(help):\n help = help.replace('Options:', str(crayons.black('Options:', bold=True)))\n\n help = help.replace('Usage: pinax', str('Usage: {0}'.format(crayons.black('pinax', bold=True))))\n\n help = help.replace(' start', str(crayons.green(' start', bold=True)))\n help = help.replace(' apps', str(crayons.yellow(' apps', bold=True)))\n help = help.replace(' demos', str(crayons.yellow(' demos', bold=True)))\n help = help.replace(' projects', str(crayons.yellow(' projects', bold=True)))\n help = help.replace(' themes', str(crayons.yellow(' themes', bold=True)))\n help = help.replace(' tools', str(crayons.yellow(' tools', bold=True)))\n\n additional_help = \\\n \"\"\"Usage Examples:\nCreate new project based on Pinax 'account' starter project:\n$ {0}\n\nCreate new project based on development version of 'blog' starter project\n$ {6}\n\nView all Pinax starter projects:\n$ {1}\n\nView all Pinax demo projects:\n$ {2}\n\nView all Pinax apps:\n$ {3}\n\nView all Pinax tools:\n$ {4}\n\nView all Pinax themes:\n$ {5}\n\nCommands:\"\"\".format(\n crayons.red('pinax start account my_project'),\n crayons.red('pinax projects'),\n crayons.red('pinax demos'),\n crayons.red('pinax apps'),\n crayons.red('pinax tools'),\n crayons.red('pinax themes'),\n crayons.red('pinax start --dev blog my_project')\n )\n\n help = help.replace('Commands:', additional_help)\n\n return help", "response": "Format the help string."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef concepts(self):\n for measure in self.measures:\n yield measure\n for aggregate in self.aggregates:\n yield aggregate\n for dimension in self.dimensions:\n yield dimension\n for attribute in dimension.attributes:\n yield attribute", "response": "Return all existing concepts i. e. dimensions measures and aggregate and their attributes within the model."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting all concepts matching this ref. For a dimension that is all its attributes but not the dimension itself. For a dimension that is all its attributes but not the dimension itself. For a dimension that is all its attributes but not the dimension itself.", "response": "def match(self, ref):\n \"\"\" Get all concepts matching this ref. For a dimension, that is all\n its attributes, but not the dimension itself. \"\"\"\n try:\n concept = self[ref]\n if not isinstance(concept, Dimension):\n return [concept]\n return [a for a in concept.attributes]\n except KeyError:\n return []"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dumps(obj, mesh_filename=None, *args, **kwargs): # pylint: disable=unused-argument\n '''\n obj: A dictionary mapping names to a 3-dimension array.\n mesh_filename: If provided, this value is included in the \n attribute, which Meshlab doesn't seem to use.\n\n TODO Maybe reconstruct this using xml.etree\n\n '''\n point_template = '\\n'\n file_template = \"\"\"\n \n \n \n \n \n \n \n %s\n \n \"\"\"\n from blmath.numerics import isnumericarray\n\n if not isinstance(obj, dict) or not all([isnumericarray(point) for point in obj.itervalues()]):\n raise ValueError('obj should be a dict of points')\n\n points = '\\n'.join([point_template % (tuple(xyz) + (name,)) for name, xyz in obj.iteritems()])\n\n return file_template % (mesh_filename, points)", "response": "Serializes a dictionary of points to a 3 - dimensional array."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef landm(self, val):\n '''\n Sets landmarks given any of:\n - ppfile\n - ldmk file\n - dict of {name:inds} (i.e. mesh.landm)\n - dict of {name:xyz} (i.e. mesh.landm_xyz)\n - Nx1 array or list of ints (treated as landm, given sequential integers as names)\n - Nx3 array or list of floats (treated as landm_xyz, given sequential integers as names)\n - pkl, json, yaml file containing either of the above dicts or arrays\n '''\n import numpy as np\n\n if val is None:\n self._landm = None\n self._raw_landmarks = None\n elif isinstance(val, basestring):\n self.landm = load_landmarks(val)\n else:\n if not hasattr(val, 'keys'):\n val = {str(ii): v for ii, v in enumerate(val)}\n landm = {}\n landm_xyz = {}\n filtered_landmarks = []\n for k, v in val.iteritems():\n if isinstance(v, (int, long)):\n landm[k] = v\n elif len(v) == 3:\n if np.all(v == [0.0, 0.0, 0.0]):\n filtered_landmarks.append(k)\n landm_xyz[k] = v\n else:\n raise Exception(\"Can't parse landmark %s: %s\" % (k, v))\n if len(filtered_landmarks) > 0:\n import warnings\n warnings.warn(\"WARNING: the following landmarks are positioned at (0.0, 0.0, 0.0) and were ignored: %s\" % \", \".join(filtered_landmarks))\n # We preserve these and calculate everything seperately so that we can recompute_landmarks if v changes\n self._raw_landmarks = {\n 'landm': landm,\n 'landm_xyz': landm_xyz\n }\n self.recompute_landmarks()", "response": "Sets the landmarks given a set of landmarks given any of the keys in val."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nadd a line item record to this Costs object.", "response": "def add(self, lineitem):\n \"\"\"\n Add a line item record to this Costs object.\n \"\"\"\n # Check for a ProductName in the lineitem.\n # If its not there, it is a subtotal line and including it\n # will throw the total cost calculation off. So ignore it.\n if lineitem['ProductName']:\n self._lineitems.append(lineitem)\n if lineitem['BlendedCost']:\n self._blended_cost += lineitem['BlendedCost']\n if lineitem['UnBlendedCost']:\n self._unblended_cost += lineitem['UnBlendedCost']"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a new Costs object with only the entries that match the given filters.", "response": "def filter(self, filters):\n \"\"\"\n Pass in a list of tuples where each tuple represents one filter.\n The first element of the tuple is the name of the column to\n filter on and the second value is a regular expression which\n each value in that column will be compared against. If the\n regular expression matches the value in that column, that\n lineitem will be included in the new Costs object returned.\n\n Example:\n\n filters=[('ProductName', '.*DynamoDB')]\n\n This filter would find all lineitems whose ``ProductName``\n column contains values that end in the string ``DynamoDB``.\n \"\"\"\n subset = Costs(self._columns)\n filters = [(col, re.compile(regex)) for col, regex in filters]\n for lineitem in self._lineitems:\n for filter in filters:\n if filter[1].search(lineitem[filter[0]]) is None:\n continue\n subset.add(lineitem)\n return subset"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nexecute a MySQL query against the database and store the result in the object store.", "response": "def query(self, query, args=None):\n\t\t\"\"\"\n\t\tExecute the passed in query against the database\n\t\t\n\t\t@param query: MySQL Query to execute. %s or %(key)s will be replaced by parameter args sequence\n\t\t@param args: Sequence of value to replace in your query. A mapping may also be used but your query must use %(key)s\n\t\t@author: Nick Verbeck\n\t\t@since: 5/12/2008\n\t\t\"\"\"\n\t\tself.affectedRows = None\n\t\tself.lastError = None\n\t\tcursor = None\n\t\t\n\t\ttry:\n\t\t\ttry:\n\t\t\t\tself._GetConnection()\n\t\t\t\t\n\t\t\t\tlog.logger.debug('Running query \"%s\" with args \"%s\"', query, args)\n\t\t\t\tself.conn.query = query\n\t\t\t\t\n\t\t\t\t#Execute query and store results\n\t\t\t\tcursor = self.conn.getCursor()\n\t\t\t\tself.affectedRows = cursor.execute(query, args)\n\t\t\t\tself.lastInsertID = self.conn.connection.insert_id()\n\t\t\t\tself.rowcount = cursor.rowcount\n\t\t\t\t\n\t\t\t\tlog.logger.debug('Query Resulted in %s affected rows, %s rows returned, %s last insert id', self.affectedRows, self.lastInsertID, self.rowcount)\n\t\t\t\t\n\t\t\t\tself.record = cursor.fetchall()\n\t\t\t\tself.conn.updateCheckTime()\n\t\t\texcept Exception, e:\n\t\t\t\tself.lastError = e\n\t\t\t\tself.affectedRows = None\n\t\tfinally:\n\t\t\tif cursor is not None:\n\t\t\t\tcursor.close()\n\t\t\tself._ReturnConnection()\n\t\t\tif self.lastError is not None:\n\t\t\t\traise self.lastError\n\t\t\telse:\n\t\t\t\treturn self.affectedRows"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nexecute a MySQL query against the database.", "response": "def queryOne(self, query, args=None):\n\t\t\"\"\"\n\t\tExecute the passed in query against the database. \n\t\tUses a Generator & fetchone to reduce your process memory size.\n\t\t\n\t\t@param query: MySQL Query to execute. %s or %(key)s will be replaced by parameter args sequence\n\t\t@param args: Sequence of value to replace in your query. A mapping may also be used but your query must use %(key)s\n\t\t@author: Nick Verbeck\n\t\t@since: 5/12/2008\n\t\t\"\"\"\n\t\t\n\t\tself.affectedRows = None\n\t\tself.lastError = None\n\t\tcursor = None\n\t\ttry:\n\t\t\ttry:\n\t\t\t\tself._GetConnection()\n\t\t\t\tself.conn.query = query\n\t\t\t\t#Execute query\n\t\t\t\tcursor = self.conn.getCursor()\n\t\t\t\tself.affectedRows = cursor.execute(query, args)\n\t\t\t\tself.conn.updateCheckTime()\n\t\t\t\twhile 1:\n\t\t\t\t\trow = cursor.fetchone()\n\t\t\t\t\tif row is None:\n\t\t\t\t\t\tbreak\n\t\t\t\t\telse:\n\t\t\t\t\t\tself.record = row\n\t\t\t\t\t\tyield row\n\t\t\t\t\t\t\n\t\t\t\tself.rowcount = cursor.rowcount\n\t\t\texcept Exception, e:\n\t\t\t\tself.lastError = e\n\t\t\t\tself.affectedRows = None\n\t\tfinally:\n\t\t\tif cursor is not None:\n\t\t\t\tcursor.close()\n\t\t\tself._ReturnConnection()\n\t\t\tif self.lastError is not None:\n\t\t\t\traise self.lastError\n\t\t\telse:\n\t\t\t\traise StopIteration"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nexecute a series of the same Insert Statments Each tuple in the args list will be applied to the query and executed.", "response": "def queryMany(self, query, args):\n\t\t\"\"\"\n\t\tExecutes a series of the same Insert Statments\n\t\t\n\t\tEach tuple in the args list will be applied to the query and executed.\n\t\tThis is the equivilant of MySQLDB.cursor.executemany()\n\t\t\n\t\t@author: Nick Verbeck\n\t\t@since: 9/7/2008\n\t\t\"\"\"\n\t\tself.lastError = None\n\t\tself.affectedRows = None\n\t\tself.rowcount = None\n\t\tself.record = None\n\t\tcursor = None\n\t\t\n\t\ttry:\n\t\t\ttry:\n\t\t\t\tself._GetConnection()\n\t\t\t\tself.conn.query = query\n\t\t\t\t#Execute query and store results\n\t\t\t\tcursor = self.conn.getCursor()\n\t\t\t\tself.affectedRows = cursor.executemany(query, args)\n\t\t\t\tself.conn.updateCheckTime()\n\t\t\texcept Exception, e:\n\t\t\t\tself.lastError = e\n\t\tfinally:\n\t\t\tif cursor is not None:\n\t\t\t\tcursor.close()\n\t\t\tself._ReturnConnection()\n\t\t\tif self.lastError is not None:\n\t\t\t\traise self.lastError\n\t\t\telse:\n\t\t\t\treturn self.affectedRows"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nexecuting a series of Delete Inserts & Updates in the Queires list.", "response": "def queryMulti(self, queries):\n\t\t\"\"\"\n\t\tExecute a series of Deletes,Inserts, & Updates in the Queires List\n\t\t\n\t\t@author: Nick Verbeck\n\t\t@since: 9/7/2008\n\t\t\"\"\"\n\t\tself.lastError = None\n\t\tself.affectedRows = 0\n\t\tself.rowcount = None\n\t\tself.record = None\n\t\tcursor = None\n\t\t\n\t\ttry:\n\t\t\ttry:\n\t\t\t\tself._GetConnection()\n\t\t\t\t#Execute query and store results\n\t\t\t\tcursor = self.conn.getCursor()\n\t\t\t\tfor query in queries:\n\t\t\t\t\tself.conn.query = query\n\t\t\t\t\tif query.__class__ == [].__class__:\n\t\t\t\t\t\tself.affectedRows += cursor.execute(query[0], query[1])\n\t\t\t\t\telse:\n\t\t\t\t\t\tself.affectedRows += cursor.execute(query)\n\t\t\t\tself.conn.updateCheckTime()\n\t\t\texcept Exception, e:\n\t\t\t\tself.lastError = e\n\t\tfinally:\n\t\t\tif cursor is not None:\n\t\t\t\tcursor.close()\n\t\t\tself._ReturnConnection()\n\t\t\tif self.lastError is not None:\n\t\t\t\traise self.lastError\n\t\t\telse:\n\t\t\t\treturn self.affectedRows"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _GetConnection(self):\n\t\t#Attempt to get a connection. If all connections are in use and we have reached the max number of connections,\n\t\t#we wait 1 second and try again.\n\t\t#The Connection is returned locked to be thread safe\n\t\twhile self.conn is None:\n\t\t\tself.conn = Pool().GetConnection(self.connInfo)\n\t\t\tif self.conn is not None:\n\t\t\t\tbreak\n\t\t\telse:\n\t\t\t\ttime.sleep(1)", "response": "Get a prelocked connection from the Pool."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a connection back to the pool Pool.", "response": "def _ReturnConnection(self):\n\t\t\"\"\"\n\t\tReturns a connection back to the pool\n\t\t\n\t\t@author: Nick Verbeck\n\t\t@since: 9/7/2008\n\t\t\"\"\"\n\t\tif self.conn is not None:\n\t\t\tif self.connInfo.commitOnEnd is True or self.commitOnEnd is True:\n\t\t\t\tself.conn.Commit()\n\t\t\t\t\t\n\t\t\tPool().returnConnection(self.conn)\n\t\t\tself.conn = None"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nbinds the fact table and column to the key.", "response": "def bind(self, cube):\n \"\"\" When one column needs to match, use the key. \"\"\"\n if self.measure:\n table, column = self.measure.bind(cube)\n else:\n table, column = cube.fact_table, cube.fact_pk\n # apply the SQL aggregation function:\n column = getattr(func, self.function)(column)\n column = column.label(self.ref)\n column.quote = True\n return table, column"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef retrieve_public_key(user_repo):\n url = 'https://api.travis-ci.org/repos/{}/key' .format(user_repo)\n response = requests.get(url)\n\n try:\n return response.json()['key'].replace(' RSA ', ' ')\n except KeyError:\n username, repository = user_repo.split('/')\n raise InvalidCredentialsError(\"Either the username: '{}' or the repository: '{}' does not exist. Please enter a valid username or repository name. The username and repository name are both case sensitive.\" .format(username, repository))", "response": "Retrieve the public key from the Travis API."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef encrypt_key(key, password):\n public_key = load_pem_public_key(key.encode(), default_backend())\n encrypted_password = public_key.encrypt(password, PKCS1v15())\n return base64.b64encode(encrypted_password).decode('ascii')", "response": "Encrypt the password with the public key and return an ASCII representation."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dump_travis_configuration(config, path):\n with open(path, 'w') as config_file:\n ordered_dump(config, config_file, default_flow_style=False)", "response": "Dump the configuration settings into the travis. yml file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load_texture(self, texture_version):\n '''\n Expect a texture version number as an integer, load the texture version from /is/ps/shared/data/body/template/texture_coordinates/.\n Currently there are versions [0, 1, 2, 3] availiable.\n '''\n import numpy as np\n lowres_tex_template = 's3://bodylabs-korper-assets/is/ps/shared/data/body/template/texture_coordinates/textured_template_low_v%d.obj' % texture_version\n highres_tex_template = 's3://bodylabs-korper-assets/is/ps/shared/data/body/template/texture_coordinates/textured_template_high_v%d.obj' % texture_version\n from lace.mesh import Mesh\n from lace.cache import sc\n mesh_with_texture = Mesh(filename=sc(lowres_tex_template))\n if not np.all(mesh_with_texture.f.shape == self.f.shape):\n mesh_with_texture = Mesh(filename=sc(highres_tex_template))\n self.transfer_texture(mesh_with_texture)", "response": "Load the texture from the file."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef show_distribution_section(config, title, section_name):\n payload = requests.get(config.apps_url).json()\n distributions = sorted(payload.keys(), reverse=True)\n latest_distribution = payload[distributions[0]]\n click.echo(\"{} {}\".format(\"Release\".rjust(7), title))\n click.echo(\"------- ---------------\")\n section = latest_distribution[section_name]\n names = sorted(section.keys())\n for name in names:\n click.echo(\"{} {}\".format(section[name].rjust(7), name))", "response": "Show latest distribution section."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef validate_django_compatible_with_python():\n python_version = sys.version[:5]\n django_version = django.get_version()\n if sys.version_info == (2, 7) and django_version >= \"2\":\n click.BadArgumentUsage(\"Please install Django v1.11 for Python {}, or switch to Python >= v3.4\".format(python_version))", "response": "Verify that Django 1. 11 is present and that Python 2. 7 is active."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\noverrides for showing commands in particular order", "response": "def list_commands(self, ctx):\n \"\"\"Override for showing commands in particular order\"\"\"\n commands = super(PinaxGroup, self).list_commands(ctx)\n return [cmd for cmd in order_manually(commands)]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn all of the faces that contain at least one of the vertices in v_indices.", "response": "def all_edges_with_verts(self, v_indices, as_boolean=False):\n '''\n returns all of the faces that contain at least one of the vertices in v_indices\n '''\n included_vertices = np.zeros(self.v.shape[0], dtype=bool)\n included_vertices[v_indices] = True\n edges_with_verts = included_vertices[self.e].all(axis=1)\n if as_boolean:\n return edges_with_verts\n return np.nonzero(edges_with_verts)[0]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef keep_vertices(self, indices_to_keep, ret_kept_edges=False):\n '''\n Keep the given vertices and discard the others, and any edges to which\n they may belong.\n\n\n If `ret_kept_edges` is `True`, return the original indices of the kept\n edges. Otherwise return `self` for chaining.\n\n '''\n\n if self.v is None:\n return\n\n initial_num_verts = self.v.shape[0]\n if self.e is not None:\n initial_num_edges = self.e.shape[0]\n e_indices_to_keep = self.all_edges_with_verts(indices_to_keep, as_boolean=True)\n\n self.v = self.v[indices_to_keep]\n if self.e is not None:\n v_old_to_new = np.zeros(initial_num_verts, dtype=int)\n e_old_to_new = np.zeros(initial_num_edges, dtype=int)\n\n v_old_to_new[indices_to_keep] = np.arange(len(indices_to_keep), dtype=int)\n self.e = v_old_to_new[self.e[e_indices_to_keep]]\n e_old_to_new[e_indices_to_keep] = np.arange(self.e.shape[0], dtype=int)\n else:\n e_indices_to_keep = []\n\n return np.nonzero(e_indices_to_keep)[0] if ret_kept_edges else self", "response": "Keep the given vertices and discard the others and any edges to which\n they may belong."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef read(filename):\n mesh = meshio.read(filename)\n\n # make sure to include the used nodes only\n if \"tetra\" in mesh.cells:\n points, cells = _sanitize(mesh.points, mesh.cells[\"tetra\"])\n return (\n MeshTetra(points, cells),\n mesh.point_data,\n mesh.cell_data,\n mesh.field_data,\n )\n elif \"triangle\" in mesh.cells:\n points, cells = _sanitize(mesh.points, mesh.cells[\"triangle\"])\n return (\n MeshTri(points, cells),\n mesh.point_data,\n mesh.cell_data,\n mesh.field_data,\n )\n else:\n raise RuntimeError(\"Unknown mesh type.\")", "response": "Reads an unstructured mesh with added data."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef lock(self, block=True):\n\t\tself._locked = True\n\t\treturn self._lock.acquire(block)", "response": "Lock connection from being used else where"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nrelease the connection lock.", "response": "def release(self):\n\t\t\"\"\"\n\t\tRelease the connection lock\n\t\t\"\"\"\n\t\tif self._locked is True:\n\t\t\tself._locked = False\n\t\t\tself._lock.release()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef getCursor(self):\n\t\tif self.connection is None:\n\t\t\tself.Connect()\n\t\t\t\n\t\treturn self.connection.cursor(MySQLdb.cursors.DictCursor)", "response": "Get a Dictionary Cursor for executing queries\n\t"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a new physical connection to the database and updates the check time.", "response": "def Connect(self):\n\t\t\"\"\"\n\t\tCreates a new physical connection to the database\n\t\t\n\t\t@author: Nick Verbeck\n\t\t@since: 5/12/2008\n\t\t\"\"\"\n\t\tif self.connection is None:\n\t\t\tself.connection = MySQLdb.connect(*[], **self.connectionInfo.info)\n\t\t\t\n\t\tif self.connectionInfo.commitOnEnd is True:\n\t\t\tself.connection.autocommit()\n\t\t\t\n\t\tself._updateCheckTime()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nbeing a Transaction @author: Nick Verbeck @since: 5/14/2011", "response": "def being(self):\n\t\t\"\"\"\n\t\tBeing a Transaction\n\t\t\n\t\t@author: Nick Verbeck\n\t\t@since: 5/14/2011\n\t\t\"\"\"\n\t\ttry:\n\t\t\tif self.connection is not None:\n\t\t\t\tself.lock()\n\t\t\t\tc = self.getCursor()\n\t\t\t\tc.execute('BEGIN;')\n\t\t\t\tc.close()\n\t\texcept Exception, e:\n\t\t\tpass"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef commit(self):\n\t\ttry:\n\t\t\tif self.connection is not None:\n\t\t\t\tself.connection.commit()\n\t\t\t\tself._updateCheckTime()\n\t\t\t\tself.release()\n\t\texcept Exception, e:\n\t\t\tpass", "response": "Commits the current transaction to the database."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrolling back the current transaction to the database.", "response": "def rollback(self):\n\t\t\"\"\"\n\t\tRollback MySQL Transaction to database.\n\t\tMySQLDB: If the database and tables support transactions, this rolls \n\t\tback (cancels) the current transaction; otherwise a \n\t\tNotSupportedError is raised.\n\t\t\n\t\t@author: Nick Verbeck\n\t\t@since: 5/12/2008\n\t\t\"\"\"\n\t\ttry:\n\t\t\tif self.connection is not None:\n\t\t\t\tself.connection.rollback()\n\t\t\t\tself._updateCheckTime()\n\t\t\t\tself.release()\n\t\texcept Exception, e:\n\t\t\tpass"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef Close(self):\n\t\tif self.connection is not None:\n\t\t\ttry:\n\t\t\t\tself.connection.commit()\n\t\t\t\tself.connection.close()\n\t\t\t\tself.connection = None\n\t\t\texcept Exception, e:\n\t\t\t\tpass", "response": "Closes the current connection."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef configure_api(app, manager):\n if not hasattr(app, 'extensions'):\n app.extensions = {} # pragma: nocover\n app.extensions['babbage'] = manager\n return blueprint", "response": "Configure the current Flask app with an instance of cubemanager."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_cube(name):\n manager = get_manager()\n if not manager.has_cube(name):\n raise NotFound('No such cube: %r' % name)\n return manager.get_cube(name)", "response": "Load the named cube from the current registered cube manager."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef cubes():\n cubes = []\n for cube in get_manager().list_cubes():\n cubes.append({\n 'name': cube\n })\n return jsonify({\n 'status': 'ok',\n 'data': cubes\n })", "response": "Get a listing of all publicly available cubes."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nperform an aggregation request.", "response": "def aggregate(name):\n \"\"\" Perform an aggregation request. \"\"\"\n cube = get_cube(name)\n result = cube.aggregate(aggregates=request.args.get('aggregates'),\n drilldowns=request.args.get('drilldown'),\n cuts=request.args.get('cut'),\n order=request.args.get('order'),\n page=request.args.get('page'),\n page_size=request.args.get('pagesize'))\n result['status'] = 'ok'\n\n if request.args.get('format', '').lower() == 'csv':\n return create_csv_response(result['cells'])\n else:\n return jsonify(result)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef facts(name):\n cube = get_cube(name)\n result = cube.facts(fields=request.args.get('fields'),\n cuts=request.args.get('cut'),\n order=request.args.get('order'),\n page=request.args.get('page'),\n page_size=request.args.get('pagesize'))\n result['status'] = 'ok'\n return jsonify(result)", "response": "List the fact table entries in the current cube. This is the full\n materialized dataset."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef members(name, ref):\n cube = get_cube(name)\n result = cube.members(ref, cuts=request.args.get('cut'),\n order=request.args.get('order'),\n page=request.args.get('page'),\n page_size=request.args.get('pagesize'))\n result['status'] = 'ok'\n return jsonify(result)", "response": "List the members of a specific dimension or the distinct values of a\n given attribute."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef apply(self, q, bindings, drilldowns):\n info = []\n for drilldown in self.parse(drilldowns):\n for attribute in self.cube.model.match(drilldown):\n info.append(attribute.ref)\n table, column = attribute.bind(self.cube)\n bindings.append(Binding(table, attribute.ref))\n q = q.column(column)\n q = q.group_by(column)\n return info, q, bindings", "response": "Apply a set of grouping criteria and project them."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncheck if the attribute exists on the dimension.", "response": "def check_attribute_exists(instance):\n \"\"\" Additional check for the dimension model, to ensure that attributes\n given as the key and label attribute on the dimension exist. \"\"\"\n attributes = instance.get('attributes', {}).keys()\n if instance.get('key_attribute') not in attributes:\n return False\n label_attr = instance.get('label_attribute')\n if label_attr and label_attr not in attributes:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if the hierarchies model is valid.", "response": "def check_valid_hierarchies(instance):\n \"\"\" Additional check for the hierarchies model, to ensure that levels\n given are pointing to actual dimensions \"\"\"\n hierarchies = instance.get('hierarchies', {}).values()\n dimensions = set(instance.get('dimensions', {}).keys())\n all_levels = set()\n for hierarcy in hierarchies:\n levels = set(hierarcy.get('levels', []))\n if len(all_levels.intersection(levels)) > 0:\n # Dimension appears in two different hierarchies\n return False\n all_levels = all_levels.union(levels)\n if not dimensions.issuperset(levels):\n # Level which is not in a dimension\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nload the JSON Schema Draft 4 validator with the given name from the local schema directory.", "response": "def load_validator(name):\n \"\"\" Load the JSON Schema Draft 4 validator with the given name from the\n local schema directory. \"\"\"\n with open(os.path.join(SCHEMA_PATH, name)) as fh:\n schema = json.load(fh)\n Draft4Validator.check_schema(schema)\n return Draft4Validator(schema, format_checker=checker)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngive a name construct a cube and return it.", "response": "def get_cube(self, name):\n \"\"\" Given a cube name, construct that cube and return it. Do not\n overwrite this method unless you need to. \"\"\"\n return Cube(self.get_engine(), name, self.get_cube_model(name))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef list_cubes(self):\n for file_name in os.listdir(self.directory):\n if '.' in file_name:\n name, ext = file_name.rsplit('.', 1)\n if ext.lower() == 'json':\n yield name", "response": "List all available JSON files."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef Terminate(self):\n\t\t\n\t\tself.lock.acquire()\n\t\ttry:\n\t\t\tfor bucket in self.connections.values():\n\t\t\t\ttry:\n\t\t\t\t\tfor conn in bucket:\n\t\t\t\t\t\tconn.lock()\n\t\t\t\t\t\ttry:\n\t\t\t\t\t\t\tconn.Close()\n\t\t\t\t\t\texcept Exception:\n\t\t\t\t\t\t\t#We may throw exceptions due to already closed connections\n\t\t\t\t\t\t\tpass\n\t\t\t\t\t\tconn.release()\n\t\t\t\texcept Exception:\n\t\t\t\t\tpass\n\t\t\tself.connections = {}\n\t\tfinally:\n\t\t\tself.lock.release()", "response": "This method closes all open connections and commits all queries and closes all the connections."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef Commit(self):\n\t\tself.lock.acquire()\n\t\ttry:\n\t\t\tfor bucket in self.connections.values():\n\t\t\t\ttry:\n\t\t\t\t\tfor conn in bucket:\n\t\t\t\t\t\tconn.lock()\n\t\t\t\t\t\ttry:\n\t\t\t\t\t\t\tconn.commit()\n\t\t\t\t\t\t\tconn.release()\n\t\t\t\t\t\texcept Exception:\n\t\t\t\t\t\t\tconn.release()\n\t\t\t\texcept Exception:\n\t\t\t\t\tpass\n\t\tfinally:\n\t\t\tself.lock.release()", "response": "Commits all currently open connections."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting a Open and active connection Returns a PySQLConnectionManager", "response": "def GetConnection(self, ConnectionObj):\n\t\t\"\"\"\n\t\tGet a Open and active connection\n\t\t\n\t\tReturns a PySQLConnectionManager if one is open else it will create a new one if the max active connections hasn't been hit.\n\t\tIf all possible connections are used. Then None is returned.\n\t\t\n\t\t@param PySQLConnectionObj: PySQLConnection Object representing your connection string\n\t\t@author: Nick Verbeck\n\t\t@since: 5/12/2008 \n\t\t\"\"\"\n\t\t\n\t\tkey = ConnectionObj.getKey()\n\t\t\n\t\tconnection = None\n\t\t\n\t\tif self.connections.has_key(key):\n\t\t\tconnection = self._getConnectionFromPoolSet(key)\n\t\t\t\n\t\t\tif connection is None:\n\t\t\t\tself.lock.acquire()\n\t\t\t\tif len(self.connections[key]) < self.maxActiveConnections:\n\t\t\t\t\t#Create a new connection\n\t\t\t\t\tconnection = self._createConnection(ConnectionObj)\n\t\t\t\t\tself.connections[key].append(connection)\n\t\t\t\t\tself.lock.release()\n\t\t\t\telse:\n\t\t\t\t\t#Wait for a free connection. We maintain the lock on the pool so we are the 1st to get a connection.\n\t\t\t\t\twhile connection is None:\n\t\t\t\t\t\tconnection = self._getConnectionFromPoolSet(key)\n\t\t\t\t\tself.lock.release()\n\t\t\t\t\t\n\t\t#Create new Connection Pool Set\n\t\telse:\n\t\t\tself.lock.acquire()\n\t\t\t#We do a double check now that its locked to be sure some other thread didn't create this while we may have been waiting.\n\t\t\tif not self.connections.has_key(key):\n\t\t\t\tself.connections[key] = []\n\t\t\t\n\t\t\tif len(self.connections[key]) < self.maxActiveConnections:\n\t\t\t\t#Create a new connection\n\t\t\t\tconnection = self._createConnection(ConnectionObj)\n\t\t\t\tself.connections[key].append(connection)\n\t\t\telse:\n\t\t\t\t#A rare thing happened. So many threads created connections so fast we need to wait for a free one.\n\t\t\t\twhile connection is None:\n\t\t\t\t\tconnection = self._getConnectionFromPoolSet(key)\n\t\t\tself.lock.release()\n\t\t\t\n\t\treturn connection"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngenerate code for name_to_rgb dict assuming an rgb. txt file available", "response": "def main():\n \"\"\" Generates code for name_to_rgb dict, assuming an rgb.txt file available (in X11 format).\"\"\"\n import re\n with open('rgb.txt') as fp:\n line = fp.readline()\n while line:\n reg = re.match(r'\\s*(\\d+)\\s*(\\d+)\\s*(\\d+)\\s*(\\w.*\\w).*', line)\n if reg:\n r = int(reg.group(1)) / 255.\n g = int(reg.group(2)) / 255.\n b = int(reg.group(3)) / 255.\n d = reg.group(4)\n print \"'%s' : np.array([%.2f, %.2f, %.2f]),\" % (d, r, g, b)\n line = fp.readline()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\nasync def on_date(self, date: datetime.date) -> dict:\n return await self._request(\n 'get', 'dailystats/{0}'.format(date.strftime('%Y-%m-%d')))", "response": "Get statistics for a certain date."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\nasync def upcoming(self, details: bool = False) -> list:\n endpoint = 'dailystats'\n key = 'DailyStats'\n if details:\n endpoint += '/details'\n key = 'DailyStatsDetails'\n data = await self._request('get', endpoint)\n return data[key]", "response": "Return watering statistics for the next 6 days."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nraising an error from the remote API if necessary.", "response": "def _raise_for_remote_status(url: str, data: dict) -> None:\n \"\"\"Raise an error from the remote API if necessary.\"\"\"\n if data.get('errorType') and data['errorType'] > 0:\n raise_remote_error(data['errorType'])\n\n if data.get('statusCode') and data['statusCode'] != 200:\n raise RequestError(\n 'Error requesting data from {0}: {1} {2}'.format(\n url, data['statusCode'], data['message']))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nauthenticate against a RainMachine device.", "response": "async def login(\n host: str,\n password: str,\n websession: ClientSession,\n *,\n port: int = 8080,\n ssl: bool = True,\n request_timeout: int = DEFAULT_TIMEOUT) -> Controller:\n \"\"\"Authenticate against a RainMachine device.\"\"\"\n print('regenmaschine.client.login() is deprecated; see documentation!')\n client = Client(websession, request_timeout)\n await client.load_local(host, password, port, ssl)\n return next(iter(client.controllers.values()))"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates a local client.", "response": "async def load_local( # pylint: disable=too-many-arguments\n self,\n host: str,\n password: str,\n port: int = DEFAULT_LOCAL_PORT,\n ssl: bool = True,\n skip_existing: bool = True) -> None:\n \"\"\"Create a local client.\"\"\"\n controller = LocalController(\n self._request, host, port, ssl, self._websession)\n await controller.login(password)\n\n wifi_data = await controller.provisioning.wifi()\n if skip_existing and wifi_data['macAddress'] in self.controllers:\n return\n\n version_data = await controller.api.versions()\n controller.api_version = version_data['apiVer']\n controller.hardware_version = version_data['hwVer']\n controller.mac = wifi_data['macAddress']\n controller.name = await controller.provisioning.device_name\n controller.software_version = version_data['swVer']\n\n self.controllers[controller.mac] = controller"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nload a remote client.", "response": "async def load_remote(\n self, email: str, password: str,\n skip_existing: bool = True) -> None:\n \"\"\"Create a local client.\"\"\"\n auth_resp = await self._request(\n 'post',\n 'https://my.rainmachine.com/login/auth',\n json={'user': {\n 'email': email,\n 'pwd': password,\n 'remember': 1\n }})\n\n access_token = auth_resp['access_token']\n\n sprinklers_resp = await self._request(\n 'post',\n 'https://my.rainmachine.com/devices/get-sprinklers',\n access_token=access_token,\n json={'user': {\n 'email': email,\n 'pwd': password,\n 'remember': 1\n }})\n\n for sprinkler in sprinklers_resp['sprinklers']:\n if skip_existing and sprinkler['mac'] in self.controllers:\n continue\n\n controller = RemoteController(self._request, self._websession)\n await controller.login(\n access_token, sprinkler['sprinklerId'], password)\n\n version_data = await controller.api.versions()\n controller.api_version = version_data['apiVer']\n controller.hardware_version = version_data['hwVer']\n controller.mac = sprinkler['mac']\n controller.name = sprinkler['name']\n controller.software_version = version_data['swVer']\n\n self.controllers[sprinkler['mac']] = controller"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nmakes a request against the RainMachine device.", "response": "async def _request(\n self,\n method: str,\n url: str,\n *,\n access_token: str = None,\n access_token_expiration: datetime = None,\n headers: dict = None,\n params: dict = None,\n json: dict = None,\n ssl: bool = True) -> dict:\n \"\"\"Make a request against the RainMachine device.\"\"\"\n if (access_token_expiration\n and datetime.now() >= access_token_expiration):\n raise TokenExpiredError('Long-lived access token has expired')\n\n if not headers:\n headers = {}\n headers.update({'Content-Type': 'application/json'})\n\n if not params:\n params = {}\n if access_token:\n params.update({'access_token': access_token})\n\n try:\n async with async_timeout.timeout(self.request_timeout):\n async with self._websession.request(\n method, url, headers=headers, params=params, json=json,\n ssl=ssl) as resp:\n resp.raise_for_status()\n data = await resp.json(content_type=None)\n _raise_for_remote_status(url, data)\n except ClientError as err:\n raise RequestError(\n 'Error requesting data from {0}: {1}'.format(url, err))\n except asyncio.TimeoutError:\n raise RequestError('Timeout during request: {0}'.format(url))\n\n return data"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef raise_remote_error(error_code: int) -> None:\n try:\n error = next((v for k, v in ERROR_CODES.items() if k == error_code))\n raise RequestError(error)\n except StopIteration:\n raise RequestError(\n 'Unknown remote error code returned: {0}'.format(error_code))", "response": "Raise the appropriate error with a remote error code."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns first and last day of a quarter", "response": "def quarter_boundaries(quarter):\n \"\"\"Returns first and last day of a quarter\n\n Args:\n quarter (str) quarter, in format '2015Q1'\n\n Returns: (tuple) datetime.dates for the first and last days of the quarter\n \"\"\"\n year, quarter = quarter.split('Q')\n year = int(year)\n quarter = int(quarter)\n first_month_of_quarter = 3 * quarter - 2\n last_month_of_quarter = 3 * quarter\n first_day = date(year, first_month_of_quarter, 1)\n last_day = date(year, last_month_of_quarter, monthrange(year, last_month_of_quarter)[1])\n return first_day, last_day"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef metta_config(quarter, num_dimensions):\n first_day, last_day = quarter_boundaries(quarter)\n return {\n 'start_time': first_day,\n 'end_time': last_day,\n 'prediction_window': 3, # ???\n 'label_name': 'onet_soc_code',\n 'label_type': 'categorical',\n 'matrix_id': 'job_postings_{}'.format(quarter),\n 'feature_names': ['doc2vec_{}'.format(i) for i in range(num_dimensions)],\n }", "response": "Returns metta metadata suitable for a quarter s SOC code classifier matrix\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef upload_to_metta(train_features_path, train_labels_path, test_features_path, test_labels_path, train_quarter, test_quarter, num_dimensions):\n train_config = metta_config(train_quarter, num_dimensions)\n test_config = metta_config(test_quarter, num_dimensions)\n\n X_train = pd.read_csv(train_features_path, sep=',')\n X_train.columns = ['doc2vec_'+str(i) for i in range(X_train.shape[1])]\n #X_train['label'] = pd.Series([randint(0,23) for i in range(len(X_train))])\n Y_train = pd.read_csv(train_labels_path)\n Y_train.columns = ['onet_soc_code']\n train = pd.concat([X_train, Y_train], axis=1)\n\n X_test = pd.read_csv(test_features_path, sep=',')\n X_test.columns = ['doc2vec_'+str(i) for i in range(X_test.shape[1])]\n #X_test['label'] = pd.Series([randint(0,23) for i in range(len(X_test))])\n Y_test = pd.read_csv(test_labels_path)\n Y_test.columns = ['onet_soc_code']\n test = pd.concat([X_test, Y_test], axis=1)\n #print(train.head())\n #print(train.shape)\n #print(test.head())\n #print(test.shape)\n metta.archive_train_test(\n train_config,\n X_train,\n test_config,\n X_test,\n directory='wdi'\n )", "response": "Uploads train and test matrices to Metta."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef usedby_packages(self, deps_file_path=None, depslock_file_path=None, packages=None):\n if deps_file_path is None:\n deps_file_path = self._deps_path\n if depslock_file_path is None:\n depslock_file_path = self._depslock_path\n if deps_file_path == depslock_file_path:\n depslock_file_path += '.lock'\n\n if packages is None:\n self.search_dependencies(deps_file_path)\n else:\n self._root_package.packages = packages\n self._log.info('Done!')", "response": "Search the dependencies of the specified packages."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nupload the given file to the s3 object", "response": "def upload(s3_conn, filepath, s3_path):\n \"\"\"Uploads the given file to s3\n\n Args:\n s3_conn: (boto.s3.connection) an s3 connection\n filepath (str) the local filename\n s3_path (str) the destination path on s3\n \"\"\"\n bucket_name, prefix = split_s3_path(s3_path)\n bucket = s3_conn.get_bucket(bucket_name)\n filename = os.path.basename(filepath)\n\n key = boto.s3.key.Key(\n bucket=bucket,\n name='{}/{}'.format(prefix, filename)\n )\n logging.info('uploading from %s to %s', filepath, key)\n key.set_contents_from_filename(filepath)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef upload_dict(s3_conn, s3_prefix, data_to_sync):\n bucket_name, prefix = split_s3_path(s3_prefix)\n bucket = s3_conn.get_bucket(bucket_name)\n\n for key, value in data_to_sync.items():\n full_name = '{}/{}.json'.format(prefix, key)\n s3_key = boto.s3.key.Key(\n bucket=bucket,\n name=full_name\n )\n logging.info('uploading key %s', full_name)\n s3_key.set_contents_from_string(json.dumps(value))", "response": "Syncs a dictionary to an S3 bucket"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef download(s3_conn, out_filename, s3_path):\n bucket_name, prefix = split_s3_path(s3_path)\n bucket = s3_conn.get_bucket(bucket_name)\n key = boto.s3.key.Key(\n bucket=bucket,\n name=prefix\n )\n logging.info('loading from %s into %s', key, out_filename)\n key.get_contents_to_filename(out_filename, cb=log_download_progress)", "response": "Downloads the given s3_path into out_filename."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\nasync def _request(\n self,\n method: str,\n endpoint: str,\n *,\n headers: dict = None,\n params: dict = None,\n json: dict = None,\n ssl: bool = True) -> dict:\n \"\"\"Wrap the generic request method to add access token, etc.\"\"\"\n return await self._client_request(\n method,\n '{0}/{1}'.format(self._host, endpoint),\n access_token=self._access_token,\n access_token_expiration=self._access_token_expiration,\n headers=headers,\n params=params,\n json=json,\n ssl=ssl)", "response": "Wrap the generic request method to add access token and expiration."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\nasync def login(self, password):\n auth_resp = await self._client_request(\n 'post',\n '{0}/auth/login'.format(self._host),\n json={\n 'pwd': password,\n 'remember': 1\n })\n\n self._access_token = auth_resp['access_token']\n self._access_token_expiration = datetime.now() + timedelta(\n seconds=int(auth_resp['expires_in']) - 10)", "response": "Authenticate against the device locally."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nauthenticating against the device.", "response": "async def login(\n self, stage_1_access_token: str, sprinkler_id: str,\n password: str) -> None:\n \"\"\"Authenticate against the device (remotely).\"\"\"\n auth_resp = await self._client_request(\n 'post',\n 'https://my.rainmachine.com/devices/login-sprinkler',\n access_token=stage_1_access_token,\n json={\n 'sprinklerId': sprinkler_id,\n 'pwd': password,\n })\n\n self._access_token = auth_resp['access_token']\n self._host = URL_BASE_REMOTE.format(sprinkler_id)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting the duplicated flag for all package set flag duplicated", "response": "def set_duplicated_flag(self):\n \"\"\"\n For all package set flag duplicated, if it's not unique package\n :return:\n \"\"\"\n package_by_name = defaultdict(list)\n\n for package1 in self._root_package.all_packages:\n if package1 is None:\n continue\n pkg_name = package1.package_name\n param_list = self._config.get_fails('unique', {})\n params1 = package1.get_params(param_list)\n for package2 in package_by_name[pkg_name]:\n params2 = package2.get_params(param_list)\n for x in param_list:\n # START HACK for cached archive\n param1 = params1[x]\n param2 = params2[x]\n if isinstance(param1, list):\n param1 = [str(x) for x in param1]\n if isinstance(param2, list):\n param2 = [str(x) for x in param2]\n # END\n\n if str(param1) != str(param2):\n package1.duplicated = True\n package2.duplicated = True\n package_by_name[pkg_name].append(package1)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn python object from string", "response": "def get_object_from_string(object_path):\n \"\"\"\n Return python object from string\n :param object_path: e.g os.path.join\n :return: python object\n \"\"\"\n # split like crosspm.template.GUS => crosspm.template, GUS\n try:\n module_name, object_name = object_path.rsplit('.', maxsplit=1)\n module_ = __import__(module_name, globals(), locals(), ['App'], 0)\n variable_ = getattr(module_, object_name)\n except Exception:\n variable_ = None\n return variable_"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nposts data to a non - existing zone.", "response": "async def _post(self, zone_id: int = None, json: dict = None) -> dict:\n \"\"\"Post data to a (non)existing zone.\"\"\"\n return await self._request(\n 'post', 'zone/{0}/properties'.format(zone_id), json=json)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\nasync def all(\n self, *, details: bool = False,\n include_inactive: bool = False) -> list:\n \"\"\"Return all zones (with optional advanced properties).\"\"\"\n endpoint = 'zone'\n if details:\n endpoint += '/properties'\n data = await self._request('get', endpoint)\n return [z for z in data['zones'] if include_inactive or z['active']]", "response": "Return all zones in the system."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a specific zone.", "response": "async def get(self, zone_id: int, *, details: bool = False) -> dict:\n \"\"\"Return a specific zone.\"\"\"\n endpoint = 'zone/{0}'.format(zone_id)\n if details:\n endpoint += '/properties'\n return await self._request('get', endpoint)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef observable_method(func, strategy='instances'):\n if strategy == 'instances':\n return ObservableMethodManager_PersistOnInstances(func)\n elif strategy == 'descriptor':\n return ObservableMethodManager_PersistOnDescriptor(func)\n else:\n raise ValueError(\"Strategy %s not recognized\"%(strategy,))", "response": "A function that can be used to make an observable method."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add_observer(self, observer, identify_observed=False):\n\n # If the observer is a bound method,\n if hasattr(observer, \"__self__\"):\n result = self._add_bound_method(observer, identify_observed)\n # Otherwise, assume observer is a normal function.\n else:\n result = self._add_function(observer, identify_observed)\n return result", "response": "Register an observer to observe me."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nadd a function to the internal dictionary of observers.", "response": "def _add_function(self, func, identify_observed):\n \"\"\"Add a function as an observer.\n\n Args:\n func: The function to register as an observer.\n identify_observed: See docstring for add_observer.\n\n Returns:\n True if the function is added, otherwise False.\n \"\"\"\n\n key = self.make_key(func)\n if key not in self.observers:\n self.observers[key] = ObserverFunction(\n func, identify_observed, (key, self.observers))\n return True\n else:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _add_bound_method(self, bound_method, identify_observed):\n\n inst = bound_method.__self__\n method_name = bound_method.__name__\n key = self.make_key(bound_method)\n if key not in self.observers:\n self.observers[key] = ObserverBoundMethod(\n inst, method_name, identify_observed, (key, self.observers))\n return True\n else:\n return False", "response": "Add an bound method to the internal dictionary."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef make_key(observer):\n\n if hasattr(observer, \"__self__\"):\n inst = observer.__self__\n method_name = observer.__name__\n key = (id(inst), method_name)\n else:\n key = id(observer)\n return key", "response": "Construct a unique hashable immutable key for an observer."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nyield a group from the iterable", "response": "def group(self):\n \"\"\"Yield a group from the iterable\"\"\"\n yield self.current\n # start enumerate at 1 because we already yielded the last saved item\n for num, item in enumerate(self.iterator, 1):\n self.current = item\n if num == self.limit:\n break\n yield item\n else:\n self.on_going = False"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nlock packages. Downloader search packages", "response": "def lock_packages(self, deps_file_path=None, depslock_file_path=None, packages=None):\n \"\"\"\n Lock packages. Downloader search packages\n \"\"\"\n if deps_file_path is None:\n deps_file_path = self._deps_path\n if depslock_file_path is None:\n depslock_file_path = self._depslock_path\n if deps_file_path == depslock_file_path:\n depslock_file_path += '.lock'\n # raise CrosspmException(\n # CROSSPM_ERRORCODE_WRONG_ARGS,\n # 'Dependencies and Lock files are same: \"{}\".'.format(deps_file_path),\n # )\n\n if packages is None:\n self.search_dependencies(deps_file_path)\n else:\n self._root_package.packages = packages\n\n self._log.info('Writing lock file [{}]'.format(depslock_file_path))\n\n output_params = {\n 'out_format': 'lock',\n 'output': depslock_file_path,\n }\n Output(config=self._config).write_output(output_params, self._root_package.packages)\n self._log.info('Done!')"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn True if stdout is set False otherwise.", "response": "def stdout(self):\n \"\"\"\n \u0424\u043b\u0430\u0433 --stdout \u043c\u043e\u0436\u0435\u0442 \u0431\u044b\u0442\u044c \u0432\u0437\u044f\u0442 \u0438\u0437 \u043f\u0435\u0440\u0435\u043c\u0435\u043d\u043d\u043e\u0439 \u043e\u043a\u0440\u0443\u0436\u0435\u043d\u0438\u044f CROSSPM_STDOUT.\n \u0415\u0441\u043b\u0438 \u0435\u0441\u0442\u044c \u043b\u044e\u0431\u043e\u0435 \u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0435 \u0432 CROSSPM_STDOUT - \u043e\u043d\u043e \u043f\u043e\u043d\u0438\u043c\u0430\u0435\u0442\u0441\u044f \u043a\u0430\u043a True\n :return:\n \"\"\"\n # --stdout\n stdout = self._args['--stdout']\n if stdout:\n return True\n\n # CROSSPM_STDOUT\n stdout_env = os.getenv('CROSSPM_STDOUT', None)\n if stdout_env is not None:\n return True\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nprepare args for the command.", "response": "def prepare_args(args, windows=None):\n \"\"\"\n Prepare args - add support for old interface, e.g:\n - --recursive was \"flag\" and for now it support True or False value\n :param args:\n :return:\n \"\"\"\n if windows is None:\n windows = \"win\" in sys.platform\n\n if isinstance(args, str):\n args = shlex.split(args, posix=not windows)\n elif isinstance(args, list):\n pass\n elif args is None:\n args = sys.argv[1:]\n else:\n raise Exception(\"Unknown args type: {}\".format(type(args)))\n\n # --recursive => --recursive=True|False convert\n for position, argument in enumerate(args):\n # Normal way, skip change\n if argument.lower() in ('--recursive=true', '--recursive=false'):\n return args\n\n elif argument.lower() == '--recursive':\n if len(args) > position + 1 and args[position + 1].lower() in [\"true\", \"false\"]:\n # --recursive true | false\n return args\n else:\n # legacy way, convert --recursive to --recursive=true\n args[position] = \"--recursive=True\"\n return args\n return args"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets watering information for X days from Y date.", "response": "async def log(\n self,\n date: datetime.date = None,\n days: int = None,\n details: bool = False) -> list:\n \"\"\"Get watering information for X days from Y date.\"\"\"\n endpoint = 'watering/log'\n if details:\n endpoint += '/details'\n\n if date and days:\n endpoint = '{0}/{1}/{2}'.format(\n endpoint, date.strftime('%Y-%m-%d'), days)\n\n data = await self._request('get', endpoint)\n return data['waterLog']['days']"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn all program runs for X days from Y date.", "response": "async def runs(self, date: datetime.date = None, days: int = None) -> list:\n \"\"\"Return all program runs for X days from Y date.\"\"\"\n endpoint = 'watering/past'\n\n if date and days:\n endpoint = '{0}/{1}/{2}'.format(\n endpoint, date.strftime('%Y-%m-%d'), days)\n\n data = await self._request('get', endpoint)\n return data['pastValues']"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsearches for auth in env cmdline str", "response": "def search_auth(self, list_or_file_path, source):\n \"\"\"\n Looking for auth in env, cmdline, str\n :param list_or_file_path:\n :param source:\n \"\"\"\n _auth = source.args['auth']\n if isinstance(_auth, str):\n if ':' in _auth:\n _auth = _auth.split(':')\n elif _auth.endswith('}') and (\n _auth.startswith('{') or ':' in _auth): # {auth}, {user}:{password}, user:{password}\n _auth = self.get_auth(list_or_file_path, _auth)\n _auth = self.split_auth(_auth)\n\n if isinstance(_auth, list):\n for i in range(len(_auth)):\n if _auth[i].endswith('}') and (\n _auth[i].startswith('{') or ':' in _auth[i]): # {auth}, {user}:{password}, user:{password}\n _auth[i] = self.get_auth(list_or_file_path, _auth[i])\n if ':' in _auth[i]:\n _auth = self.split_auth(_auth[i])\n\n source.args['auth'] = _auth"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nposts data to a non - existing program.", "response": "async def _post(self, program_id: int = None, json: dict = None) -> dict:\n \"\"\"Post data to a (non)existing program.\"\"\"\n return await self._request(\n 'post', 'program/{0}'.format(program_id), json=json)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef cache_json(filename):\n def cache_decorator(cacheable_function):\n @wraps(cacheable_function)\n def cache_wrapper(*args, **kwargs):\n path = CACHE_DIRECTORY + filename\n check_create_folder(path)\n if os.path.exists(path):\n with open(path) as infile:\n return json.load(infile)\n else:\n function_output = cacheable_function(*args, **kwargs)\n with open(path, 'w') as outfile:\n json.dump(function_output, outfile)\n return function_output\n return cache_wrapper\n return cache_decorator", "response": "A function decorator that caches the JSON - serializable output of the function to a given file"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef check_create_folder(filename):\n os.makedirs(os.path.dirname(filename), exist_ok=True)", "response": "Check if the folder exisits. If not create the folder"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_payment_request(self, cart, request):\n shop_ns = resolve(request.path).namespace\n return_url = reverse('{}:{}:return'.format(shop_ns, self.namespace))\n cancel_url = reverse('{}:{}:cancel'.format(shop_ns, self.namespace))\n cart = CartModel.objects.get_from_request(request)\n cart.update(request) # to calculate the total\n auth_token_hash = self.get_auth_token()\n payload = {\n 'url': '{API_ENDPOINT}/v1/payments/payment'.format(**settings.SHOP_PAYPAL),\n 'method': 'POST',\n 'headers': {\n 'Content-Type': 'application/json',\n 'Authorization': '{token_type} {access_token}'.format(**auth_token_hash),\n },\n 'data': {\n 'intent': 'sale',\n 'redirect_urls': {\n 'return_url': request.build_absolute_uri(return_url),\n 'cancel_url': request.build_absolute_uri(cancel_url),\n },\n 'payer': {\n 'payment_method': 'paypal',\n },\n 'transactions': [{\n 'amount': {\n 'total': cart.total.as_decimal(),\n 'currency': cart.total.currency,\n }\n }]\n }\n }\n config = json.dumps(payload, cls=DjangoJSONEncoder)\n success_handler = \"\"\"\n function successCallback(r) {\n console.log(r);\n $window.location.href=r.data.links.filter(function(e){\n return e.rel==='approval_url';\n })[0].href;\n }\"\"\".replace(' ', '').replace('\\n', '')\n error_handler = \"\"\"\n function errorCallback(r) {\n console.error(r);\n }\"\"\".replace(' ', '').replace('\\n', '')\n js_expression = '$http({0}).then({1},{2})'.format(config, success_handler, error_handler)\n return js_expression", "response": "Returns a payment request from the given request"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nyields all job postings in common schema format", "response": "def postings(self, quarter, stats_counter=None):\n \"\"\"Yield job postings in common schema format\n\n Args:\n quarter (str) The quarter, in format '2015Q1'\n stats_counter (object, optional) A counter that can track both\n input and output documents using a 'track' method.\n \"\"\"\n logging.info('Finding postings for %s', quarter)\n for posting in self._iter_postings(quarter):\n transformed = self._transform(posting)\n transformed['id'] = '{}_{}'.format(\n self.partner_id,\n self._id(posting)\n )\n if stats_counter:\n stats_counter.track(\n input_document=posting,\n output_document=transformed\n )\n yield transformed"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngenerates a dictionary of search terms to what test case(s) they re related to", "response": "def _generate_search_terms(self):\n \"\"\"\n Map search terms to what test case(s) they're related to\n\n Returns:\n dict: maps search terms to what test case(s) it's relevant to\n\n Example:\n {\n '12034': ['ui.tests.TestSomething.test_hello_world'],\n 'buggy': ['ui.tests.TestSomething.test_hello_world', 'ui.tests.TestSomething.buggy_test_case'],\n 'ui.tests.TestAnother.test_fail': ['ui.tests.TestAnother.test_fail']\n }\n \"\"\"\n search_terms = {}\n\n for test_result in self.test_results:\n # search for the test name itself maps to the test case\n search_terms[test_result['name']] = test_result['name']\n\n if test_result['description']:\n for token in test_result['description'].split():\n if token in search_terms:\n search_terms[token].append(test_result['name'])\n else:\n search_terms[token] = [test_result['name']]\n\n return search_terms"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngenerates the HTML report for the current test set.", "response": "def afterSummaryReport(self, event):\n \"\"\"\n After everything is done, generate the report\n \"\"\"\n logger.info('Generating HTML report...')\n\n sorted_test_results = self._sort_test_results()\n\n context = {\n 'test_report_title': 'Test Report',\n 'test_summary': self.summary_stats,\n 'test_results': sorted_test_results,\n 'autocomplete_terms': json.dumps(self._generate_search_terms()),\n 'timestamp': datetime.utcnow().strftime('%Y/%m/%d %H:%M:%S UTC')\n }\n template = load_template(self._config['template'])\n rendered_template = render_template(template, context)\n with open(self._config['report_path'], 'w') as template_file:\n template_file.write(rendered_template)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef quarter_to_daterange(quarter):\n assert len(quarter) == 6\n year = int(quarter[0:4])\n quarter = quarter[5]\n MONTH_DAY = {\n '1': ((1, 1), (3, 31)),\n '2': ((4, 1), (6, 30)),\n '3': ((7, 1), (9, 30)),\n '4': ((10, 1), (12, 31))\n }\n md = MONTH_DAY[quarter]\n start_md, end_md = md\n return (\n date(year, *start_md),\n date(year, *end_md)\n )", "response": "Convert a quarter in arbitrary filename - ready format into start and end datetimes"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a tuple of the datetime s year and quarter", "response": "def datetime_to_year_quarter(dt):\n \"\"\"\n Args:\n dt: a datetime\n Returns:\n tuple of the datetime's year and quarter\n \"\"\"\n year = dt.year\n quarter = int(math.ceil(float(dt.month)/3))\n return (year, quarter)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef dates_in_range(start_date, end_date):\n return [\n start_date + timedelta(n)\n for n in range(int((end_date - start_date).days))\n ]", "response": "Returns all dates between two dates."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nloads the exposed hooks.", "response": "def load_hooks():\n \"\"\"\n Load the exposed hooks.\n\n Returns a dict of hooks where the keys are the name of the hook and the\n values are the actual hook functions.\n \"\"\"\n hooks = {}\n for entrypoint in pkg_resources.iter_entry_points(ENTRYPOINT):\n name = str(entrypoint).split('=')[0].strip()\n try:\n hook = entrypoint.load()\n except Exception as e:\n write_message('failed to load entry-point %r (error=\"%s\")' % (name, e), 'yellow')\n else:\n hooks[name] = hook\n return hooks"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nlets you use gettext instead of the Qt tools for l18n", "response": "def gettext(ui_file_path):\n \"\"\"\n Let you use gettext instead of the Qt tools for l18n\n \"\"\"\n with open(ui_file_path, 'r') as fin:\n content = fin.read()\n\n # replace ``_translate(\"context\", `` by ``_(``\n content = re.sub(r'_translate\\(\".*\",\\s', '_(', content)\n content = content.replace(\n ' _translate = QtCore.QCoreApplication.translate', '')\n\n with open(ui_file_path, 'w') as fout:\n fout.write(content)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nstream a JSON file in JSON - per - line format", "response": "def stream_json_file(local_file):\n \"\"\"Stream a JSON file (in JSON-per-line format)\n\n Args:\n local_file (file-like object) an open file-handle that contains a\n JSON string on each line\n Yields:\n (dict) JSON objects\n \"\"\"\n for i, line in enumerate(local_file):\n try:\n data = json.loads(line.decode('utf-8'))\n yield data\n except ValueError as e:\n logging.warning(\"Skipping line %d due to error: %s\", i, e)\n continue"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef download(self, force=False):\n exists, dest_path = self._downloader.cache.exists_packed(package=self, pkg_path=self.packed_path,\n check_stat=not self._in_cache)\n unp_exists, unp_path = self._downloader.cache.exists_unpacked(package=self, pkg_path=self.unpacked_path)\n\n # \u0415\u0441\u043b\u0438 \u0430\u0440\u0445\u0438\u0432\u0430 \u043d\u0435\u0442, \u0442\u043e \u0438 \u043a\u0435\u0448\u0443 \u0434\u043e\u0432\u0435\u0440\u044f\u0442\u044c \u043d\u0435 \u0441\u0442\u043e\u0438\u0442\n if not exists:\n unp_exists = False\n\n if exists and not self.packed_path:\n self.packed_path = dest_path\n\n if force or not exists:\n # _packed_path = self._packed_path\n dest_path_tmp = dest_path + \".tmp\"\n if os.path.exists(dest_path_tmp):\n os.remove(dest_path_tmp)\n self._adapter.download_package(self._pkg, dest_path_tmp)\n os.rename(dest_path_tmp, dest_path)\n self.packed_path = dest_path\n # if not _packed_path:\n self._not_cached = True\n else:\n if unp_exists and not self.unpacked_path:\n self.unpacked_path = unp_path\n self._not_cached = False\n\n if self._not_cached and unp_exists:\n shutil.rmtree(unp_path, ignore_errors=True)\n\n return self.packed_path", "response": "Download the package and return the path of the downloaded file."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfind all dependencies by package", "response": "def find_dependencies(self, depslock_file_path, property_validate=True, deps_content=None):\n \"\"\"\n Find all dependencies by package\n :param depslock_file_path:\n :param property_validate: for `root` packages we need check property, bad if we find packages from `lock` file,\n :param deps_content: HACK for use --dependencies-content and existed dependencies.txt.lock file\n we can skip validate part\n :return:\n \"\"\"\n self._raw = [x for x in\n self._downloader.common_parser.iter_packages_params(depslock_file_path, deps_content=deps_content)]\n self.packages = self._downloader.get_dependency_packages({'raw': self._raw},\n property_validate=property_validate)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfinds all dependencies by package", "response": "def find_usedby(self, depslock_file_path, property_validate=True):\n \"\"\"\n Find all dependencies by package\n :param depslock_file_path:\n :param property_validate: for `root` packages we need check property, bad if we find packages from `lock` file,\n we can skip validate part\n :return:\n \"\"\"\n if depslock_file_path is None:\n self._raw = [self._params]\n self._raw[0]['repo'] = None\n self._raw[0]['server'] = None\n else:\n self._raw = [x for x in self._downloader.common_parser.iter_packages_params(depslock_file_path)]\n self.packages = self._downloader.get_usedby_packages({'raw': self._raw},\n property_validate=property_validate)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget the parameters of the current object.", "response": "def get_params(self, param_list=None, get_path=False, merged=False, raw=False):\n \"\"\"\n Get Package params\n :param param_list: name or list of parameters\n :param get_path:\n :param merged: if version splited, True return version in string\n :param raw:\n :return:\n \"\"\"\n # Convert parameter name to list\n if param_list and isinstance(param_list, str):\n param_list = [param_list]\n if param_list and isinstance(param_list, (list, tuple)):\n result = {k: v for k, v in self._params_found.items() if k in param_list}\n result.update({k: v for k, v in self._params.items() if (k in param_list and k not in result)})\n else:\n result = {k: v for k, v in self._params_found.items()}\n result.update({k: v for k, v in self._params.items() if k not in result})\n if get_path:\n result['path'] = self.unpacked_path\n if merged:\n result.update(self._parser.merge_valued(result))\n if raw:\n result.update({k: v for k, v in self._params_found_raw.items()})\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget packages with None ( not founded ) recursively", "response": "def get_none_packages(self):\n \"\"\"\n Get packages with None (not founded), recursively\n \"\"\"\n not_found = set()\n for package_name, package in self.packages.items():\n if package is None:\n not_found.add(package_name)\n else:\n if package.packages:\n not_found = not_found | package.get_none_packages()\n return not_found"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets params with extra vars like any", "response": "def get_params_with_extra(self, rule_name, params):\n \"\"\"\n Get params with extra, like 'any'\n :param rule_name: 'path'\n :param params: default params\n :return: list of combination params\n \"\"\"\n # HACK for prefer-local\n result = []\n extra_params = self._rules_vars_extra.get(rule_name, {})[0]\n _tmp_params = copy.deepcopy(params)\n _fixed_params = {}\n\n # Save params with list type - this type not changed\n for key, value in _tmp_params.items():\n if isinstance(value, list):\n _fixed_params[key] = value\n _tmp_params = {k: v for k, v in _tmp_params.items() if k not in _fixed_params}\n\n # extend with extra_vars - like 'any'\n for key, value in _tmp_params.items():\n if not isinstance(value, list) and key:\n _tmp_params[key] = list([value])\n if key in extra_params:\n _tmp_params[key].extend(extra_params[key])\n\n # get combinations\n keys = sorted(_tmp_params)\n combinations = itertools.product(*(_tmp_params[x] for x in keys))\n for comb in combinations:\n _dict = dict(zip(keys, comb))\n _dict.update(_fixed_params)\n result.append(_dict)\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef split_fixed_pattern(path):\n _first_pattern_pos = path.find('*')\n _path_separator_pos = path.rfind('/', 0, _first_pattern_pos) + 1\n _path_fixed = path[:_path_separator_pos]\n _path_pattern = path[_path_separator_pos:]\n return _path_fixed, _path_pattern", "response": "Split path into fixed and masked parts\n "} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsplitting path into fixed parts and filename", "response": "def split_fixed_pattern_with_file_name(path):\n \"\"\"\n Split path into fixed, masked parts and filename\n :param path: e.g\nhttps://repo.example.com/artifactory/libs-cpp-release.snapshot/boost/1.60-pm/*.*.*/vc110/x86/win/boost.*.*.*.tar.gz\n :return:\n _path_fixed: https://repo.example.com/artifactory/libs-cpp-release.snapshot/boost/1.60-pm/\n _path_pattern: *.*.*/vc100/x86/win\n _file_name_pattern: boost.*.*.*.tar.gz\n \"\"\"\n _first_pattern_pos = path.find('*')\n _path_separator_pos = path.rfind('/', 0, _first_pattern_pos)\n _path_fixed = path[:_path_separator_pos]\n _path_pattern = path[_path_separator_pos + 1:]\n _file_name_pattern_separator_pos = _path_pattern.rfind('/', 0)\n _file_name_pattern = _path_pattern[_file_name_pattern_separator_pos + 1:]\n\n if _path_pattern.find('*') == -1 or _file_name_pattern_separator_pos == -1:\n _path_pattern = \"\"\n else:\n _path_pattern = _path_pattern[:_file_name_pattern_separator_pos]\n\n return _path_fixed, _path_pattern, _file_name_pattern"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn an Elasticsearch basic client that is responsive racket to the environment variable ELASTICSEARCH_ENDPOINT", "response": "def basic_client():\n \"\"\"Returns an Elasticsearch basic client that is responsive\n to the environment variable ELASTICSEARCH_ENDPOINT\"\"\"\n es_connected = False\n while not es_connected:\n try:\n ES = Elasticsearch(\n hosts=[HOSTNAME]\n )\n es_connected = True\n except TransportError as e:\n logging.info('Not yet connected: %s, sleeping for 1s', e)\n time.sleep(1)\n return ES"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_index(index_name, index_config, client):\n client.create(index=index_name, body=index_config)", "response": "Creates an index with a given configuration"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nretrieve the base index name from an alias", "response": "def get_index_from_alias(alias_name, index_client=None):\n \"\"\"Retrieve the base index name from an alias\n\n Args:\n alias_name (str) Name of the alias\n index_client (Elasticsearch.IndicesClient) an Elasticsearch index\n client. Optional, will create one if not given\n\n Returns: (str) Name of index\n \"\"\"\n index_client = index_client or indices_client()\n if not index_client.exists_alias(name=alias_name):\n return None\n return list(index_client.get_alias(name=alias_name).keys())[0]"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef atomic_swap(alias_name, new_index_name, index_client):\n logging.info('Performing atomic index alias swap')\n if index_client.exists_alias(name=alias_name):\n old_index_name = get_index_from_alias(alias_name, index_client)\n logging.info('Removing old as well as adding new')\n actions = {'actions': [\n {'remove': {'index': old_index_name, 'alias': alias_name}},\n {'add': {'index': new_index_name, 'alias': alias_name}}\n ]}\n index_client.update_aliases(body=actions)\n index_client.delete(index=old_index_name)\n else:\n logging.info('Old alias not found, only adding new')\n actions = {'actions': [\n {'add': {'index': new_index_name, 'alias': alias_name}}\n ]}\n index_client.update_aliases(body=actions)", "response": "This function performs an atomic index alias swap."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef zero_downtime_index(index_name, index_config):\n client = indices_client()\n temporary_name = index_name + '_' + str(uuid.uuid4())\n logging.info('creating index with config %s', index_config)\n create_index(temporary_name, index_config, client)\n try:\n yield temporary_name\n atomic_swap(index_name, temporary_name, client)\n except Exception:\n logging.error(\n 'deleting temporary index %s due to error:',\n temporary_name,\n exc_info=True\n )\n client.delete(index=temporary_name)", "response": "Context manager that creates a new index based on a given alias and then assigns it to the alias."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef replace(self):\n with zero_downtime_index(self.alias_name, self.index_config()) as target_index:\n self.index_all(target_index)", "response": "Replace index with a new one."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef append(self):\n target_index = get_index_from_alias(self.alias_name)\n if not target_index:\n self.replace()\n else:\n self.index_all(target_index)", "response": "Index documents onto an existing index."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nindex all available documents using streaming_bulk for speed", "response": "def index_all(self, index_name):\n \"\"\"Index all available documents, using streaming_bulk for speed\n Args:\n\n index_name (string): The index\n \"\"\"\n oks = 0\n notoks = 0\n for ok, item in streaming_bulk(\n self.es_client,\n self._iter_documents(index_name)\n ):\n if ok:\n oks += 1\n else:\n notoks += 1\n logging.info(\n \"Import results: %d ok, %d not ok\",\n oks,\n notoks\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef build_args(cmd, src, dst):\n cmd = cmd % (quote(src), quote(dst))\n args = shlex.split(cmd)\n\n return [arg for arg in args if arg]", "response": "Build the arguments list for passing to subprocess. call_check\nTaxonomy"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nregistering a function as an output format.", "response": "def register_output_format(name):\n \"\"\"\n Load output format function to dictionary (decorator with this function name)\n \"\"\"\n\n def check_decorator(fn):\n _output_format_map[name] = fn\n\n def wrapper(*args, **kwargs):\n return fn(*args, **kwargs)\n\n return wrapper\n\n return check_decorator"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef write_output(self, params, packages):\n if params['out_format'] not in _output_format_map:\n raise CrosspmException(\n CROSSPM_ERRORCODE_UNKNOWN_OUT_TYPE,\n 'Unknown out_format: [{}]'.format(params['out_format']),\n )\n\n f = _output_format_map[params['out_format']]\n result = f(self, packages, **params)\n\n if result:\n out_file_path = os.path.realpath(os.path.expanduser(params['output']))\n self.write_to_file(result, out_file_path)\n self._log.info(\n 'Write packages info to file [%s]\\ncontent:\\n\\n%s',\n out_file_path,\n result,\n )", "response": "Write the output of a set of packages to a file."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef output_format_module(self, packages, esc_path=False):\n\n def create_ordered_list(packages_):\n \"\"\"\n Recursive for package.packages\n \"\"\"\n list_ = []\n for _pkg_name in packages_:\n _pkg = packages_[_pkg_name]\n if _pkg and _pkg.packages:\n list_.extend(create_ordered_list(_pkg.packages))\n if _pkg:\n _pkg_params = _pkg.get_params(self._columns, True)\n _res_item = {}\n for item in self._output_config['columns']:\n name = item['name'].format(OutFormat(item['column']))\n value = _pkg_params.get(item['column'], '')\n if not isinstance(value, (list, dict, tuple)):\n try:\n value = item['value'].format(\n OutFormat(value, (item['column'] == 'path') if esc_path else False))\n except Exception:\n value = ''\n # TODO: implement this:\n # if not value:\n # try:\n # value = item['value'].format(OutFormat(_pkg.get_params('', True)))\n # except Exception as e:\n # pass\n _res_item[name] = value\n list_.append(_res_item)\n return list_\n\n result_list = create_ordered_list(packages, )\n\n if self._output_config['type'] == LIST:\n return result_list\n\n result = OrderedDict()\n for item in result_list:\n # TODO: Error handling\n name = item[self._output_config['key']]\n if self._output_config['value']:\n value = item[self._output_config['value']]\n else:\n value = OrderedDict([(k, v) for k, v in item.items() if k != self._output_config['key']])\n\n result[name] = value\n\n return result", "response": "Formats the module and returns the output as a dict."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef smart(**kwargs):\n\n def decorator(func):\n for key, value in kwargs.items():\n setattr(func, key, value)\n return func\n\n return decorator", "response": "Simple decorator to get custom fields on admin class"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_buffer(params, value):\n '''\n If the value is a float, create a numpy array of the required length, filled with value\n If the value is a numpy array, check its length\n Otherwise throw a type error\n '''\n try:\n fv = float(value)\n return np.full(params.length, fv, np.float)\n except TypeError:\n if isinstance(value, np.ndarray):\n if (len(value)>=params.length):\n return value\n raise TypeError('Value must be a float or a numpy array ofthe required length')", "response": "Create a numpy array of the required length filled with value\n If the value is a float create a numpy array of the required length filled with value\n Otherwise throw a type error\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef set_time(self, time):\n '''\n Update the length of the buffer in seconds\n :param time: number of seconds\n :return: self\n '''\n self.length = int(time*self.sample_rate)\n return self", "response": "Update the length of the buffer in seconds\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef make_request(self, method, service, path, body=None,\n query_params: QueryParams=None,\n headers: dict=None,\n correlation_id: str=None,\n content_type: str='application/json',\n context: Request=None,\n timeout=30,\n **kwargs) -> asyncio.coroutine:\n \"\"\"\n Make a request to another service. If `context` is provided, then\n context and correlation will be pulled from the provided request\n object for you. This includes credentials, correlationid,\n and service-headers.\n\n :param method: GET/PUT/PATCH, etc.\n :param service: name of service\n :param path: request object path\n :param body: body of request\n :param query_params:\n :param headers:\n :param correlation_id:\n :param content_type:\n :param context: A request object from which a \"child-request\"\n will be made\n :param timeout: Time in seconds the client will wait befor raising\n an asyncio.TimeoutError\n :param kwargs: Just a place holder so transport specific options\n can be passed through\n :return:\n \"\"\"\n if not isinstance(method, Methods):\n method = Methods(method.upper())\n if content_type == 'application/json' and isinstance(body, dict):\n body = json.dumps(body)\n if isinstance(query_params, dict):\n query_string = parse.urlencode(query_params)\n elif isinstance(query_params, QueryParams):\n query_string = str(query_params)\n else:\n query_string = ''\n headers = headers or {}\n\n ctx = request_context.get()\n if context:\n warnings.warn(\"Passing in a context to waspy client is deprecated. \"\n \"Passed in context will be ignored\", DeprecationWarning)\n\n if not correlation_id:\n correlation_id = ctx['correlation_id']\n\n headers = {**headers, **ctx['ctx_headers']}\n\n exchange = headers.get('ctx-exchange-override', None)\n if exchange:\n kwargs['exchange'] = 'amq.headers'\n\n if isinstance(body, str):\n body = body.encode()\n response = asyncio.wait_for(\n self.transport.make_request(\n service, method.name, path, body=body, query=query_string,\n headers=headers, correlation_id=correlation_id,\n content_type=content_type, timeout=timeout, **kwargs),\n timeout=timeout)\n return response", "response": "Make a request to another service."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nmake a GET request.", "response": "def get(self, service, path, **kwargs):\n \"\"\" Make a get request (this returns a coroutine)\"\"\"\n return self.make_request(Methods.GET, service, path, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nmakes a PUT request.", "response": "def put(self, service, path, body, **kwargs):\n \"\"\" Make a put request (this returns a coroutine)\"\"\"\n return self.make_request(Methods.POST, service, path, body=body,\n **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef patch(self, service, path, body, **kwargs):\n return self.make_request(Methods.PATCH, service, path, body=body,\n **kwargs)", "response": "Make a PATCH request"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmakes a DELETE request", "response": "def delete(self, service, path, **kwargs):\n \"\"\" Make a delete requests (this returns a coroutine)\"\"\"\n return self.make_request(Methods.DELETE, service, path, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef process_request(self, request_object):\n\n identifier = request_object.identifier\n\n # Look for the object by its ID and return it\n resource = request_object.entity_cls.get(identifier)\n return ResponseSuccess(Status.SUCCESS, resource)", "response": "Fetch Resource and return Entity"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef from_dict(cls, entity_cls, adict):\n invalid_req = InvalidRequestObject()\n\n # Extract the pagination parameters from the input\n page = int(adict.pop('page', 1))\n per_page = int(adict.pop(\n 'per_page', getattr(active_config, 'PER_PAGE', 10)))\n order_by = adict.pop('order_by', ())\n\n # Check for invalid request conditions\n if page < 0:\n invalid_req.add_error('page', 'is invalid')\n\n if invalid_req.has_errors:\n return invalid_req\n\n # Do we need to pop out random?\n # adict.pop('random', None)\n\n return cls(entity_cls, page, per_page, order_by, adict)", "response": "Initialize a ListRequestObject object from a dictionary."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef process_request(self, request_object):\n resources = (request_object.entity_cls.query\n .filter(**request_object.filters)\n .offset((request_object.page - 1) * request_object.per_page)\n .limit(request_object.per_page)\n .order_by(request_object.order_by)\n .all())\n return ResponseSuccess(Status.SUCCESS, resources)", "response": "Return a list of resources"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nprocess Create Resource Request", "response": "def process_request(self, request_object):\n \"\"\"Process Create Resource Request\"\"\"\n\n resource = request_object.entity_cls.create(**request_object.data)\n return ResponseSuccessCreated(resource)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nprocesses an update resource request", "response": "def process_request(self, request_object):\n \"\"\"Process Update Resource Request\"\"\"\n\n # Retrieve the object by its identifier\n entity = request_object.entity_cls.get(request_object.identifier)\n\n # Update the object and return the updated data\n resource = entity.update(request_object.data)\n return ResponseSuccess(Status.SUCCESS, resource)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nprocessing the Delete Resource Request", "response": "def process_request(self, request_object):\n \"\"\"Process the Delete Resource Request\"\"\"\n\n # Delete the object by its identifier\n entity = request_object.entity_cls.get(request_object.identifier)\n entity.delete()\n\n # FIXME Check for return value of `delete()`\n\n # We have successfully deleted the object.\n # Sending a 204 Response code.\n return ResponseSuccessWithNoContent()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _cast_to_type(self, value):\n if isinstance(value, str) or value is None:\n return value\n return str(value)", "response": "Convert the value to its string representation"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts the value to an int and raise error on failures", "response": "def _cast_to_type(self, value):\n \"\"\" Convert the value to an int and raise error on failures\"\"\"\n try:\n return int(value)\n except (ValueError, TypeError):\n self.fail('invalid', value=value)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts the value to a float and raise error on failures", "response": "def _cast_to_type(self, value):\n \"\"\" Convert the value to a float and raise error on failures\"\"\"\n try:\n return float(value)\n except (ValueError, TypeError):\n self.fail('invalid', value=value)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting the value to a boolean and raise error on failures", "response": "def _cast_to_type(self, value):\n \"\"\" Convert the value to a boolean and raise error on failures\"\"\"\n if value in (True, False):\n return bool(value)\n if value in ('t', 'True', '1'):\n return True\n if value in ('f', 'False', '0'):\n return False\n self.fail('invalid', value=value)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncasts the value to the type of the object. Raise error if the value is not a list.", "response": "def _cast_to_type(self, value):\n \"\"\" Raise error if the value is not a list \"\"\"\n if not isinstance(value, list):\n self.fail('invalid', value=value)\n return value"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _cast_to_type(self, value):\n if not isinstance(value, dict):\n self.fail('invalid', value=value)\n return value", "response": "Cast the value to the type of the object. Raise error if the value is not a dict."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts the value to a date and raise error on failures", "response": "def _cast_to_type(self, value):\n \"\"\" Convert the value to a date and raise error on failures\"\"\"\n if isinstance(value, datetime.datetime):\n return value.date()\n if isinstance(value, datetime.date):\n return value\n try:\n value = date_parser(value)\n return value.date()\n except ValueError:\n self.fail('invalid', value=value)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _cast_to_type(self, value):\n if isinstance(value, datetime.datetime):\n return value\n if isinstance(value, datetime.date):\n value = datetime.datetime(value.year, value.month, value.day)\n return value\n try:\n value = date_parser(value)\n return value\n except ValueError:\n self.fail('invalid', value=value)", "response": "Convert the value to a datetime and raise error on failures"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nmakes a request to the specified service.", "response": "async def make_request(self, service: str, method: str, path: str,\n body: bytes=None, query: str=None,\n headers: dict=None, correlation_id: str=None,\n content_type: str=None,\n timeout:int = 30,\n **kwargs) -> webtypes.Response:\n \"\"\"\n Method for actually making a request\n :param service: service to make request too\n :param method: HTTP method: GET/PUT/POST etc.\n :param path: routing path.\n Should support dots `foo.2.bars` or slashes `foo/2/bars`\n :param body: request body. Bytes-like object\n :param query: query string. Example: `foo=bar&cabbage=green`\n :param headers: Dictionary of headers\n :param correlation_id:\n :param content_type: example: `application/json`\n :param timeout: time to wait for response in seconds before getting\n an asyncio.TimeoutError\n :param kwargs: Should except **kwargs for compatability for\n other possible options on other transports\n (for example, http might need a `port` option)\n :return:\n \"\"\""} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef execute(self, request_object):\n\n # If the request object is not valid then return a failure response\n if not request_object.is_valid:\n return ResponseFailure.build_from_invalid_request(\n request_object)\n\n # Try to process the request and handle any errors encountered\n try:\n return self.process_request(request_object)\n\n except ValidationError as err:\n return ResponseFailure.build_unprocessable_error(err.normalized_messages)\n\n except ObjectNotFoundError:\n return ResponseFailure.build_not_found(\n [{'identifier': 'Object with this ID does not exist.'}])\n\n except Exception as exc:\n logger.error(\n f'{self.__class__.__name__} execution failed due to error {exc}',\n exc_info=True)\n return ResponseFailure.build_system_error([{exc.__class__.__name__: exc}])", "response": "Generic executor method of all UseCases"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef FunctionTimer(on_done=None):\n '''\n To check execution time of a function\n borrowed from https://medium.com/pythonhive/python-decorator-to-measure-the-execution-time-of-methods-fa04cb6bb36d\n\n >>> def logger(details, args, kwargs): #some function that uses the time output\n ... print(details)\n ...\n >>> @FunctionTimer(on_done= logger)\n ... def foo(t=10):\n ... print('foo executing...')\n ... time.sleep(t)\n ...\n >>> @FunctionTimer(on_done= logger)\n ... def bar(t, n):\n ... for i in range(n):\n ... print('bar executing...')\n ... time.sleep(1)\n ... foo(t)\n ...\n >>> bar(3,2)\n bar executing...\n bar executing...\n foo executing...\n ('foo', 3)\n ('bar', 5)\n '''\n def decfn(fn):\n\n def timed(*args, **kwargs):\n ts = time.time()\n result = fn(*args, **kwargs)\n te = time.time()\n if on_done:\n on_done((fn.__name__,int(te - ts)), args, kwargs)\n else:\n print(('%r %d sec(s)' % (fn.__name__, (te - ts))))\n\n return result\n\n return timed\n\n return decfn", "response": "Decorator to measure execution time of a function in a sequence."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\nasync def exchange_declare(self):\n await self.channel.exchange_declare(\n self.exchange,\n self.exchange_type,\n durable=self.durable,\n auto_delete=self.auto_delete,\n no_wait=self.no_wait,\n )", "response": "Override this method to change how an exchange is declared"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\nasync def queue_declare(self):\n await self.channel.queue_declare(\n self.queue,\n durable=self.durable,\n exclusive=self.exclusive,\n no_wait=self.no_wait\n )", "response": "Override this method to change how a queue is declared"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts the given entity to a dictionary record", "response": "def from_entity(cls, entity: Entity) -> 'DictModel':\n \"\"\"Convert the entity to a dictionary record \"\"\"\n dict_obj = {}\n for field_name in entity.meta_.attributes:\n dict_obj[field_name] = getattr(entity, field_name)\n return dict_obj"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_connection(self):\n database = {\n 'data': _databases.setdefault(self.identifier, defaultdict(dict)),\n 'lock': _locks.setdefault(self.identifier, Lock()),\n 'counters': _counters\n }\n return database", "response": "Return the dictionary database object"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a repository object configured with a live connection", "response": "def get_repository(self, entity_cls):\n \"\"\"Return a repository object configured with a live connection\"\"\"\n model_cls = self.get_model(entity_cls)\n return DictRepository(self, entity_cls, model_cls)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _evaluate_lookup(self, key, value, negated, db):\n results = {}\n for record_key, record_value in db.items():\n match = True\n\n stripped_key, lookup_class = self._extract_lookup(key)\n lookup = lookup_class(record_value[stripped_key], value)\n\n if negated:\n match &= not eval(lookup.as_expression())\n else:\n match &= eval(lookup.as_expression())\n\n if match:\n results[record_key] = record_value\n\n return results", "response": "Evaluate the lookup criteria for the given key and value in the given database."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nrun raw queries on the database and return all results.", "response": "def raw(self, query: Any, data: Any = None):\n \"\"\"Run raw queries on the database\n\n As an example of running ``raw`` queries on a Dict repository, we will run the query\n on all possible schemas, and return all results.\n \"\"\"\n assert isinstance(query, str)\n\n conn = self.get_connection()\n items = []\n\n for schema_name in conn['data']:\n input_db = conn['data'][schema_name]\n try:\n # Ensures that the string contains double quotes around keys and values\n query = query.replace(\"'\", \"\\\"\")\n criteria = json.loads(query)\n\n for key, value in criteria.items():\n input_db = self._evaluate_lookup(key, value, False, input_db)\n\n items.extend(list(input_db.values()))\n\n except json.JSONDecodeError:\n # FIXME Log Exception\n raise Exception(\"Query Malformed\")\n except KeyError:\n # We encountered a repository where the key was not found\n pass\n\n return items"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsets the values of the auto fields using counter", "response": "def _set_auto_fields(self, model_obj):\n \"\"\"Set the values of the auto field using counter\"\"\"\n for field_name, field_obj in \\\n self.entity_cls.meta_.auto_fields:\n counter_key = f'{self.schema_name}_{field_name}'\n if not (field_name in model_obj and model_obj[field_name] is not None):\n # Increment the counter and it should start from 1\n counter = next(self.conn['counters'][counter_key])\n if not counter:\n counter = next(self.conn['counters'][counter_key])\n\n model_obj[field_name] = counter\n return model_obj"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nwrite a record to the dict repository", "response": "def create(self, model_obj):\n \"\"\"Write a record to the dict repository\"\"\"\n # Update the value of the counters\n model_obj = self._set_auto_fields(model_obj)\n\n # Add the entity to the repository\n identifier = model_obj[self.entity_cls.meta_.id_field.field_name]\n with self.conn['lock']:\n self.conn['data'][self.schema_name][identifier] = model_obj\n\n return model_obj"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _filter(self, criteria: Q, db):\n # Filter the dictionary objects based on the filters\n negated = criteria.negated\n input_db = None\n\n if criteria.connector == criteria.AND:\n # Trim database records over successive iterations\n # Whatever is left at the end satisfy all criteria (AND)\n input_db = db\n for child in criteria.children:\n if isinstance(child, Q):\n input_db = self._filter(child, input_db)\n else:\n input_db = self.provider._evaluate_lookup(child[0], child[1],\n negated, input_db)\n else:\n # Grow database records over successive iterations\n # Whatever is left at the end satisfy any criteria (OR)\n input_db = {}\n for child in criteria.children:\n if isinstance(child, Q):\n results = self._filter(child, db)\n else:\n results = self.provider._evaluate_lookup(child[0], child[1], negated, db)\n\n input_db = {**input_db, **results}\n\n return input_db", "response": "Recursive function to filter items from dictionary based on the filters\n "} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef filter(self, criteria: Q, offset: int = 0, limit: int = 10, order_by: list = ()):\n\n if criteria.children:\n items = list(self._filter(criteria, self.conn['data'][self.schema_name]).values())\n else:\n items = list(self.conn['data'][self.schema_name].values())\n\n # Sort the filtered results based on the order_by clause\n for o_key in order_by:\n reverse = False\n if o_key.startswith('-'):\n reverse = True\n o_key = o_key[1:]\n items = sorted(items, key=itemgetter(o_key), reverse=reverse)\n\n result = ResultSet(\n offset=offset,\n limit=limit,\n total=len(items),\n items=items[offset: offset + limit])\n return result", "response": "Read the repository and return results as per the filer"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nupdate the entity record in the dictionary", "response": "def update(self, model_obj):\n \"\"\"Update the entity record in the dictionary \"\"\"\n identifier = model_obj[self.entity_cls.meta_.id_field.field_name]\n with self.conn['lock']:\n # Check if object is present\n if identifier not in self.conn['data'][self.schema_name]:\n raise ObjectNotFoundError(\n f'`{self.__class__.__name__}` object with identifier {identifier} '\n f'does not exist.')\n\n self.conn['data'][self.schema_name][identifier] = model_obj\n return model_obj"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update_all(self, criteria: Q, *args, **kwargs):\n items = self._filter(criteria, self.conn['data'][self.schema_name])\n\n update_count = 0\n for key in items:\n item = items[key]\n item.update(*args)\n item.update(kwargs)\n self.conn['data'][self.schema_name][key] = item\n\n update_count += 1\n\n return update_count", "response": "Update all objects satisfying the criteria"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef delete_all(self, criteria: Q = None):\n if criteria:\n # Delete the object from the dictionary and return the deletion count\n items = self._filter(criteria, self.conn['data'][self.schema_name])\n\n # Delete all the matching identifiers\n with self.conn['lock']:\n for identifier in items:\n self.conn['data'][self.schema_name].pop(identifier, None)\n\n return len(items)\n else:\n with self.conn['lock']:\n if self.schema_name in self.conn['data']:\n del self.conn['data'][self.schema_name]", "response": "Delete all the items in the dictionary by its criteria"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrunning raw query on Repository.", "response": "def raw(self, query: Any, data: Any = None):\n \"\"\"Run raw query on Repository.\n\n For this stand-in repository, the query string is a json string that contains kwargs\n criteria with straigh-forward equality checks. Individual criteria are always ANDed\n and the result is always a subset of the full repository.\n\n We will ignore the `data` parameter for this kind of repository.\n \"\"\"\n # Ensure that we are dealing with a string, for this repository\n assert isinstance(query, str)\n\n input_db = self.conn['data'][self.schema_name]\n result = None\n\n try:\n # Ensures that the string contains double quotes around keys and values\n query = query.replace(\"'\", \"\\\"\")\n criteria = json.loads(query)\n\n for key, value in criteria.items():\n input_db = self.provider._evaluate_lookup(key, value, False, input_db)\n\n items = list(input_db.values())\n result = ResultSet(\n offset=1,\n limit=len(items),\n total=len(items),\n items=items)\n\n except json.JSONDecodeError:\n # FIXME Log Exception\n raise Exception(\"Query Malformed\")\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef process_source(self):\n if isinstance(self.source, str):\n # Replace single and double quotes with escaped single-quote\n self.source = self.source.replace(\"'\", \"\\'\").replace('\"', \"\\'\")\n return \"\\\"{source}\\\"\".format(source=self.source)\n return self.source", "response": "Return source with transformations if any"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn target with transformations if any", "response": "def process_target(self):\n \"\"\"Return target with transformations, if any\"\"\"\n if isinstance(self.target, str):\n # Replace single and double quotes with escaped single-quote\n self.target = self.target.replace(\"'\", \"\\'\").replace('\"', \"\\'\")\n return \"\\\"{target}\\\"\".format(target=self.target)\n return self.target"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a response dict like Trial because Status can t be pickled.", "response": "def challenge(arg: ChallengeArg) -> dict:\n \"\"\" Response is dict like `Trial` because Status(OwlEnum) can't be pickled.\n \"\"\"\n\n name: str = arg.req.name.get_or(str(arg.seq))\n log_prefix = f\"[{arg.seq} / {arg.number_of_request}]\"\n\n logger.info_lv3(f\"{log_prefix} {'-'*80}\")\n logger.info_lv3(f\"{log_prefix} {arg.seq}. {arg.req.name.get_or(arg.req.path)}\")\n logger.info_lv3(f\"{log_prefix} {'-'*80}\")\n\n path_str_one = arg.path_one.map(lambda x: re.sub(x.before, x.after, arg.req.path)).get_or(arg.req.path)\n path_str_other = arg.path_other.map(lambda x: re.sub(x.before, x.after, arg.req.path)).get_or(arg.req.path)\n qs_str_one = create_query_string(arg.req.qs, arg.query_one, arg.req.url_encoding)\n qs_str_other = create_query_string(arg.req.qs, arg.query_other, arg.req.url_encoding)\n url_one = f'{arg.host_one}{path_str_one}?{qs_str_one}'\n url_other = f'{arg.host_other}{path_str_other}?{qs_str_other}'\n\n # Get two responses\n req_time = now()\n try:\n logger.info_lv3(f\"{log_prefix} One URL: {url_one}\")\n logger.debug(f\"{log_prefix} One PROXY: {arg.proxy_one.map(lambda x: x.to_dict()).get()}\")\n logger.info_lv3(f\"{log_prefix} Other URL: {url_other}\")\n logger.debug(f\"{log_prefix} Other PROXY: {arg.proxy_other.map(lambda x: x.to_dict()).get()}\")\n r_one, r_other = concurrent_request(arg.session, arg.req.headers,\n url_one, url_other,\n arg.proxy_one, arg.proxy_other)\n logger.info_lv3(\n f\"{log_prefix} One: {r_one.status_code} / {to_sec(r_one.elapsed)}s / {len(r_one.content)}b / {r_one.headers.get('content-type')}\" # noqa\n )\n logger.info_lv3(\n f\"{log_prefix} Other: {r_other.status_code} / {to_sec(r_other.elapsed)}s / {len(r_other.content)}b / {r_other.headers.get('content-type')}\" # noqa\n )\n except ConnectionError:\n logger.info_lv1(f\"{log_prefix} \ud83d\udc80 {arg.req.name.get()}\")\n # TODO: Integrate logic into create_trial\n return {\n \"seq\": arg.seq,\n \"name\": name,\n \"tags\": [],\n \"request_time\": req_time.isoformat(),\n \"status\": 'failure',\n \"path\": arg.req.path,\n \"queries\": arg.req.qs,\n \"headers\": arg.req.headers,\n \"one\": {\n \"url\": url_one,\n \"type\": \"unknown\",\n },\n \"other\": {\n \"url\": url_other,\n \"type\": \"unknown\",\n }\n }\n\n res_one: Response = res2res(Response.from_requests(r_one, arg.default_response_encoding_one), arg.req)\n res_other: Response = res2res(Response.from_requests(r_other, arg.default_response_encoding_other), arg.req)\n\n dict_one: TOption[dict] = res2dict(res_one)\n dict_other: TOption[dict] = res2dict(res_other)\n\n # Create diff\n # Either dict_one or dic_other is None, it means that it can't be analyzed, therefore return None\n ddiff = None if dict_one.is_none() or dict_other.is_none() \\\n else {} if res_one.body == res_other.body \\\n else DeepDiff(dict_one.get(), dict_other.get())\n\n initial_diffs_by_cognition: Optional[TDict[DiffKeys]] = TDict({\n \"unknown\": DiffKeys.from_dict({\n \"changed\": TList(ddiff.get('type_changes', {}).keys() | ddiff.get('values_changed', {}).keys())\n .map(to_jumeaux_xpath)\n .order_by(_),\n \"added\": TList(ddiff.get('dictionary_item_added', {}) | ddiff.get('iterable_item_added', {}).keys())\n .map(to_jumeaux_xpath)\n .order_by(_),\n \"removed\": TList(ddiff.get('dictionary_item_removed', {}) | ddiff.get('iterable_item_removed', {}).keys())\n .map(to_jumeaux_xpath)\n .order_by(_)\n })\n }) if ddiff is not None else None\n\n # Judgement\n status, diffs_by_cognition = judgement(res_one, res_other, dict_one, dict_other,\n name, arg.req.path, arg.req.qs, arg.req.headers, initial_diffs_by_cognition)\n status_symbol = \"O\" if status == Status.SAME else \"X\"\n log_msg = f\"{log_prefix} {status_symbol} ({res_one.status_code} - {res_other.status_code}) <{res_one.elapsed_sec}s - {res_other.elapsed_sec}s> {arg.req.name.get_or(arg.req.path)}\" # noqa\n (logger.info_lv2 if status == Status.SAME else logger.info_lv1)(log_msg)\n\n file_one: Optional[str] = None\n file_other: Optional[str] = None\n prop_file_one: Optional[str] = None\n prop_file_other: Optional[str] = None\n if store_criterion(status, name, arg.req, res_one, res_other):\n dir = f'{arg.res_dir}/{arg.key}'\n file_one = f'one/({arg.seq}){name}'\n file_other = f'other/({arg.seq}){name}'\n write_to_file(file_one, dir, dump(res_one))\n write_to_file(file_other, dir, dump(res_other))\n if not dict_one.is_none():\n prop_file_one = f'one-props/({arg.seq}){name}.json'\n write_to_file(prop_file_one, dir, TDict(dict_one.get()).to_json().encode('utf-8', errors='replace'))\n if not dict_other.is_none():\n prop_file_other = f'other-props/({arg.seq}){name}.json'\n write_to_file(prop_file_other, dir, TDict(dict_other.get()).to_json().encode('utf-8', errors='replace'))\n\n return global_addon_executor.apply_did_challenge(\n DidChallengeAddOnPayload.from_dict({\n \"trial\": Trial.from_dict({\n \"seq\": arg.seq,\n \"name\": name,\n \"tags\": [], # TODO: tags created by reqs2reqs\n \"request_time\": req_time.isoformat(),\n \"status\": status,\n \"path\": arg.req.path,\n \"queries\": arg.req.qs,\n \"headers\": arg.req.headers,\n \"diffs_by_cognition\": diffs_by_cognition,\n \"one\": {\n \"url\": res_one.url,\n \"type\": res_one.type,\n \"status_code\": res_one.status_code,\n \"byte\": res_one.byte,\n \"response_sec\": res_one.elapsed_sec,\n \"content_type\": res_one.content_type,\n \"mime_type\": res_one.mime_type,\n \"encoding\": res_one.encoding,\n \"file\": file_one,\n \"prop_file\": prop_file_one,\n },\n \"other\": {\n \"url\": res_other.url,\n \"type\": res_other.type,\n \"status_code\": res_other.status_code,\n \"byte\": res_other.byte,\n \"response_sec\": res_other.elapsed_sec,\n \"content_type\": res_other.content_type,\n \"mime_type\": res_other.mime_type,\n \"encoding\": res_other.encoding,\n \"file\": file_other,\n \"prop_file\": prop_file_other,\n }\n })\n }),\n DidChallengeAddOnReference.from_dict({\n \"res_one\": res_one,\n \"res_other\": res_other,\n \"res_one_props\": dict_one,\n \"res_other_props\": dict_other,\n })\n ).trial.to_dict()"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nparses a URL to a topic.", "response": "def parse_url_to_topic(method, route):\n \"\"\"\n Transforms a URL to a topic.\n\n `GET /bar/{id}` -> `get.bar.*`\n `POST /bar/{id}` -> `post.bar.*`\n `GET /foo/bar/{id}/baz` -? `get.foo.bar.*.baz`\n\n Possible gotchas\n\n `GET /foo/{id}` -> `get.foo.*`\n `GET /foo/{id}:action` -> `get.foo.*`\n\n However, once it hits the service the router will be able to distinguish the two requests.\n \"\"\"\n route = route.replace('.', '?')\n route = route.replace('/', '.').strip('.')\n topic = f'{method.value.lower()}.{route}'\n # need to replace `{id}` and `{id}:some_method` with just `*`\n return re.sub(r\"\\.\\{[^\\}]*\\}[:\\w\\d_-]*\", \".*\", topic)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nuses for example of Transformer. function", "response": "def wrap(anything: bytes, encoding: str) -> str:\n \"\"\"Use for example of Transformer.function\n \"\"\"\n return json.dumps({\n \"wrap\": load_json(anything.decode(encoding))\n }, ensure_ascii=False)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a ResultSet containing only the records matching the given criteria.", "response": "def filter(self, criteria: Q, offset: int = 0, limit: int = 10,\n order_by: list = ()) -> ResultSet:\n \"\"\"\n Filter objects from the repository. Method must return a `ResultSet`\n object\n \"\"\""} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _format_fields(cls, declared_fields: typing.List[tuple]):\n formatted_fields = []\n for declared_field in declared_fields:\n field_name = field_type = field_defn = None\n\n # Case when only (name), or \"name\", is specified\n if isinstance(declared_field, str) or len(declared_field) == 1:\n field_name = declared_field\n field_type = typing.Any\n field_defn = field(default=None)\n\n # Case when (name, type) are specified\n elif len(declared_field) == 2:\n field_name = declared_field[0]\n field_type = declared_field[1]\n field_defn = field(default=None)\n\n # Case when (name, type, field) are specified\n elif len(declared_field) == 3:\n field_name = declared_field[0]\n field_type = declared_field[1]\n\n # Process the definition and create a `field` object\n # Definition will be of the form `{'required': False, 'default': 'John'}`\n assert isinstance(declared_field[2], dict)\n metadata = default = None\n if 'required' in declared_field[2] and declared_field[2]['required']:\n metadata = {'required': True}\n if 'default' in declared_field[2]:\n default = declared_field[2]['default']\n field_defn = field(default=default, metadata=metadata)\n\n formatted_fields.append((field_name, field_type, field_defn))\n\n return formatted_fields", "response": "Process the declared fields and construct a list of tuples that can be fed into dataclass constructor factory factory."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsave a sequence of samples as a WAV file", "response": "def save(params, filename, source):\n '''\n Write a sequence of samples as a WAV file\n Currently a 16 bit mono file\n '''\n writer = wave.open(filename, 'wb');\n # Set the WAV file parameters, currently default values\n writer.setnchannels(1)\n writer.setsampwidth(2)\n writer.setframerate(params.sample_rate)\n data_out = array.array('h')\n for x in source:\n data_out.append(int(x * 32766))\n writer.writeframes(data_out.tostring())\n writer.close()"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the item with the lowest priority. Raises IndexError if the object is empty.", "response": "def smallest(self):\n \"\"\"Return the item with the lowest priority.\n\n Raises IndexError if the object is empty.\n \"\"\"\n\n heap = self._heap\n v, k = heap[0]\n while k not in self or self[k] != v:\n heappop(heap)\n v, k = heap[0]\n\n return k"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef pop_smallest(self):\n\n heap = self._heap\n v, k = heappop(heap)\n while k not in self or self[k] != v:\n v, k = heappop(heap)\n del self[k]\n return k", "response": "Return the item with the lowest priority and remove it. Raises IndexError if the object is empty. Returns the item with the lowest priority and remove it. Raises IndexError if the object is empty."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndecodes the original body.", "response": "def body(self) -> dict:\n \"\"\" Decoded Body \"\"\"\n if self.ignore_content_type:\n return self.original_body\n if self._body is None and self.original_body is not None:\n if isinstance(self.original_body, bytes):\n self._body = self.parser.decode(self.original_body)\n else:\n self._body = self.original_body\n return self._body"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the raw HTTP request body.", "response": "def raw_body(self) -> bytes:\n \"\"\" Encoded Body \"\"\"\n if self._raw_body is None and self.original_body is not None:\n if isinstance(self.original_body, dict):\n self._raw_body = self.parser.encode(self.original_body)\n if isinstance(self._raw_body, str):\n self._raw_body = self._raw_body.encode()\n elif isinstance(self.original_body, str):\n self._raw_body = self.original_body.encode()\n elif isinstance(self.original_body, bytes):\n self._raw_body = self.original_body\n else:\n self._raw_body = self.parser.encode(self.original_body)\n if isinstance(self._raw_body, str):\n self._raw_body = self._raw_body.encode()\n return self._raw_body"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef send_messages(self, messages):\n msg_count = 0\n for message in messages: # .message() triggers header validation\n message.message()\n email.outbox.append(message)\n msg_count += 1\n\n return msg_count", "response": "Redirect messages to the dummy outbox"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngenerating a square wave of the given frequency amplitude and offset ratio", "response": "def square_wave(params, frequency=400, amplitude=1,\n offset=0, ratio=0.5):\n '''\n Generate a square wave\n :param params: buffer parameters, controls length of signal created\n :param frequency: wave frequency (array or value)\n :param amplitude: wave amplitude (array or value)\n :param offset: offset of wave mean from zero (array or value)\n :param ratio: ratio of high to low time (array or value)\n :return: array of resulting signal\n '''\n frequency = create_buffer(params, frequency)\n amplitude = create_buffer(params, amplitude)\n offset = create_buffer(params, offset)\n ratio = create_buffer(params, ratio)\n phase = np.add.accumulate(frequency / params.sample_rate) % 1\n output = offset + amplitude * np.where(phase < ratio, 1, -1)\n return output"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngenerates a noise signal", "response": "def noise(params, amplitude=1, offset=0):\n '''\n Generate a noise signal\n :param params: buffer parameters, controls length of signal created\n :param amplitude: wave amplitude (array or value)\n :param offset: offset of wave mean from zero (array or value)\n :return: array of resulting signal\n '''\n amplitude = create_buffer(params, amplitude)\n offset = create_buffer(params, offset)\n output = offset + amplitude * (np.random.random(params.length)*2 - 1)\n return output"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef table_wave(params, frequency=400, amplitude=1,\n offset=0, table=None):\n '''\n Generate a wave from a wavetable\n :param params: buffer parameters, controls length of signal created\n :param frequency: wave frequency (array or value)\n :param amplitude: wave amplitude (array or value)\n :param offset: offset of wave mean from zero (array or value)\n :param table: table of values representing one full cycle of the waveform. Nominally the values\n should be between +/-1 (array). The array can be any size.\n :return: array of resulting signal\n '''\n if not table:\n table = create_sine_table(65536)\n size = table.size\n frequency = create_buffer(params, frequency)\n amplitude = create_buffer(params, amplitude)\n offset = create_buffer(params, offset)\n index = np.floor((np.add.accumulate(frequency / params.sample_rate) % 1) * size).astype(int)\n output = offset + amplitude * table[index]\n return output", "response": "Generate a wave from a wavetable."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sine_wave(params, frequency=400, amplitude=1, offset=0):\n '''\n Generate a sine wave\n Convenience function, table_wave generates a sine wave by default\n :param params: buffer parameters, controls length of signal created\n :param frequency: wave frequency (array or value)\n :param amplitude: wave amplitude (array or value)\n :param offset: offset of wave mean from zero (array or value)\n :return: array of resulting signal\n '''\n return table_wave(params, frequency, amplitude, offset)", "response": "Generates a sine wave from a buffer parameters controls length of signal created\n "} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts a midi value to a frequency.", "response": "def midi2f(params, midi=69):\n '''\n Convert a midi value to a frequency.\n Midi value 69 corresponds to A4 (440Hz). Changing the midi value by 1 corresponds to\n one semitone\n :param params: buffer parameters, controls length of signal created\n :param midi: midi value\n :return: array of resulting frequency\n '''\n midi = create_buffer(params, midi)\n output = 2**((midi - 69)/12)*440\n return output"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfetches Lookup by name", "response": "def get_lookup(self, lookup_name):\n \"\"\"Fetch Lookup by name\"\"\"\n from protean.core.repository import BaseLookup\n lookup = self._get_lookup(lookup_name)\n\n # If unable to find Lookup, or if Lookup is the wrong class, raise Error\n if lookup is None or (lookup is not None and not issubclass(lookup, BaseLookup)):\n raise NotImplementedError\n\n return lookup"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nmerging a list of dicts into a single dict.", "response": "def merge_dicts(dicts):\n \"\"\"\n Merge dicts in reverse to preference the order of the original list. e.g.,\n merge_dicts([a, b]) will preference the keys in 'a' over those in 'b'.\n \"\"\"\n merged = {}\n for d in reversed(dicts):\n merged.update(d)\n return merged"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nregister a Lookup to a class", "response": "def register_lookup(cls, lookup, lookup_name=None):\n \"\"\"Register a Lookup to a class\"\"\"\n if lookup_name is None:\n lookup_name = lookup.lookup_name\n if 'class_lookups' not in cls.__dict__:\n cls.class_lookups = {}\n\n cls.class_lookups[lookup_name] = lookup\n cls._clear_cached_lookups()\n\n return lookup"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nremoving given lookup from cls. class_lookups.", "response": "def _unregister_lookup(cls, lookup, lookup_name=None):\n \"\"\"\n Remove given lookup from cls lookups. For use in tests only as it's\n not thread-safe.\n \"\"\"\n if lookup_name is None:\n lookup_name = lookup.lookup_name\n del cls.class_lookups[lookup_name]"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef deconstruct(self):\n path = '%s.%s' % (self.__class__.__module__, self.__class__.__name__)\n args, kwargs = (), {}\n\n if len(self.children) == 1 and not isinstance(self.children[0], Q):\n child = self.children[0]\n kwargs = {child[0]: child[1]}\n else:\n args = tuple(self.children)\n if self.connector != self.default:\n kwargs = {'_connector': self.connector}\n if self.negated:\n kwargs['_negated'] = True\n return path, args, kwargs", "response": "Deconstruct a Q Object"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef perform_import(val):\n if val is None:\n return None\n elif isinstance(val, str):\n return import_from_string(val)\n elif isinstance(val, (list, tuple)):\n return [import_from_string(item) for item in val]\n return val", "response": "Perform a necessary import or imports."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nattempting to import a class from a string representation.", "response": "def import_from_string(val):\n \"\"\"\n Attempt to import a class from a string representation.\n \"\"\"\n try:\n module_path, class_name = val.rsplit('.', 1)\n module = import_module(module_path)\n return getattr(module, class_name)\n except (ImportError, AttributeError) as e:\n msg = f\"Could not import {val}. {e.__class__.__name__}: {e}\"\n raise ImportError(msg)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nload Context with data", "response": "def set_context(self, data):\n \"\"\"Load Context with data\"\"\"\n for key in data:\n setattr(self.local_context, key, data[key])"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nresets all associated values and clean up dictionary items", "response": "def _reset_values(self, instance):\n \"\"\"Reset all associated values and clean up dictionary items\"\"\"\n self.value = None\n self.reference.value = None\n instance.__dict__.pop(self.field_name, None)\n instance.__dict__.pop(self.reference.field_name, None)\n self.reference.delete_cached_value(instance)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nchooses the Linkage attribute between via and designated id_field of the target class.", "response": "def linked_attribute(self):\n \"\"\"Choose the Linkage attribute between `via` and designated `id_field` of the target class\n\n This method is initially called from `__set_name__()` -> `get_attribute_name()`\n at which point, the `to_cls` has not been initialized properly. We simply default\n the linked attribute to 'id' in that case.\n\n Eventually, when setting value the first time, the `to_cls` entity is initialized\n and the attribute name is reset correctly.\n \"\"\"\n if isinstance(self.to_cls, str):\n return 'id'\n else:\n return self.via or self.to_cls.meta_.id_field.attribute_name"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchoosing the Linkage attribute between via and own entity s id_field", "response": "def _linked_attribute(self, owner):\n \"\"\"Choose the Linkage attribute between `via` and own entity's `id_field`\n\n FIXME Explore converting this method into an attribute, and treating it\n uniformly at `association` level.\n \"\"\"\n return self.via or (utils.inflection.underscore(owner.__name__) + '_id')"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _fetch_objects(self, key, value):\n return self.to_cls.query.filter(**{key: value})", "response": "Fetch Multiple linked objects"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nwriting a frame and set a waiter for the response ( unless no_wait is set", "response": "async def _write_frame_awaiting_response(self, waiter_id, frame, request,\n no_wait, check_open=True, drain=True):\n '''Write a frame and set a waiter for\n the response (unless no_wait is set)'''\n if no_wait:\n await self._write_frame(frame, request, check_open=check_open,\n drain=drain)\n return None\n\n f = self._set_waiter(waiter_id)\n try:\n await self._write_frame(frame, request, check_open=check_open,\n drain=drain)\n except Exception:\n self._get_waiter(waiter_id)\n f.cancel()\n raise\n result = await f\n try:\n self._get_waiter(waiter_id)\n except aioamqp.SynchronizationError:\n # no waiter to get\n pass\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef sequencer(params, source_specs):\n '''\n Add a sequence of sounds to a buffer\n :param params: buffer parameters, controls length of signal created\n :param source_spec: list of tuples containing (source, sample). The source is added into the output buffer\n at the sample position given by the second element in the tuple\n :return:\n '''\n output = np.zeros(params.length, dtype=np.float)\n for source, pos in source_specs:\n if pos >= 0:\n length = min(source.size, params.length - pos)\n if length > 0:\n output[pos:pos+length] += source[:length]\n return output", "response": "This function adds a sequence of sounds to a buffer containing the length of the source_specs."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nloading the fields from the base classes.", "response": "def _load_base_class_fields(new_class, bases, attrs):\n \"\"\"If this class is subclassing another Entity, add that Entity's\n fields. Note that we loop over the bases in *reverse*.\n This is necessary in order to maintain the correct order of fields.\n \"\"\"\n for base in reversed(bases):\n if hasattr(base, 'meta_') and \\\n hasattr(base.meta_, 'declared_fields'):\n base_class_fields = {\n field_name: field_obj for (field_name, field_obj)\n in base.meta_.declared_fields.items()\n if field_name not in attrs and not field_obj.identifier\n }\n new_class._load_fields(base_class_fields)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _load_fields(new_class, attrs):\n for attr_name, attr_obj in attrs.items():\n if isinstance(attr_obj, (Field, Reference)):\n setattr(new_class, attr_name, attr_obj)\n new_class.meta_.declared_fields[attr_name] = attr_obj", "response": "Load field items into Class.\nCOOKIES This method sets up the primary attribute of an association.\nCOOKIES"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _set_up_reference_fields(new_class):\n if new_class.meta_.declared_fields:\n for _, field in new_class.meta_.declared_fields.items():\n if isinstance(field, Reference):\n shadow_field_name, shadow_field = field.get_shadow_field()\n setattr(new_class, shadow_field_name, shadow_field)\n shadow_field.__set_name__(new_class, shadow_field_name)", "response": "Walk through relation fields and setup shadow attributes"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate and return a default ID field that is Auto generated", "response": "def _create_id_field(new_class):\n \"\"\"Create and return a default ID field that is Auto generated\"\"\"\n id_field = Auto(identifier=True)\n\n setattr(new_class, 'id', id_field)\n id_field.__set_name__(new_class, 'id')\n\n # Ensure ID field is updated properly in Meta attribute\n new_class.meta_.declared_fields['id'] = id_field\n new_class.meta_.id_field = id_field"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nloads list of attributes from declared fields", "response": "def _load_attributes(new_class):\n \"\"\"Load list of attributes from declared fields\"\"\"\n for field_name, field_obj in new_class.meta_.declared_fields.items():\n new_class.meta_.attributes[field_obj.get_attribute_name()] = field_obj"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef unique_fields(self):\n return [(field_name, field_obj)\n for field_name, field_obj in self.declared_fields.items()\n if field_obj.unique]", "response": "Return the unique fields for this entity"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn entity data as a dictionary", "response": "def to_dict(self):\n \"\"\" Return entity data as a dictionary \"\"\"\n return {field_name: getattr(self, field_name, None)\n for field_name in self.meta_.declared_fields}"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef clone(self):\n clone_copy = copy.deepcopy(self)\n clone_copy.state_ = EntityState()\n\n return clone_copy", "response": "Deepclone the entity but reset state"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget a specific Record from the Repository.", "response": "def get(cls, identifier: Any) -> 'Entity':\n \"\"\"Get a specific Record from the Repository\n\n :param identifier: id of the record to be fetched from the repository.\n \"\"\"\n logger.debug(f'Lookup `{cls.__name__}` object with identifier {identifier}')\n # Get the ID field for the entity\n filters = {\n cls.meta_.id_field.field_name: identifier\n }\n\n # Find this item in the repository or raise Error\n results = cls.query.filter(**filters).limit(1).all()\n if not results:\n raise ObjectNotFoundError(\n f'`{cls.__name__}` object with identifier {identifier} '\n f'does not exist.')\n\n # Return the first result\n return results.first"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef reload(self) -> None:\n if not self.state_.is_persisted or self.state_.is_changed:\n raise InvalidStateError(f'`{self.__class__.__name__}` object is in invalid state')\n\n # Retrieve the entity's ID by the configured Identifier field\n identifier = getattr(self, self.meta_.id_field.field_name)\n logger.debug(f'Lookup `{self.__class__.__name__}` object with '\n f'identifier {self.meta_.id_field}')\n\n # Fetch the entity data from db by its identifier\n db_value = self.get(identifier)\n\n # Update own data from fetched entity data\n # This allows us to ``dog.reload()`` instead of ``dog = dog.reload()``\n self._update_data(db_value.to_dict())", "response": "Reloads the entity from the repository"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef find_by(cls, **kwargs) -> 'Entity':\n logger.debug(f'Lookup `{cls.__name__}` object with values '\n f'{kwargs}')\n\n # Find this item in the repository or raise Error\n results = cls.query.filter(**kwargs).limit(1).all()\n\n if not results:\n raise ObjectNotFoundError(\n f'`{cls.__name__}` object with values {[item for item in kwargs.items()]} '\n f'does not exist.')\n\n # Return the first result\n return results.first", "response": "Find a specific entity record that matches one or more criteria."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn True if objects matching the provided filters and excludes exist if not return false.", "response": "def exists(cls, excludes_, **filters):\n \"\"\" Return `True` if objects matching the provided filters and excludes\n exist if not return false.\n\n Calls the `filter` method by default, but can be overridden for better and\n quicker implementations that may be supported by a database.\n\n :param excludes_: entities without this combination of field name and\n values will be returned\n \"\"\"\n results = cls.query.filter(**filters).exclude(**excludes_)\n return bool(results)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a new record in the repository.", "response": "def create(cls, *args, **kwargs) -> 'Entity':\n \"\"\"Create a new record in the repository.\n\n Also performs unique validations before creating the entity\n\n :param args: positional arguments for the entity\n :param kwargs: keyword arguments for the entity\n \"\"\"\n logger.debug(\n f'Creating new `{cls.__name__}` object using data {kwargs}')\n\n model_cls = repo_factory.get_model(cls)\n repository = repo_factory.get_repository(cls)\n\n try:\n # Build the entity from the input arguments\n # Raises validation errors, if any, at this point\n entity = cls(*args, **kwargs)\n\n # Do unique checks, create this object and return it\n entity._validate_unique()\n\n # Perform Pre-Save Actions\n entity.pre_save()\n\n # Build the model object and create it\n model_obj = repository.create(model_cls.from_entity(entity))\n\n # Update the auto fields of the entity\n for field_name, field_obj in entity.meta_.declared_fields.items():\n if isinstance(field_obj, Auto):\n if isinstance(model_obj, dict):\n field_val = model_obj[field_name]\n else:\n field_val = getattr(model_obj, field_name)\n setattr(entity, field_name, field_val)\n\n # Set Entity status to saved\n entity.state_.mark_saved()\n\n # Perform Post-Save Actions\n entity.post_save()\n\n return entity\n except ValidationError:\n # FIXME Log Exception\n raise"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsave a new Entity into the repository.", "response": "def save(self):\n \"\"\"Save a new Entity into repository.\n\n Performs unique validations before creating the entity.\n \"\"\"\n logger.debug(\n f'Saving `{self.__class__.__name__}` object')\n\n # Fetch Model class and connected repository from Repository Factory\n model_cls = repo_factory.get_model(self.__class__)\n repository = repo_factory.get_repository(self.__class__)\n\n try:\n # Do unique checks, update the record and return the Entity\n self._validate_unique(create=False)\n\n # Perform Pre-Save Actions\n self.pre_save()\n\n # Build the model object and create it\n model_obj = repository.create(model_cls.from_entity(self))\n\n # Update the auto fields of the entity\n for field_name, field_obj in self.meta_.declared_fields.items():\n if isinstance(field_obj, Auto):\n if isinstance(model_obj, dict):\n field_val = model_obj[field_name]\n else:\n field_val = getattr(model_obj, field_name)\n setattr(self, field_name, field_val)\n\n # Set Entity status to saved\n self.state_.mark_saved()\n\n # Perform Post-Save Actions\n self.post_save()\n\n return self\n except Exception:\n # FIXME Log Exception\n raise"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef update(self, *data, **kwargs) -> 'Entity':\n logger.debug(f'Updating existing `{self.__class__.__name__}` object with id {self.id}')\n\n # Fetch Model class and connected repository from Repository Factory\n model_cls = repo_factory.get_model(self.__class__)\n repository = repo_factory.get_repository(self.__class__)\n\n try:\n # Update entity's data attributes\n self._update_data(*data, **kwargs)\n\n # Do unique checks, update the record and return the Entity\n self._validate_unique(create=False)\n\n # Perform Pre-Save Actions\n self.pre_save()\n\n repository.update(model_cls.from_entity(self))\n\n # Set Entity status to saved\n self.state_.mark_saved()\n\n # Perform Post-Save Actions\n self.post_save()\n\n return self\n except Exception:\n # FIXME Log Exception\n raise", "response": "Update an existing entity s data attributes and returns the new entity."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nvalidate the unique constraints for the entity.", "response": "def _validate_unique(self, create=True):\n \"\"\" Validate the unique constraints for the entity \"\"\"\n # Build the filters from the unique constraints\n filters, excludes = {}, {}\n\n for field_name, field_obj in self.meta_.unique_fields:\n lookup_value = getattr(self, field_name, None)\n # Ignore empty lookup values\n if lookup_value in Field.empty_values:\n continue\n # Ignore identifiers on updates\n if not create and field_obj.identifier:\n excludes[field_name] = lookup_value\n continue\n filters[field_name] = lookup_value\n\n # Lookup the objects by the filters and raise error on results\n for filter_key, lookup_value in filters.items():\n if self.exists(excludes, **{filter_key: lookup_value}):\n field_obj = self.meta_.declared_fields[filter_key]\n field_obj.fail('unique',\n entity_name=self.__class__.__name__,\n field_name=filter_key)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndelete a Record from the Repository and return the object.", "response": "def delete(self):\n \"\"\"Delete a Record from the Repository\n\n will perform callbacks and run validations before deletion.\n\n Throws ObjectNotFoundError if the object was not found in the repository.\n \"\"\"\n # Fetch Model class and connected repository from Repository Factory\n model_cls = repo_factory.get_model(self.__class__)\n repository = repo_factory.get_repository(self.__class__)\n\n try:\n if not self.state_.is_destroyed:\n # Update entity's data attributes\n repository.delete(model_cls.from_entity(self))\n\n # Set Entity status to saved\n self.state_.mark_destroyed()\n\n return self\n except Exception:\n # FIXME Log Exception\n raise"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nadds an item to the internal dictionary.", "response": "def add(self, item, count=1):\n '''\n When we receive stream of data, we add them in the chunk\n which has limit on the no. of items that it will store.\n >>> s = StreamCounter(5,5)\n >>> data_stream = ['a','b','c','d']\n >>> for item in data_stream:\n ... s.add(item)\n >>> s.chunk_size\n 5\n >>> s.n_items_seen\n 4\n >>> s.n_chunk_items_seen\n 4\n >>> s.n_chunks\n 0\n >>> from pprint import pprint\n >>> pprint(s.chunked_counts.get(s.n_chunks, {}))\n {'a': 1, 'b': 1, 'c': 1, 'd': 1}\n >>> s.counts_total\n 4\n >>> data_stream = ['a','b','c','d','e','f','g','e']\n >>> for item in data_stream:\n ... s.add(item)\n >>> s.chunk_size\n 5\n >>> s.n_items_seen\n 12\n >>> s.n_chunk_items_seen\n 2\n >>> s.n_chunks\n 2\n >>> s.chunked_counts.get(s.n_chunks, {})\n {'g': 1, 'e': 1}\n '''\n self.n_items_seen += count\n self.n_chunk_items_seen += count\n\n # get current chunk\n chunk_id = self.n_chunks\n chunk = self.chunked_counts.get(chunk_id, {})\n self.chunked_counts[chunk_id] = chunk\n\n # update count in the current chunk counter dict\n if item in chunk:\n chunk[item] += count\n else:\n self.n_counts += 1\n chunk[item] = count\n\n # is the current chunk done?\n if self.n_chunk_items_seen >= self.chunk_size:\n self.n_chunks += 1\n self.n_chunk_items_seen = 0\n\n # In case we reached max capacity in count entries,\n # drop oldest chunks until we come back within limit\n while self.n_counts >= self.max_counts:\n self._drop_oldest_chunk()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _drop_oldest_chunk(self):\n '''\n To handle the case when the items comming in the chunk\n is more than the maximum capacity of the chunk. Our intent\n behind is to remove the oldest chunk. So that the items come\n flowing in.\n >>> s = StreamCounter(5,5)\n >>> data_stream = ['a','b','c','d']\n >>> for item in data_stream:\n ... s.add(item)\n >>> min(s.chunked_counts.keys())\n 0\n >>> s.chunked_counts\n {0: {'a': 1, 'b': 1, 'c': 1, 'd': 1}}\n >>> data_stream = ['a','b','c','d','a','e','f']\n >>> for item in data_stream:\n ... s.add(item)\n >>> min(s.chunked_counts.keys())\n 2\n >>> s.chunked_counts\n {2: {'f': 1}}\n '''\n chunk_id = min(self.chunked_counts.keys())\n chunk = self.chunked_counts.pop(chunk_id)\n\n self.n_counts -= len(chunk)\n for k, v in list(chunk.items()):\n self.counts[k] -= v\n self.counts_total -= v", "response": "Drop the oldest chunk from the internal counts."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get(self, item, default=0, normalized=False):\n '''\n When we have the stream of data pushed in the chunk\n we can retrive count of an item using this method.\n >>> stream_counter_obj = StreamCounter(5,5)\n >>> data_stream = ['a','b','c']\n >>> for item in data_stream:\n ... stream_counter_obj.add(item)\n >>> stream_counter_obj.get('a')\n 1\n >>> stream_counter_obj.get('b')\n 1\n >>> stream_counter_obj.get('c')\n 1\n >>> stream_counter_obj.get('d')\n 0\n >>> data_stream.extend(['d','e','f'])\n >>> for item in data_stream:\n ... stream_counter_obj.add(item)\n >>> stream_counter_obj.get('a')\n 0\n >>> stream_counter_obj.get('b')\n 0\n >>> stream_counter_obj.get('c')\n 1\n >>> stream_counter_obj.get('d')\n 1\n >>> stream_counter_obj.get('e')\n 1\n >>> stream_counter_obj.get('f')\n 1\n '''\n c = self.counts.get(item, default)\n if not normalized:\n return c\n\n return c / float(self.counts_total)", "response": "Get the count of an item from the cache."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef message(self):\n return '\\n'.join(\n [self.from_email, str(self.to), self.subject, self.body])", "response": "Convert the message to a mime compliant email string"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a list of all recipients of the email.", "response": "def recipients(self):\n \"\"\"\n Return a list of all recipients of the email (includes direct\n addressees as well as Cc and Bcc entries).\n \"\"\"\n return [email for email in (self.to + self.cc + self.bcc) if email]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nretrieve connection to send email", "response": "def get_connection(self, fail_silently=False):\n \"\"\"Retrieve connection to send email\"\"\"\n from protean.services.email import get_connection\n\n if not self.connection:\n self.connection = get_connection(fail_silently=fail_silently)\n\n return self.connection"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef send(self, fail_silently=False):\n if not self.recipients():\n # Don't bother creating the network connection if\n # there's nobody to send to.\n return 0\n return self.get_connection(fail_silently).send_messages([self])", "response": "Send the email message."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_static_route(self, method: Union[str, Methods], route: str, handler: Callable,\n skip_middleware=False):\n \"\"\"\n Adds a static route. A static route is a special route that\n doesnt follow any of the normal rules, and never has any path\n parameters.\n Ideally, this is used for non-public facing endpoints such as\n \"/healthcheck\", or \"/stats\" or something of that nature.\n\n All static routes SKIP middlewares\n \"\"\"\n if isinstance(method, str):\n method = Methods(method.upper())\n route = self._prefix + route\n route = route.strip('/')\n if route not in self._static_routes:\n self._static_routes[route] = {}\n self._static_routes[route][method] = handler", "response": "Adds a static route to the internal routes dict."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nadd a prefix to routes contained within.", "response": "def prefix(self, prefix):\n \"\"\"\n Adds a prefix to routes contained within.\n \"\"\"\n original_prefix = self._prefix\n self._prefix += prefix\n yield self\n self._prefix = original_prefix"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns all the error messages as a dictionary", "response": "def normalized_messages(self, no_field_name='_entity'):\n \"\"\"Return all the error messages as a dictionary\"\"\"\n if isinstance(self.messages, dict):\n return self.messages\n if not self.field_names:\n return {no_field_name: self.messages}\n\n return dict((name, self.messages) for name in self.field_names)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef generate_random_string(length=6):\n '''\n Returns a random string of a specified length.\n\n >>> len(generate_random_string(length=25))\n 25\n\n Test randomness. Try N times and observe no duplicaton\n >>> N = 100\n >>> len(set(generate_random_string(10) for i in range(N))) == N\n True\n '''\n n = int(length / 2 + 1)\n x = binascii.hexlify(os.urandom(n))\n s = x[:length]\n return s.decode('utf-8')", "response": "Returns a random string of a specified length."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns current timestamp of the last 1432166400 - > 1432166400 - > 1432166400", "response": "def get_timestamp(dt=None):\n '''\n Return current timestamp if @dt is None\n else return timestamp of @dt.\n\n >>> t = datetime.datetime(2015, 0o5, 21)\n >>> get_timestamp(t)\n 1432166400\n '''\n\n if dt is None: dt = datetime.datetime.utcnow()\n t = dt.utctimetuple()\n\n return calendar.timegm(t)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets datetime from an epoch timestamp", "response": "def get_datetime(epoch):\n '''\n get datetime from an epoch timestamp\n\n >>> get_datetime(1432188772)\n datetime.datetime(2015, 5, 21, 6, 12, 52)\n '''\n\n t = time.gmtime(epoch)\n dt = datetime.datetime(*t[:6])\n\n return dt"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef convert_ts(tt):\n '''\n tt: time.struct_time(tm_year=2012, tm_mon=10, tm_mday=23, tm_hour=0,\n tm_min=0, tm_sec=0, tm_wday=1, tm_yday=297, tm_isdst=-1)\n\n >>> tt = time.strptime(\"23.10.2012\", \"%d.%m.%Y\")\n >>> convert_ts(tt)\n 1350950400\n\n tt: time.struct_time(tm_year=1513, tm_mon=1, tm_mday=1, tm_hour=0,\n tm_min=0, tm_sec=0, tm_wday=2, tm_yday=1, tm_isdst=0)\n\n >>> tt = time.strptime(\"1.1.1513\", \"%d.%m.%Y\")\n >>> convert_ts(tt)\n 0\n >>> tt = 12\n >>> convert_ts(tt)\n '''\n try:\n ts = calendar.timegm(tt)\n\n '''\n As from the github issue https://github.com/prashanthellina/rsslurp/issues/680,\n there are some cases where we might get timestamp in negative values, so consider\n 0 if the converted timestamp is negative value.\n '''\n if ts < 0: ts = 0\n except TypeError:\n ts = None\n return ts", "response": "Convert a timestamp to a date in the current date format."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef xcode(text, encoding='utf8', mode='ignore'):\n '''\n Converts unicode encoding to str\n\n >>> xcode(b'hello')\n b'hello'\n >>> xcode('hello')\n b'hello'\n '''\n return text.encode(encoding, mode) if isinstance(text, str) else text", "response": "Converts unicode encoding to str\n "} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning all the keys in nested a dictionary.", "response": "def serialize_dict_keys(d, prefix=\"\"):\n \"\"\"returns all the keys in nested a dictionary.\n >>> sorted(serialize_dict_keys({\"a\": {\"b\": {\"c\": 1, \"b\": 2} } }))\n ['a', 'a.b', 'a.b.b', 'a.b.c']\n \"\"\"\n keys = []\n for k, v in d.items():\n fqk = '{}{}'.format(prefix, k)\n keys.append(fqk)\n if isinstance(v, dict):\n keys.extend(serialize_dict_keys(v, prefix=\"{}.\".format(fqk)))\n\n return keys"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nflatten a nested dictionary d into a nested dictionary of items.", "response": "def flatten_dict(d,\n parent_key='', sep='.',\n ignore_under_prefixed=True,\n mark_value=True):\n '''\n Flattens a nested dictionary\n\n >>> from pprint import pprint\n >>> d = {\"a\": {\"b\": {\"c\": 1, \"b\": 2, \"__d\": 'ignore', \"_e\": \"mark\"} } }\n >>> fd = flatten_dict(d)\n >>> pprint(fd)\n {'a.b._e': \"'mark'\", 'a.b.b': 2, 'a.b.c': 1}\n '''\n items = {}\n for k in d:\n if ignore_under_prefixed and k.startswith('__'): continue\n v = d[k]\n if mark_value and k.startswith('_') and not k.startswith('__'):\n v = MarkValue(repr(v))\n\n new_key = sep.join((parent_key, k)) if parent_key else k\n if isinstance(v, collections.MutableMapping):\n items.update(flatten_dict(v, new_key, sep=sep,\n ignore_under_prefixed=True,\n mark_value=True)\n )\n else:\n items[new_key] = v\n\n return items"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef deepgetattr(obj, attr, default=AttributeError):\n try:\n return reduce(getattr, attr.split('.'), obj)\n except AttributeError:\n if default is not AttributeError:\n return default\n raise", "response": "Get the ultimate value of an attribute in an object."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsets the limit on number of file descriptors that this process can open.", "response": "def set_file_limits(n):\n '''\n Set the limit on number of file descriptors\n that this process can open.\n\n '''\n\n try:\n resource.setrlimit(resource.RLIMIT_NOFILE, (n, n))\n return True\n except ValueError:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef memoize(f):\n '''\n Caches result of a function\n From: https://goo.gl/aXt4Qy\n\n >>> import time\n \n >>> @memoize\n ... def test(msg):\n ... # Processing for result that takes time\n ... time.sleep(1)\n ... return msg\n >>>\n >>> for i in range(5):\n ... start = time.time()\n ... test('calling memoized function')\n ... time_taken = time.time() - start\n ... # For first time it takes usual time\n ... if i == 0 and time_taken >= 1: print('ok')\n ... # Faster from the 2nd time\n ... elif i != 0 and time_taken <= 1: print('ok')\n ... else: print('NOT ok!')\n 'calling memoized function'\n ok\n 'calling memoized function'\n ok\n 'calling memoized function'\n ok\n 'calling memoized function'\n ok\n 'calling memoized function'\n ok\n '''\n class memodict(dict):\n\n @wraps(f)\n def __getitem__(self, *args):\n return super(memodict, self).__getitem__(*args)\n\n def __missing__(self, key):\n self[key] = ret = f(key)\n return ret\n return memodict().__getitem__", "response": "Memoizes the result of a function f."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef load_object(imp_path):\n '''\n Given a python import path, load the object\n For dynamic imports in a program\n\n >>> isdir = load_object('os.path.isdir')\n >>> isdir('/tmp')\n True\n\n >>> num = load_object('numbers.Number')\n >>> isinstance('x', num)\n False\n >>> isinstance(777, num)\n True\n '''\n module_name, obj_name = imp_path.split('.', 1)\n module = __import__(module_name)\n obj = attrgetter(obj_name)(module)\n\n return obj", "response": "Given a python import path load the object"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nperforming validation on value. Raise a : exc : ValidationException if validation does not succeed.", "response": "def _run_validators(self, value):\n \"\"\"Perform validation on ``value``. Raise a :exc:`ValidationError` if\n validation does not succeed.\n \"\"\"\n if value in self.empty_values:\n return\n\n errors = []\n for validator in self.validators:\n try:\n validator(value)\n except exceptions.ValidationError as err:\n if isinstance(err.messages, dict):\n errors.append(err.messages)\n else:\n errors.extend(err.messages)\n\n if errors:\n raise exceptions.ValidationError(errors)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _load(self, value: Any):\n\n if value in self.empty_values:\n # If a default has been set for the field return it\n if self.default is not None:\n default = self.default\n value = default() if callable(default) else default\n return value\n\n # If no default is set and this field is required\n elif self.required:\n self.fail('required')\n\n # In all other cases just return `None` as we do not want to\n # run validations against an empty value\n else:\n return None\n\n # If choices exist then validate that value is be one of the choices\n if self.choices:\n value_list = value\n if not isinstance(value, (list, tuple)):\n value_list = [value]\n for v in value_list:\n if v not in self.choice_dict:\n self.fail(\n 'invalid_choice', value=v,\n choices=list(self.choice_dict))\n\n # Cast and Validate the value for this Field\n value = self._cast_to_type(value)\n\n # Call the rest of the validators defined for this Field\n self._run_validators(value)\n\n return value", "response": "Load the value for the field and run validators and return the value."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _extract_lookup(self, key):\n parts = key.split('__')\n # 'exact' is the default lookup if there was no explicit comparison op in `key`\n # Assume there is only one `__` in the key.\n # FIXME Change for child attribute query support\n op = 'exact' if len(parts) == 1 else parts[1]\n\n # Construct and assign the lookup class as a filter criteria\n return parts[0], self.get_lookup(op)", "response": "Extract lookup method based on key name format"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntransforming from json to Request", "response": "def exec(self, payload: Log2ReqsAddOnPayload) -> TList[Request]:\n \"\"\"Transform from json to Request\n\n Exception:\n ValueError: If path does not exist.\n \"\"\"\n try:\n return Request.from_jsonf_to_list(payload.file, encoding=self.config.encoding)\n except TypeError as e:\n raise ValueError(e)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the expiry value usable by this backend based upon the provided timeout.", "response": "def get_backend_expiry(self, expiry=DEFAULT_EXPIRY):\n \"\"\"\n Return the expiry value usable by this backend based upon the provided\n timeout.\n \"\"\"\n if expiry == DEFAULT_EXPIRY:\n expiry = self.default_expiry\n elif expiry == 0:\n # avoid time.time() related precision issues\n expiry = -1\n return None if expiry is None else time.time() + expiry"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfetch a bunch of keys from the cache.", "response": "def get_many(self, keys):\n \"\"\"\n Fetch a bunch of keys from the cache. For certain backends (memcached,\n pgsql) this can be *much* faster when fetching multiple values.\n Return a dict mapping each key in keys to its value. If the given\n key is missing, it will be missing from the response dict.\n \"\"\"\n d = {}\n for k in keys:\n val = self.get(k)\n if val is not None:\n d[k] = val\n\n return d"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nincrements the value of a key in the cache.", "response": "def incr(self, key, delta=1):\n \"\"\"\n Add delta to value in the cache. If the key does not exist, raise a\n ValueError exception.\n \"\"\"\n value = self.get(key)\n if value is None:\n raise ValueError(\"Key '%s' not found\" % key)\n new_value = value + delta\n self.set(key, new_value)\n return new_value"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set_many(self, data, expiry=DEFAULT_EXPIRY):\n for key, value in data.items():\n self.set(key, value, expiry=expiry)\n return []", "response": "Set a bunch of values in the cache at once from a dict of key - value pairs."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef init_logger(v_num: int):\n logging.addLevelName(LogLevel.INFO_LV1.value, 'INFO_LV1') # type: ignore # Prevent for enum problem\n logging.addLevelName(LogLevel.INFO_LV2.value, 'INFO_LV2') # type: ignore # Prevent for enum problem\n logging.addLevelName(LogLevel.INFO_LV3.value, 'INFO_LV3') # type: ignore # Prevent for enum problem\n\n logging.config.dictConfig(create_logger_config({ # type: ignore # Prevent for enum problem\n 0: LogLevel.INFO_LV1,\n 1: LogLevel.INFO_LV2,\n 2: LogLevel.INFO_LV3,\n 3: LogLevel.DEBUG,\n }[v_num]))", "response": "Call when initialize Jumeaux!!\n "} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreads config file and initialize providers", "response": "def _initialize_providers(self):\n \"\"\"Read config file and initialize providers\"\"\"\n configured_providers = active_config.DATABASES\n provider_objects = {}\n\n if not isinstance(configured_providers, dict) or configured_providers == {}:\n raise ConfigurationError(\n \"'DATABASES' config must be a dict and at least one \"\n \"provider must be defined\")\n\n if 'default' not in configured_providers:\n raise ConfigurationError(\n \"You must define a 'default' provider\")\n\n for provider_name, conn_info in configured_providers.items():\n provider_full_path = conn_info['PROVIDER']\n provider_module, provider_class = provider_full_path.rsplit('.', maxsplit=1)\n\n provider_cls = getattr(importlib.import_module(provider_module), provider_class)\n provider_objects[provider_name] = provider_cls(conn_info)\n\n return provider_objects"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfetching a specific provider with the specified name in Configuration file", "response": "def get_provider(self, provider_name='default'):\n \"\"\"Fetch provider with the name specified in Configuration file\"\"\"\n try:\n if self._providers is None:\n self._providers = self._initialize_providers()\n return self._providers[provider_name]\n except KeyError:\n raise AssertionError(f'No Provider registered with name {provider_name}')"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_connection(self, provider_name='default'):\n try:\n return self._providers[provider_name].get_connection()\n except KeyError:\n raise AssertionError(f'No Provider registered with name {provider_name}')", "response": "Fetch connection from Provider"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nupdate the default settings for an extension from an object", "response": "def update_defaults(self, ext_config):\n \"\"\" Update the default settings for an extension from an object\"\"\"\n for setting in dir(ext_config):\n if setting.isupper() and not hasattr(self, setting):\n setattr(self, setting, getattr(ext_config, setting))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef keeprunning(wait_secs=0, exit_on_success=False,\n on_success=None, on_error=None, on_done=None):\n '''\n Example 1: dosomething needs to run until completion condition\n without needing to have a loop in its code. Also, when error\n happens, we should NOT terminate execution\n\n >>> from deeputil import AttrDict\n >>> @keeprunning(wait_secs=1)\n ... def dosomething(state):\n ... state.i += 1\n ... print (state)\n ... if state.i % 2 == 0:\n ... print(\"Error happened\")\n ... 1 / 0 # create an error condition\n ... if state.i >= 7:\n ... print (\"Done\")\n ... raise keeprunning.terminate\n ...\n\n >>> state = AttrDict(i=0)\n >>> dosomething(state)\n AttrDict({'i': 1})\n AttrDict({'i': 2})\n Error happened\n AttrDict({'i': 3})\n AttrDict({'i': 4})\n Error happened\n AttrDict({'i': 5})\n AttrDict({'i': 6})\n Error happened\n AttrDict({'i': 7})\n Done\n\n Example 2: In case you want to log exceptions while\n dosomething keeps running, or perform any other action\n when an exceptions arise\n\n >>> def some_error(__exc__):\n ... print (__exc__)\n ...\n >>> @keeprunning(on_error=some_error)\n ... def dosomething(state):\n ... state.i += 1\n ... print (state)\n ... if state.i % 2 == 0:\n ... print(\"Error happened\")\n ... 1 / 0 # create an error condition\n ... if state.i >= 7:\n ... print (\"Done\")\n ... raise keeprunning.terminate\n ...\n\n >>> state = AttrDict(i=0)\n >>> dosomething(state)\n AttrDict({'i': 1})\n AttrDict({'i': 2})\n Error happened\n division by zero\n AttrDict({'i': 3})\n AttrDict({'i': 4})\n Error happened\n division by zero\n AttrDict({'i': 5})\n AttrDict({'i': 6})\n Error happened\n division by zero\n AttrDict({'i': 7})\n Done\n\n Example 3: Full set of arguments that can be passed in @keeprunning()\n with class implementations\n\n >>> # Class that has some class variables\n ... class Demo(object):\n ... SUCCESS_MSG = 'Yay!!'\n ... DONE_MSG = 'STOPPED AT NOTHING!'\n ... ERROR_MSG = 'Error'\n ...\n ... # Functions to be called by @keeprunning\n ... def success(self):\n ... print((self.SUCCESS_MSG))\n ...\n ... def failure(self, __exc__):\n ... print((self.ERROR_MSG, __exc__))\n ...\n ... def task_done(self):\n ... print((self.DONE_MSG))\n ...\n ... #Actual use of keeprunning with all arguments passed\n ... @keeprunning(wait_secs=1, exit_on_success=False,\n ... on_success=success, on_error=failure, on_done=task_done)\n ... def dosomething(self, state):\n ... state.i += 1\n ... print (state)\n ... if state.i % 2 == 0:\n ... print(\"Error happened\")\n ... # create an error condition\n ... 1 / 0\n ... if state.i >= 7:\n ... print (\"Done\")\n ... raise keeprunning.terminate\n ...\n\n >>> demo = Demo()\n >>> state = AttrDict(i=0)\n >>> demo.dosomething(state)\n AttrDict({'i': 1})\n Yay!!\n AttrDict({'i': 2})\n Error happened\n ('Error', ZeroDivisionError('division by zero'))\n AttrDict({'i': 3})\n Yay!!\n AttrDict({'i': 4})\n Error happened\n ('Error', ZeroDivisionError('division by zero'))\n AttrDict({'i': 5})\n Yay!!\n AttrDict({'i': 6})\n Error happened\n ('Error', ZeroDivisionError('division by zero'))\n AttrDict({'i': 7})\n Done\n STOPPED AT NOTHING!\n '''\n def decfn(fn):\n\n def _call_callback(cb, fargs):\n if not cb: return\n # get the getargspec fn in inspect module (python 2/3 support)\n G = getattr(inspect, 'getfullargspec', getattr(inspect, 'getargspec'))\n cb_args = G(cb).args\n cb_args = dict([(a, fargs.get(a, None)) for a in cb_args])\n cb(**cb_args)\n\n def _fn(*args, **kwargs):\n fargs = inspect.getcallargs(fn, *args, **kwargs)\n fargs.update(dict(__fn__=fn, __exc__=None))\n\n while 1:\n try:\n fn(*args, **kwargs)\n if exit_on_success: break\n except (SystemExit, KeyboardInterrupt):\n raise\n except KeepRunningTerminate:\n break\n except Exception as exc:\n fargs.update(dict(__exc__=exc))\n _call_callback(on_error, fargs)\n fargs.update(dict(__exc__=None))\n if wait_secs: time.sleep(wait_secs)\n continue\n\n _call_callback(on_success, fargs)\n\n _call_callback(on_done, fargs)\n\n return _fn\n\n return decfn", "response": "This function is a generator function that returns a sequence of nodes that are not yet in the tree."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nregisters the given entity with the factory.", "response": "def register(self, entity_cls, provider_name=None):\n \"\"\" Register the given model with the factory\n :param entity_cls: Entity class to be registered\n :param provider: Optional provider to associate with Entity class\n \"\"\"\n self._validate_entity_cls(entity_cls)\n\n # Register the entity if not registered already\n entity_name = fully_qualified_name(entity_cls)\n provider_name = provider_name or entity_cls.meta_.provider or 'default'\n\n try:\n entity = self._get_entity_by_class(entity_cls)\n\n if entity:\n # This probably is an accidental re-registration of the entity\n # and we should warn the user of a possible repository confusion\n raise ConfigurationError(\n f'Entity {entity_name} has already been registered')\n except AssertionError:\n # Entity has not been registered yet. Let's go ahead and add it to the registry.\n entity_record = RepositoryFactory.EntityRecord(\n name=entity_cls.__name__,\n qualname=entity_name,\n entity_cls=entity_cls,\n provider_name=provider_name,\n model_cls=None\n )\n self._registry[entity_name] = entity_record\n logger.debug(\n f'Registered entity {entity_name} with provider {provider_name}')"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfetching by Entity Name in values", "response": "def _find_entity_in_records_by_class_name(self, entity_name):\n \"\"\"Fetch by Entity Name in values\"\"\"\n records = {\n key: value for (key, value)\n in self._registry.items()\n if value.name == entity_name\n }\n # If more than one record was found, we are dealing with the case of\n # an Entity name present in multiple places (packages or plugins). Throw an error\n # and ask for a fully qualified Entity name to be specified\n if len(records) > 1:\n raise ConfigurationError(\n f'Entity with name {entity_name} has been registered twice. '\n f'Please use fully qualified Entity name to specify the exact Entity.')\n elif len(records) == 1:\n return next(iter(records.values()))\n else:\n raise AssertionError(f'No Entity registered with name {entity_name}')"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_entity_by_class(self, entity_cls):\n entity_qualname = fully_qualified_name(entity_cls)\n if entity_qualname in self._registry:\n return self._registry[entity_qualname]\n else:\n return self._find_entity_in_records_by_class_name(entity_cls.__name__)", "response": "Fetch Entity record with Entity class details"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfetches an Entity record with an Entity name", "response": "def _get_entity_by_name(self, entity_name):\n \"\"\"Fetch Entity record with an Entity name\"\"\"\n if entity_name in self._registry:\n return self._registry[entity_name]\n else:\n return self._find_entity_in_records_by_class_name(entity_name)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _validate_entity_cls(self, entity_cls):\n # Import here to avoid cyclic dependency\n from protean.core.entity import Entity\n\n if not issubclass(entity_cls, Entity):\n raise AssertionError(\n f'Entity {entity_cls.__name__} must be subclass of `Entity`')\n\n if entity_cls.meta_.abstract is True:\n raise NotSupportedError(\n f'{entity_cls.__name__} class has been marked abstract'\n f' and cannot be instantiated')", "response": "Validate that the given Entity class is a valid class."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_model(self, entity_cls):\n entity_record = self._get_entity_by_class(entity_cls)\n\n model_cls = None\n if entity_record.model_cls:\n model_cls = entity_record.model_cls\n else:\n # We should ask the Provider to give a fully baked model the first time\n # that has been initialized properly for this entity\n provider = self.get_provider(entity_record.provider_name)\n baked_model_cls = provider.get_model(entity_record.entity_cls)\n\n # Record for future reference\n new_entity_record = entity_record._replace(model_cls=baked_model_cls)\n self._registry[entity_record.qualname] = new_entity_record\n\n model_cls = baked_model_cls\n\n return model_cls", "response": "Retrieve Model class connected to Entity"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_repository(self, entity_cls):\n entity_record = self._get_entity_by_class(entity_cls)\n provider = self.get_provider(entity_record.provider_name)\n\n return provider.get_repository(entity_record.entity_cls)", "response": "Retrieve a Repository for the Model with a live connection"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\nasync def set_heater_values(heater_data, heater):\n heater.current_temp = heater_data.get('currentTemp')\n heater.device_status = heater_data.get('deviceStatus')\n heater.available = heater.device_status == 0\n heater.name = heater_data.get('deviceName')\n heater.fan_status = heater_data.get('fanStatus')\n heater.is_holiday = heater_data.get('isHoliday')\n\n # Room assigned devices don't report canChangeTemp\n # in selectDevice response.\n if heater.room is None:\n heater.can_change_temp = heater_data.get('canChangeTemp')\n\n # Independent devices report their target temperature via\n # holidayTemp value. But isHoliday is still set to 0.\n # Room assigned devices may have set \"Control Device individually\"\n # which effectively set their isHoliday value to 1.\n # In this mode they behave similar to independent devices\n # reporting their target temperature also via holidayTemp.\n if heater.independent_device or heater.is_holiday == 1:\n heater.set_temp = heater_data.get('holidayTemp')\n elif heater.room is not None:\n if heater.room.current_mode == 1:\n heater.set_temp = heater.room.comfort_temp\n elif heater.room.current_mode == 2:\n heater.set_temp = heater.room.sleep_temp\n elif heater.room.current_mode == 3:\n heater.set_temp = heater.room.away_temp\n heater.power_status = heater_data.get('powerStatus')\n heater.tibber_control = heater_data.get('tibberControl')\n heater.open_window = heater_data.get('open_window',\n heater_data.get('open')\n )\n heater.is_heating = heater_data.get('heatStatus',\n heater_data.get('heaterFlag')\n )\n try:\n heater.sub_domain = int(float(heater_data.get('subDomain',\n heater_data.get('subDomainId',\n heater.sub_domain)\n )))\n except ValueError:\n pass", "response": "Set heater values from heater data."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sync_connect(self):\n loop = asyncio.get_event_loop()\n task = loop.create_task(self.connect())\n loop.run_until_complete(task)", "response": "Close the Mill connection."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nclose the Mill connection.", "response": "def sync_close_connection(self):\n \"\"\"Close the Mill connection.\"\"\"\n loop = asyncio.get_event_loop()\n task = loop.create_task(self.close_connection())\n loop.run_until_complete(task)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\nasync def request(self, command, payload, retry=3):\n # pylint: disable=too-many-return-statements\n\n if self._token is None:\n _LOGGER.error(\"No token\")\n return None\n\n _LOGGER.debug(command, payload)\n\n nonce = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16))\n url = API_ENDPOINT_2 + command\n timestamp = int(time.time())\n signature = hashlib.sha1(str(REQUEST_TIMEOUT\n + str(timestamp)\n + nonce\n + self._token).encode(\"utf-8\")).hexdigest()\n\n headers = {\n \"Content-Type\": \"application/x-zc-object\",\n \"Connection\": \"Keep-Alive\",\n \"X-Zc-Major-Domain\": \"seanywell\",\n \"X-Zc-Msg-Name\": \"millService\",\n \"X-Zc-Sub-Domain\": \"milltype\",\n \"X-Zc-Seq-Id\": \"1\",\n \"X-Zc-Version\": \"1\",\n \"X-Zc-Timestamp\": str(timestamp),\n \"X-Zc-Timeout\": REQUEST_TIMEOUT,\n \"X-Zc-Nonce\": nonce,\n \"X-Zc-User-Id\": str(self._user_id),\n \"X-Zc-User-Signature\": signature,\n \"X-Zc-Content-Length\": str(len(payload)),\n }\n try:\n with async_timeout.timeout(self._timeout):\n resp = await self.websession.post(url,\n data=json.dumps(payload),\n headers=headers)\n except asyncio.TimeoutError:\n if retry < 1:\n _LOGGER.error(\"Timed out sending command to Mill: %s\", command)\n return None\n return await self.request(command, payload, retry - 1)\n except aiohttp.ClientError:\n _LOGGER.error(\"Error sending command to Mill: %s\", command, exc_info=True)\n return None\n\n result = await resp.text()\n\n _LOGGER.debug(result)\n\n if not result or result == '{\"errorCode\":0}':\n return None\n\n if 'access token expire' in result or 'invalid signature' in result:\n if retry < 1:\n return None\n if not await self.connect():\n return None\n return await self.request(command, payload, retry - 1)\n\n if '\"error\":\"device offline\"' in result:\n if retry < 1:\n _LOGGER.error(\"Failed to send request, %s\", result)\n return None\n _LOGGER.debug(\"Failed to send request, %s. Retrying...\", result)\n await asyncio.sleep(3)\n return await self.request(command, payload, retry - 1)\n\n if 'errorCode' in result:\n _LOGGER.error(\"Failed to send request, %s\", result)\n return None\n data = json.loads(result)\n return data", "response": "Send a command to Mill."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef sync_request(self, command, payload, retry=2):\n loop = asyncio.get_event_loop()\n task = loop.create_task(self.request(command, payload, retry))\n return loop.run_until_complete(task)", "response": "Send a request to the server and return the response."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nrequesting data for all rooms and heaters.", "response": "async def update_rooms(self):\n \"\"\"Request data.\"\"\"\n homes = await self.get_home_list()\n for home in homes:\n payload = {\"homeId\": home.get(\"homeId\"), \"timeZoneNum\": \"+01:00\"}\n data = await self.request(\"selectRoombyHome\", payload)\n rooms = data.get('roomInfo', [])\n for _room in rooms:\n _id = _room.get('roomId')\n room = self.rooms.get(_id, Room())\n room.room_id = _id\n room.comfort_temp = _room.get(\"comfortTemp\")\n room.away_temp = _room.get(\"awayTemp\")\n room.sleep_temp = _room.get(\"sleepTemp\")\n room.name = _room.get(\"roomName\")\n room.current_mode = _room.get(\"currentMode\")\n room.heat_status = _room.get(\"heatStatus\")\n room.home_name = data.get(\"homeName\")\n room.avg_temp = _room.get(\"avgTemp\")\n\n self.rooms[_id] = room\n payload = {\"roomId\": _room.get(\"roomId\"), \"timeZoneNum\": \"+01:00\"}\n room_device = await self.request(\"selectDevicebyRoom\", payload)\n\n if room_device is None:\n continue\n heater_info = room_device.get('deviceInfo', [])\n for _heater in heater_info:\n _id = _heater.get('deviceId')\n heater = self.heaters.get(_id, Heater())\n heater.device_id = _id\n heater.independent_device = False\n heater.can_change_temp = _heater.get('canChangeTemp')\n heater.name = _heater.get('deviceName')\n heater.room = room\n self.heaters[_id] = heater"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nrequest data from the server.", "response": "def sync_update_rooms(self):\n \"\"\"Request data.\"\"\"\n loop = asyncio.get_event_loop()\n task = loop.create_task(self.update_rooms())\n return loop.run_until_complete(task)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\nasync def set_room_temperatures_by_name(self, room_name, sleep_temp=None,\n comfort_temp=None, away_temp=None):\n \"\"\"Set room temps by name.\"\"\"\n if sleep_temp is None and comfort_temp is None and away_temp is None:\n return\n for room_id, _room in self.rooms.items():\n if _room.name == room_name:\n await self.set_room_temperatures(room_id, sleep_temp,\n comfort_temp, away_temp)\n return\n _LOGGER.error(\"Could not find a room with name %s\", room_name)", "response": "Set room temps by name."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nupdates the list of available HEALPix entries.", "response": "def sync_update_heaters(self):\n \"\"\"Request data.\"\"\"\n loop = asyncio.get_event_loop()\n task = loop.create_task(self.update_heaters())\n loop.run_until_complete(task)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nthrottle update the list of available HEALPix entries.", "response": "async def throttle_update_heaters(self):\n \"\"\"Throttle update device.\"\"\"\n if (self._throttle_time is not None\n and dt.datetime.now() - self._throttle_time < MIN_TIME_BETWEEN_UPDATES):\n return\n self._throttle_time = dt.datetime.now()\n await self.update_heaters()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nthrottle update all devices and rooms.", "response": "async def throttle_update_all_heaters(self):\n \"\"\"Throttle update all devices and rooms.\"\"\"\n if (self._throttle_all_time is not None\n and dt.datetime.now() - self._throttle_all_time\n < MIN_TIME_BETWEEN_UPDATES):\n return\n self._throttle_all_time = dt.datetime.now()\n await self.find_all_heaters()"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntransforming csv as below.", "response": "def exec(self, payload: Log2ReqsAddOnPayload) -> TList[Request]:\n \"\"\"Transform csv as below.\n \"title1\",\"/path1\",\"a=1&b=2\",\"header1=1&header2=2\"\n \"title2\",\"/path2\",\"c=1\"\n \"title3\",\"/path3\",,\"header1=1&header2=2\"\n \"title4\",\"/path4\"\n\n Exception:\n ValueError: If fomat is invalid.\n \"\"\"\n outputs = []\n\n with open(payload.file, encoding=self.config.encoding) as f:\n rs = csv.DictReader(f, ('name', 'path', 'qs', 'headers'), restval={}, dialect=self.config.dialect)\n for r in rs:\n if len(r) > 4:\n raise ValueError\n r['qs'] = urlparser.parse_qs(r['qs'], keep_blank_values=self.config.keep_blank)\n\n # XXX: This is bad implementation but looks simple...\n r['headers'] = urlparser.parse_qs(r['headers'], keep_blank_values=self.config.keep_blank)\n for k, v in r['headers'].items():\n r['headers'][k] = v[0]\n\n outputs.append(r)\n\n return Request.from_dicts(outputs)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nechoing a single nucite object.", "response": "def echo(params, source, delay, strength):\n '''\n Create an echo\n :param params:\n :param source:\n :param delay:\n :param strength:\n :return:\n '''\n source = create_buffer(params, source)\n delay = create_buffer(params, delay)\n strength = create_buffer(params, strength)\n output = source[:]\n for i in range(params.length):\n d = int(i - delay[i])\n if 0 <= d < params.length:\n output[i] += source[d]*strength[i]\n return output"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a copy of the current QuerySet.", "response": "def _clone(self):\n \"\"\"\n Return a copy of the current QuerySet.\n \"\"\"\n clone = self.__class__(self._entity_cls, criteria=self._criteria,\n offset=self._offset, limit=self._limit,\n order_by=self._order_by)\n return clone"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _add_q(self, q_object):\n self._criteria = self._criteria._combine(q_object, q_object.connector)", "response": "Add a Q - object to the current filter."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nlimits number of records", "response": "def limit(self, limit):\n \"\"\"Limit number of records\"\"\"\n clone = self._clone()\n\n if isinstance(limit, int):\n clone._limit = limit\n\n return clone"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfetching results after offset value", "response": "def offset(self, offset):\n \"\"\"Fetch results after `offset` value\"\"\"\n clone = self._clone()\n\n if isinstance(offset, int):\n clone._offset = offset\n\n return clone"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nupdating order_by setting for filter set", "response": "def order_by(self, order_by: Union[set, str]):\n \"\"\"Update order_by setting for filter set\"\"\"\n clone = self._clone()\n if isinstance(order_by, str):\n order_by = {order_by}\n\n clone._order_by = clone._order_by.union(order_by)\n\n return clone"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef all(self):\n logger.debug(f'Query `{self.__class__.__name__}` objects with filters {self}')\n\n # Destroy any cached results\n self._result_cache = None\n\n # Fetch Model class and connected repository from Repository Factory\n model_cls = repo_factory.get_model(self._entity_cls)\n repository = repo_factory.get_repository(self._entity_cls)\n\n # order_by clause must be list of keys\n order_by = self._entity_cls.meta_.order_by if not self._order_by else self._order_by\n\n # Call the read method of the repository\n results = repository.filter(self._criteria, self._offset, self._limit, order_by)\n\n # Convert the returned results to entity and return it\n entity_items = []\n for item in results.items:\n entity = model_cls.to_entity(item)\n entity.state_.mark_retrieved()\n entity_items.append(entity)\n results.items = entity_items\n\n # Cache results\n self._result_cache = results\n\n return results", "response": "Fetch all the items in the QuerySet"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef update(self, *data, **kwargs):\n updated_item_count = 0\n try:\n items = self.all()\n\n for item in items:\n item.update(*data, **kwargs)\n updated_item_count += 1\n except Exception:\n # FIXME Log Exception\n raise\n\n return updated_item_count", "response": "Updates all objects with details given if they match a set of conditions supplied."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef raw(self, query: Any, data: Any = None):\n logger.debug(f'Query `{self.__class__.__name__}` objects with raw query {query}')\n\n # Destroy any cached results\n self._result_cache = None\n\n # Fetch Model class and connected repository from Repository Factory\n model_cls = repo_factory.get_model(self._entity_cls)\n repository = repo_factory.get_repository(self._entity_cls)\n\n try:\n # Call the raw method of the repository\n results = repository.raw(query, data)\n\n # Convert the returned results to entity and return it\n entity_items = []\n for item in results.items:\n entity = model_cls.to_entity(item)\n entity.state_.mark_retrieved()\n entity_items.append(entity)\n results.items = entity_items\n\n # Cache results\n self._result_cache = results\n except Exception:\n # FIXME Log Exception\n raise\n\n return results", "response": "Runs raw query on the database and returns the objects of the specified type."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef delete(self):\n # Fetch Model class and connected repository from Repository Factory\n deleted_item_count = 0\n try:\n items = self.all()\n\n for item in items:\n item.delete()\n deleted_item_count += 1\n except Exception:\n # FIXME Log Exception\n raise\n\n return deleted_item_count", "response": "Deletes matching objects from the Repository and returns the number of objects deleted."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef update_all(self, *args, **kwargs):\n updated_item_count = 0\n\n repository = repo_factory.get_repository(self._entity_cls)\n\n try:\n updated_item_count = repository.update_all(self._criteria, *args, **kwargs)\n except Exception:\n # FIXME Log Exception\n raise\n\n return updated_item_count", "response": "Updates all objects with details given if they match a set of conditions supplied."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef delete_all(self, *args, **kwargs):\n deleted_item_count = 0\n repository = repo_factory.get_repository(self._entity_cls)\n try:\n deleted_item_count = repository.delete_all(self._criteria)\n except Exception:\n # FIXME Log Exception\n raise\n\n return deleted_item_count", "response": "Deletes objects that match a set of conditions supplied."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the total number of records in the database", "response": "def total(self):\n \"\"\"Return the total number of records\"\"\"\n if self._result_cache:\n return self._result_cache.total\n\n return self.all().total"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the first result", "response": "def first(self):\n \"\"\"Return the first result\"\"\"\n if self._result_cache:\n return self._result_cache.first\n\n return self.all().first"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning True if there are more values present", "response": "def has_next(self):\n \"\"\"Return True if there are more values present\"\"\"\n if self._result_cache:\n return self._result_cache.has_next\n\n return self.all().has_next"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn True if there are previous values present", "response": "def has_prev(self):\n \"\"\"Return True if there are previous values present\"\"\"\n if self._result_cache:\n return self._result_cache.has_prev\n\n return self.all().has_prev"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nload an email backend and return an instance of it.", "response": "def get_connection(backend=None, fail_silently=False, **kwargs):\n \"\"\"Load an email backend and return an instance of it.\n If backend is None (default), use settings.EMAIL_BACKEND.\n Both fail_silently and other keyword arguments are used in the\n constructor of the backend.\n \"\"\"\n klass = perform_import(backend or active_config.EMAIL_BACKEND)\n return klass(fail_silently=fail_silently, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsending an email to a list of recipients.", "response": "def send_mail(subject, message, recipient_list, from_email=None,\n fail_silently=False, auth_user=None, auth_password=None,\n connection=None, **kwargs):\n \"\"\"\n Easy wrapper for sending a single message to a recipient list. All members\n of the recipient list will see the other recipients in the 'To' field.\n\n \"\"\"\n connection = connection or get_connection(\n username=auth_user,\n password=auth_password,\n fail_silently=fail_silently,\n )\n mail_message = EmailMessage(subject, message, from_email, recipient_list,\n **kwargs)\n\n return connection.send_messages([mail_message])"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsend a mass mail to a node.", "response": "def send_mass_mail(data_tuple, fail_silently=False, auth_user=None,\n auth_password=None, connection=None):\n \"\"\"\n Given a data_tuple of (subject, message, from_email, recipient_list), send\n each message to each recipient list. Return the number of emails sent.\n If from_email is None, use the DEFAULT_FROM_EMAIL setting.\n\n \"\"\"\n connection = connection or get_connection(\n username=auth_user,\n password=auth_password,\n fail_silently=fail_silently,\n )\n messages = [\n EmailMessage(subject, message, sender, recipient)\n for subject, message, sender, recipient in data_tuple\n ]\n return connection.send_messages(messages)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef build_response(cls, code=Status.SYSTEM_ERROR, errors=None):\n errors = [errors] if not isinstance(errors, list) else errors\n return cls(code, errors)", "response": "Utility method to build a Resource Error object."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef build_not_found(cls, errors=None):\n errors = [errors] if not isinstance(errors, list) else errors\n return cls(Status.NOT_FOUND, errors)", "response": "Utility method to build a HTTP 404 Resource Error response"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef build_system_error(cls, errors=None):\n errors = [errors] if not isinstance(errors, list) else errors\n return cls(Status.SYSTEM_ERROR, errors)", "response": "Utility method to build a HTTP 500 System Error response"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef build_parameters_error(cls, errors=None):\n errors = [errors] if not isinstance(errors, list) else errors\n return cls(Status.PARAMETERS_ERROR, errors)", "response": "Utility method to build a HTTP 400 Parameter Error response"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef build_unprocessable_error(cls, errors=None):\n errors = [errors] if not isinstance(errors, list) else errors\n return cls(Status.UNPROCESSABLE_ENTITY, errors)", "response": "Utility method to build a 422 Parameter Error object"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef linseg(params, start=0, end=1):\n '''\n Signal starts at start value, ramps linearly up to end value\n :param params: buffer parameters, controls length of signal created\n :param start: start value (number)\n :param end: end value (number)\n :return: array of resulting signal\n '''\n return np.linspace(start, end, num=params.length, endpoint=True)", "response": "Returns a linearly - up to end value of the sequence of signal created by this function"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef attack_decay(params, attack, start=0, peak=1):\n '''\n Signal starts at min value, ramps linearly up to max value during the\n attack time, than ramps back down to min value over remaining time\n :param params: buffer parameters, controls length of signal created\n :param attack: attack time, in samples\n :param start: start value (number)\n :param peak: peak value (number)\n :return:\n '''\n builder = GenericEnvelope(params)\n builder.set(start)\n builder.linseg(peak, attack)\n if attack < params.length:\n builder.linseg(start, params.length - attack)\n return builder.build()", "response": "This function is used to create a new object that is a linearly up to max value during the attack time."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nset the current value and optionally maintain it for a period", "response": "def set(self, value, samples=0):\n '''\n Set the current value and optionally maintain it for a period\n :param value: New current value\n :param samples: Add current value for this number of samples (if not zero)\n :return:\n '''\n if self.params.length > self.pos and samples > 0:\n l = min(samples, self.params.length-self.pos)\n self.data[self.pos:self.pos+l] = np.full(l, value, dtype=np.float)\n self.pos += l\n self.latest = value\n return self"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef linseg(self, value, samples):\n '''\n Create a linear section moving from current value to new value over acertain number of\n samples.\n :param value: New value\n :param samples: Length of segment in samples\n :return:\n '''\n if self.params.length > self.pos and samples > 0:\n len = min(samples, self.params.length - self.pos)\n end = value if len == samples else self.latest + (value - self.latest)*len/samples\n self.data[self.pos:self.pos + len] = np.linspace(self.latest, end, num=len, endpoint=False, dtype=np.float)\n self.pos += len\n self.latest = value\n return self", "response": "Create a linear section of the current value to new value over acertain number of samples."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef oauth_flow(s, oauth_url, username=None, password=None, sandbox=False):\n r = s.get(oauth_url)\n if r.status_code >= 300:\n raise RuntimeError(r.text)\n\n params = urlparse.parse_qs(urlparse.urlparse(r.url).query)\n\n data = {\"un\":username,\n \"width\":2560,\n \"height\":1440,\n \"hasRememberUn\":True,\n \"startURL\":params['startURL'],\n \"loginURL\":\"\",\n \"loginType\":6,\n \"useSecure\":True,\n \"local\":\"\",\n \"lt\":\"OAUTH\",\n \"qs\":\"r=https%3A%2F%2Flocalhost%3A8443%2Fsalesforce%2F21\",\n \"locale\":\"\",\n \"oauth_token\":\"\",\n \"oauth_callback\":\"\",\n \"login\":\"\",\n \"serverid\":\"\",\n \"display\":\"popup\",\n \"username\":username,\n \"pw\":password,\n \"Login\":\"\"}\n\n base = \"https://login.salesforce.com\" if not sandbox else \"https://test.salesforce.com\"\n r2 = s.post(base, data)\n m = re.search(\"window.location.href\\s*='(.[^']+)'\", r2.text)\n assert m is not None, \"Couldn't find location.href expression in page %s (Username or password is wrong)\" % r2.url\n\n u3 = \"https://\" + urlparse.urlparse(r2.url).hostname + m.group(1)\n r3 = s.get(u3)\n\n m = re.search(\"window.location.href\\s*='(.[^']+)'\", r3.text)\n\n assert m is not None, \"Couldn't find location.href expression in page %s:\\n%s\" % (r3.url, r3.text)\n\n return m.group(1)", "response": "This function is used to flow the user into the Salesforce website."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nupdates a single case in the database.", "response": "def update_database(adapter, variant_file=None, sv_file=None, family_file=None, family_type='ped', \n skip_case_id=False, gq_treshold=None, case_id=None, max_window = 3000):\n \"\"\"Update a case in the database\n \n Args:\n adapter: Connection to database\n variant_file(str): Path to variant file\n sv_file(str): Path to sv variant file\n family_file(str): Path to family file\n family_type(str): Format of family file\n skip_case_id(bool): If no case information should be added to variants\n gq_treshold(int): If only quality variants should be considered\n case_id(str): If different case id than the one in family file should be used\n max_window(int): Specify the max size for sv windows\n\n Returns:\n nr_inserted(int)\n \"\"\"\n vcf_files = []\n nr_variants = None\n vcf_individuals = None\n if variant_file:\n vcf_info = check_vcf(variant_file)\n nr_variants = vcf_info['nr_variants']\n variant_type = vcf_info['variant_type']\n vcf_files.append(variant_file)\n # Get the indivuduals that are present in vcf file\n vcf_individuals = vcf_info['individuals']\n\n nr_sv_variants = None\n sv_individuals = None\n if sv_file:\n vcf_info = check_vcf(sv_file, 'sv')\n nr_sv_variants = vcf_info['nr_variants']\n vcf_files.append(sv_file)\n sv_individuals = vcf_info['individuals']\n\n # If a gq treshold is used the variants needs to have GQ\n for _vcf_file in vcf_files:\n # Get a cyvcf2.VCF object\n vcf = get_vcf(_vcf_file)\n \n if gq_treshold:\n if not vcf.contains('GQ'):\n LOG.warning('Set gq-treshold to 0 or add info to vcf {0}'.format(_vcf_file))\n raise SyntaxError('GQ is not defined in vcf header')\n\n # Get a ped_parser.Family object from family file\n family = None\n family_id = None\n if family_file:\n with open(family_file, 'r') as family_lines:\n family = get_case(\n family_lines=family_lines, \n family_type=family_type\n )\n family_id = family.family_id\n \n # There has to be a case_id or a family at this stage.\n case_id = case_id or family_id\n\n # Convert infromation to a loqusdb Case object\n case_obj = build_case(\n case=family, \n case_id=case_id,\n vcf_path=variant_file,\n vcf_individuals=vcf_individuals,\n nr_variants=nr_variants,\n vcf_sv_path=sv_file,\n sv_individuals=sv_individuals,\n nr_sv_variants=nr_sv_variants,\n )\n\n existing_case = adapter.case(case_obj)\n if not existing_case:\n raise CaseError(\"Case {} does not exist in database\".format(case_obj['case_id']))\n\n # Update the existing case in database\n case_obj = load_case(\n adapter=adapter,\n case_obj=case_obj,\n update=True,\n )\n \n nr_inserted = 0\n # If case was succesfully added we can store the variants\n for file_type in ['vcf_path','vcf_sv_path']:\n variant_type = 'snv'\n if file_type == 'vcf_sv_path':\n variant_type = 'sv'\n if case_obj.get(file_type) is None:\n continue\n\n vcf_obj = get_vcf(case_obj[file_type])\n try:\n nr_inserted += load_variants( \n adapter=adapter,\n vcf_obj=vcf_obj,\n case_obj=case_obj, \n skip_case_id=skip_case_id,\n gq_treshold=gq_treshold,\n max_window=max_window,\n variant_type=variant_type,\n )\n except Exception as err:\n # If something went wrong do a rollback\n LOG.warning(err)\n delete(\n adapter=adapter,\n case_obj=case_obj,\n update=True,\n existing_case=existing_case,\n )\n raise err\n return nr_inserted"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef make_error_router():\n sink_observer = None\n\n def on_subscribe(observer):\n nonlocal sink_observer\n sink_observer = observer\n\n def dispose():\n nonlocal sink_observer\n sink_observer = None\n\n return dispose\n\n def route_error(obs, convert):\n \"\"\" Handles error raised by obs observable\n\n catches any error raised by obs, maps it to anther object with the\n convert function, and emits in on the error observer.\n\n \"\"\"\n def catch_error(e):\n sink_observer.on_next(convert(e))\n return Observable.empty()\n\n return obs.catch_exception(catch_error)\n\n def catch_or_flat_map(source, error_map, source_map=lambda i: i):\n return source.flat_map(lambda i: route_error(source_map(i), error_map))\n\n return Observable.create(on_subscribe), catch_or_flat_map", "response": "Creates an error router that routes errors to the input observable."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nload the variants of a single case into the database.", "response": "def load(ctx, variant_file, sv_variants, family_file, family_type, skip_case_id, gq_treshold,\n case_id, ensure_index, max_window, check_profile, hard_threshold, soft_threshold):\n \"\"\"Load the variants of a case\n\n A variant is loaded if it is observed in any individual of a case\n If no family file is provided all individuals in vcf file will be considered.\n \"\"\"\n if not (family_file or case_id):\n LOG.warning(\"Please provide a family file or a case id\")\n ctx.abort()\n\n if not (variant_file or sv_variants):\n LOG.warning(\"Please provide a VCF file\")\n ctx.abort()\n\n variant_path = None\n if variant_file:\n variant_path = os.path.abspath(variant_file)\n\n variant_sv_path = None\n if sv_variants:\n variant_sv_path = os.path.abspath(sv_variants)\n\n variant_profile_path = None\n if check_profile:\n variant_profile_path = os.path.abspath(check_profile)\n\n adapter = ctx.obj['adapter']\n\n start_inserting = datetime.now()\n\n try:\n nr_inserted = load_database(\n adapter=adapter,\n variant_file=variant_path,\n sv_file=variant_sv_path,\n family_file=family_file,\n family_type=family_type,\n skip_case_id=skip_case_id,\n case_id=case_id,\n gq_treshold=gq_treshold,\n max_window=max_window,\n profile_file=variant_profile_path,\n hard_threshold=hard_threshold,\n soft_threshold=soft_threshold\n )\n except (SyntaxError, CaseError, IOError) as error:\n LOG.warning(error)\n ctx.abort()\n\n LOG.info(\"Nr variants inserted: %s\", nr_inserted)\n LOG.info(\"Time to insert variants: {0}\".format(\n datetime.now() - start_inserting))\n\n if ensure_index:\n adapter.ensure_indexes()\n else:\n adapter.check_indexes()"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef wipe_db(self):\n logger.warning(\"Wiping the whole database\")\n self.client.drop_database(self.db_name)\n logger.debug(\"Database wiped\")", "response": "Wipe the whole database"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nchecking if the indexes exists", "response": "def check_indexes(self):\n \"\"\"Check if the indexes exists\"\"\"\n for collection_name in INDEXES:\n existing_indexes = self.indexes(collection_name)\n indexes = INDEXES[collection_name]\n for index in indexes:\n index_name = index.document.get('name')\n if not index_name in existing_indexes:\n logger.warning(\"Index {0} missing. Run command `loqusdb index`\".format(index_name))\n return\n logger.info(\"All indexes exists\")"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef ensure_indexes(self):\n for collection_name in INDEXES:\n existing_indexes = self.indexes(collection_name)\n indexes = INDEXES[collection_name]\n for index in indexes:\n index_name = index.document.get('name')\n if index_name in existing_indexes:\n logger.debug(\"Index exists: %s\" % index_name)\n self.db[collection_name].drop_index(index_name)\n logger.info(\"creating indexes for collection {0}: {1}\".format(\n collection_name,\n ', '.join([index.document.get('name') for index in indexes]),\n )\n )\n self.db[collection_name].create_indexes(indexes)", "response": "Update the indexes for the current user"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef format_info(variant, variant_type='snv'):\n \n observations = variant.get('observations',0)\n\n homozygotes = variant.get('homozygote')\n hemizygotes = variant.get('hemizygote')\n\n \n vcf_info = f\"Obs={observations}\"\n if homozygotes:\n vcf_info += f\";Hom={homozygotes}\"\n if hemizygotes:\n vcf_info += f\";Hem={hemizygotes}\"\n\n # This is SV specific\n if variant_type == 'sv':\n end = int((variant['end_left'] + variant['end_right'])/2)\n \n vcf_info += f\";SVTYPE={variant['sv_type']};END={end};SVLEN={variant['length']}\"\n\n return vcf_info", "response": "Format the info field for the SNV variants\n \n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconvert a variant dictionary to a VCF formated string", "response": "def format_variant(variant, variant_type='snv'):\n \"\"\"Convert variant information to a VCF formated string\n \n Args:\n variant(dict)\n variant_type(str)\n \n Returns:\n vcf_variant(str)\n \"\"\"\n chrom = variant.get('chrom')\n pos = variant.get('start')\n \n ref = variant.get('ref')\n alt = variant.get('alt')\n \n if variant_type == 'sv':\n pos = int((variant['pos_left'] + variant['pos_right'])/2)\n ref = 'N'\n alt = f\"<{variant['sv_type']}>\"\n\n info = None\n \n info = format_info(variant, variant_type=variant_type)\n\n variant_line = f\"{chrom}\\t{pos}\\t.\\t{ref}\\t{alt}\\t.\\t.\\t{info}\"\n \n return variant_line"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate an objective function and its derivative for M given W and X.", "response": "def _create_m_objective(w, X):\n \"\"\"\n Creates an objective function and its derivative for M, given W and X\n\n Args:\n w (array): clusters x cells\n X (array): genes x cells\n \"\"\"\n clusters, cells = w.shape\n genes = X.shape[0]\n w_sum = w.sum(1)\n def objective(m):\n m = m.reshape((X.shape[0], w.shape[0]))\n d = m.dot(w)+eps\n temp = X/d\n w2 = w.dot(temp.T)\n deriv = w_sum - w2.T\n return np.sum(d - X*np.log(d))/genes, deriv.flatten()/genes\n return objective"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates a weight initialization matrix from Poisson clustering assignments.", "response": "def initialize_from_assignments(assignments, k, max_assign_weight=0.75):\n \"\"\"\n Creates a weight initialization matrix from Poisson clustering assignments.\n\n Args:\n assignments (array): 1D array of integers, of length cells\n k (int): number of states/clusters\n max_assign_weight (float, optional): between 0 and 1 - how much weight to assign to the highest cluster. Default: 0.75\n\n Returns:\n init_W (array): k x cells\n \"\"\"\n cells = len(assignments)\n init_W = np.zeros((k, cells))\n for i, a in enumerate(assignments):\n # entirely arbitrary... maybe it would be better to scale\n # the weights based on k?\n init_W[a, i] = max_assign_weight\n for a2 in range(k):\n if a2!=a:\n init_W[a2, i] = (1-max_assign_weight)/(k-1)\n return init_W/init_W.sum(0)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef initialize_means(data, clusters, k):\n init_w = np.zeros((data.shape[0], k))\n if sparse.issparse(data):\n for i in range(k):\n if data[:,clusters==i].shape[1]==0:\n point = np.random.randint(0, data.shape[1])\n init_w[:,i] = data[:,point].toarray().flatten()\n else:\n # memory usage might be a problem here?\n init_w[:,i] = np.array(data[:,clusters==i].mean(1)).flatten() + eps\n else:\n for i in range(k):\n if data[:,clusters==i].shape[1]==0:\n point = np.random.randint(0, data.shape[1])\n init_w[:,i] = data[:,point].flatten()\n else:\n init_w[:,i] = data[:,clusters==i].mean(1) + eps\n return init_w", "response": "Initializes the M matrix given the data and a set of cluster labels."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef initialize_weights_nn(data, means, lognorm=True):\n # TODO\n genes, cells = data.shape\n k = means.shape[1]\n if lognorm:\n data = log1p(cell_normalize(data))\n for i in range(cells):\n for j in range(k):\n pass", "response": "Initializes the weights with a nearest - neighbor approach using the means."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngenerates initial means and weights for state estimation.", "response": "def initialize_means_weights(data, clusters, init_means=None, init_weights=None, initialization='tsvd', max_assign_weight=0.75):\n \"\"\"\n Generates initial means and weights for state estimation.\n \"\"\"\n genes, cells = data.shape\n if init_means is None:\n if init_weights is not None:\n if len(init_weights.shape)==1:\n means = initialize_means(data, init_weights, clusters)\n else:\n means = initialize_means(data, init_weights.argmax(0),\n clusters, max_assign_weight=max_assign_weight)\n elif initialization=='cluster':\n assignments, means = poisson_cluster(data, clusters)\n if init_weights is None:\n init_weights = initialize_from_assignments(assignments, clusters,\n max_assign_weight=max_assign_weight)\n elif initialization=='kmpp':\n means, assignments = kmeans_pp(data, clusters)\n elif initialization=='km':\n km = KMeans(clusters)\n assignments = km.fit_predict(log1p(cell_normalize(data)).T)\n init_weights = initialize_from_assignments(assignments, clusters,\n max_assign_weight)\n means = initialize_means(data, assignments, clusters)\n elif initialization=='tsvd':\n n_components = min(50, genes-1)\n #tsvd = TruncatedSVD(min(50, genes-1))\n km = KMeans(clusters)\n # remove dependence on sklearn tsvd b/c it has a bug that\n # prevents it from working properly on long inputs \n # if num elements > 2**31\n #data_reduced = tsvd.fit_transform(log1p(cell_normalize(data)).T)\n U, Sigma, VT = randomized_svd(log1p(cell_normalize(data)).T,\n n_components)\n data_reduced = U*Sigma\n assignments = km.fit_predict(data_reduced)\n init_weights = initialize_from_assignments(assignments, clusters,\n max_assign_weight)\n means = initialize_means(data, assignments, clusters)\n elif initialization == 'random' or initialization == 'rand':\n # choose k random cells and set means to those\n selected_cells = np.random.choice(range(cells), size=clusters,\n replace=False)\n means = data[:, selected_cells]\n if sparse.issparse(means):\n means = means.toarray()\n else:\n means = init_means.copy()\n means = means.astype(float)\n if init_weights is None:\n if init_means is not None:\n if initialization == 'cluster':\n assignments, means = poisson_cluster(data, clusters,\n init=init_means, max_iters=1)\n w_init = initialize_from_assignments(assignments, clusters,\n max_assign_weight)\n elif initialization == 'km':\n km = KMeans(clusters, init=log1p(init_means.T), max_iter=1)\n assignments = km.fit_predict(log1p(cell_normalize(data)).T)\n w_init = initialize_from_assignments(assignments, clusters,\n max_assign_weight)\n else:\n w_init = np.random.random((clusters, cells))\n w_init = w_init/w_init.sum(0)\n else:\n w_init = np.random.random((clusters, cells))\n w_init = w_init/w_init.sum(0)\n else:\n if len(init_weights.shape)==1:\n init_weights = initialize_from_assignments(init_weights, clusters,\n max_assign_weight)\n w_init = init_weights.copy()\n return means, w_init"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nuse a Poisson Covex Mixture model to estimate cell states and cell mixing weights.", "response": "def poisson_estimate_state(data, clusters, init_means=None, init_weights=None, method='NoLips', max_iters=30, tol=0, disp=False, inner_max_iters=100, normalize=True, initialization='tsvd', parallel=True, threads=4, max_assign_weight=0.75, run_w_first=True, constrain_w=False, regularization=0.0, write_progress_file=None, **kwargs):\n \"\"\"\n Uses a Poisson Covex Mixture model to estimate cell states and\n cell state mixing weights.\n\n To lower computational costs, use a sparse matrix, set disp to False, and set tol to 0.\n\n Args:\n data (array): genes x cells array or sparse matrix.\n clusters (int): number of mixture components\n init_means (array, optional): initial centers - genes x clusters. Default: from Poisson kmeans\n init_weights (array, optional): initial weights - clusters x cells, or assignments as produced by clustering. Default: from Poisson kmeans\n method (str, optional): optimization method. Current options are 'NoLips' and 'L-BFGS-B'. Default: 'NoLips'.\n max_iters (int, optional): maximum number of iterations. Default: 30\n tol (float, optional): if both M and W change by less than tol (RMSE), then the iteration is stopped. Default: 1e-10\n disp (bool, optional): whether or not to display optimization progress. Default: False\n inner_max_iters (int, optional): Number of iterations to run in the optimization subroutine for M and W. Default: 100\n normalize (bool, optional): True if the resulting W should sum to 1 for each cell. Default: True.\n initialization (str, optional): If initial means and weights are not provided, this describes how they are initialized. Options: 'cluster' (poisson cluster for means and weights), 'kmpp' (kmeans++ for means, random weights), 'km' (regular k-means), 'tsvd' (tsvd(50) + k-means). Default: tsvd.\n parallel (bool, optional): Whether to use parallel updates (sparse NoLips only). Default: True\n threads (int, optional): How many threads to use in the parallel computation. Default: 4\n max_assign_weight (float, optional): If using a clustering-based initialization, how much weight to assign to the max weight cluster. Default: 0.75\n run_w_first (bool, optional): Whether or not to optimize W first (if false, M will be optimized first). Default: True\n constrain_w (bool, optional): If True, then W is normalized after every iteration. Default: False\n regularization (float, optional): Regularization coefficient for M and W. Default: 0 (no regularization).\n write_progress_file (str, optional): filename to write progress updates to.\n\n Returns:\n M (array): genes x clusters - state means\n W (array): clusters x cells - state mixing components for each cell\n ll (float): final log-likelihood\n \"\"\"\n genes, cells = data.shape\n means, w_init = initialize_means_weights(data, clusters, init_means, init_weights, initialization, max_assign_weight)\n X = data.astype(float)\n XT = X.T\n is_sparse = False\n if sparse.issparse(X):\n is_sparse = True\n update_fn = sparse_nolips_update_w\n # convert to csc\n X = sparse.csc_matrix(X)\n XT = sparse.csc_matrix(XT)\n if parallel:\n update_fn = parallel_sparse_nolips_update_w\n Xsum = np.asarray(X.sum(0)).flatten()\n Xsum_m = np.asarray(X.sum(1)).flatten()\n # L-BFGS-B won't work right now for sparse matrices\n method = 'NoLips'\n objective_fn = _call_sparse_obj\n else:\n objective_fn = objective\n update_fn = nolips_update_w\n Xsum = X.sum(0)\n Xsum_m = X.sum(1)\n # If method is NoLips, converting to a sparse matrix\n # will always improve the performance (?) and never lower accuracy...\n if method == 'NoLips':\n is_sparse = True\n X = sparse.csc_matrix(X)\n XT = sparse.csc_matrix(XT)\n update_fn = sparse_nolips_update_w\n if parallel:\n update_fn = parallel_sparse_nolips_update_w\n objective_fn = _call_sparse_obj\n w_new = w_init\n for i in range(max_iters):\n if disp:\n print('iter: {0}'.format(i))\n if run_w_first:\n # step 1: given M, estimate W\n w_new = _estimate_w(X, w_new, means, Xsum, update_fn, objective_fn, is_sparse, parallel, threads, method, tol, disp, inner_max_iters, 'W', regularization)\n if constrain_w:\n w_new = w_new/w_new.sum(0)\n if disp:\n w_ll = objective_fn(X, means, w_new)\n print('Finished updating W. Objective value: {0}'.format(w_ll))\n # step 2: given W, update M\n means = _estimate_w(XT, means.T, w_new.T, Xsum_m, update_fn, objective_fn, is_sparse, parallel, threads, method, tol, disp, inner_max_iters, 'M', regularization)\n means = means.T\n if disp:\n w_ll = objective_fn(X, means, w_new)\n print('Finished updating M. Objective value: {0}'.format(w_ll))\n else:\n # step 1: given W, update M\n means = _estimate_w(XT, means.T, w_new.T, Xsum_m, update_fn, objective_fn, is_sparse, parallel, threads, method, tol, disp, inner_max_iters, 'M', regularization)\n means = means.T\n if disp:\n w_ll = objective_fn(X, means, w_new)\n print('Finished updating M. Objective value: {0}'.format(w_ll))\n # step 2: given M, estimate W\n w_new = _estimate_w(X, w_new, means, Xsum, update_fn, objective_fn, is_sparse, parallel, threads, method, tol, disp, inner_max_iters, 'W', regularization)\n if constrain_w:\n w_new = w_new/w_new.sum(0)\n if disp:\n w_ll = objective_fn(X, means, w_new)\n print('Finished updating W. Objective value: {0}'.format(w_ll))\n # write progress to progress file\n if write_progress_file is not None:\n progress = open(write_progress_file, 'w')\n progress.write(str(i))\n progress.close()\n if normalize:\n w_new = w_new/w_new.sum(0)\n m_ll = objective_fn(X, means, w_new)\n return means, w_new, m_ll"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef update_m(data, old_M, old_W, selected_genes, disp=False, inner_max_iters=100, parallel=True, threads=4, write_progress_file=None, tol=0.0, regularization=0.0, **kwargs):\n genes, cells = data.shape\n k = old_M.shape[1]\n non_selected_genes = [x for x in range(genes) if x not in set(selected_genes)]\n # 1. initialize new M\n new_M = np.zeros((genes, k))\n new_M[selected_genes, :] = old_M\n # TODO: how to initialize rest of genes?\n # data*w?\n if disp:\n print('computing initial guess for M by data*W.T')\n new_M_non_selected = data[non_selected_genes, :] * sparse.csc_matrix(old_W.T)\n new_M[non_selected_genes, :] = new_M_non_selected.toarray()\n X = data.astype(float)\n XT = X.T\n is_sparse = False\n if sparse.issparse(X):\n is_sparse = True\n update_fn = sparse_nolips_update_w\n # convert to csc\n X = sparse.csc_matrix(X)\n XT = sparse.csc_matrix(XT)\n if parallel:\n update_fn = parallel_sparse_nolips_update_w\n Xsum = np.asarray(X.sum(0)).flatten()\n Xsum_m = np.asarray(X.sum(1)).flatten()\n # L-BFGS-B won't work right now for sparse matrices\n method = 'NoLips'\n objective_fn = _call_sparse_obj\n else:\n objective_fn = objective\n update_fn = nolips_update_w\n Xsum = X.sum(0)\n Xsum_m = X.sum(1)\n # If method is NoLips, converting to a sparse matrix\n # will always improve the performance (?) and never lower accuracy...\n # will almost always improve performance?\n # if sparsity is below 40%?\n if method == 'NoLips':\n is_sparse = True\n X = sparse.csc_matrix(X)\n XT = sparse.csc_matrix(XT)\n update_fn = sparse_nolips_update_w\n if parallel:\n update_fn = parallel_sparse_nolips_update_w\n objective_fn = _call_sparse_obj\n if disp:\n print('starting estimating M')\n new_M = _estimate_w(XT, new_M.T, old_W.T, Xsum_m, update_fn, objective_fn, is_sparse, parallel, threads, method, tol, disp, inner_max_iters, 'M', regularization)\n if write_progress_file is not None:\n progress = open(write_progress_file, 'w')\n progress.write('0')\n progress.close()\n return new_M.T", "response": "This function updates the M matrix with the given data and a subset of genes."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef setup(hass, config):\n from pyvesync.vesync import VeSync\n\n conf = config[DOMAIN]\n\n manager = VeSync(conf.get(CONF_USERNAME), conf.get(CONF_PASSWORD),\n time_zone=conf.get(CONF_TIME_ZONE))\n\n if not manager.login():\n _LOGGER.error(\"Unable to login to VeSync\")\n return\n\n manager.update()\n\n hass.data[DOMAIN] = {\n 'manager': manager\n }\n\n discovery.load_platform(hass, 'switch', DOMAIN, {}, config)\n\n return True", "response": "Setup the VeSync component."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef state_estimation_ensemble(data, k, n_runs=10, M_list=[], **se_params):\n if len(M_list)==0:\n M_list = []\n for i in range(n_runs):\n M, W, ll = poisson_estimate_state(data, k, **se_params)\n M_list.append(M)\n M_stacked = np.hstack(M_list)\n M_new, W_new, ll = poisson_estimate_state(M_stacked, k, **se_params)\n W_new = np.dot(data.T, M_new)\n W_new = W_new/W_new.sum(0)\n return M_new, W_new, ll", "response": "Runs an ensemble method on the list of M arrays from state estimation\n M_list is a list of M arrays from state estimation\n se_params is a dictionary of parameters to be used in the state estimation algorithm."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrun an ensemble method on the list of NMF W matrices...", "response": "def nmf_ensemble(data, k, n_runs=10, W_list=[], **nmf_params):\n \"\"\"\n Runs an ensemble method on the list of NMF W matrices...\n\n Args:\n data: genes x cells array (should be log + cell-normalized)\n k: number of classes\n n_runs (optional): number of random initializations of state estimation\n M_list (optional): list of M arrays from state estimation\n se_params (optional): optional poisson_estimate_state params\n\n Returns:\n W_new\n H_new\n \"\"\"\n nmf = NMF(k)\n if len(W_list)==0:\n W_list = []\n for i in range(n_runs):\n W = nmf.fit_transform(data)\n W_list.append(W)\n W_stacked = np.hstack(W_list)\n nmf_w = nmf.fit_transform(W_stacked)\n nmf_h = nmf.components_\n H_new = data.T.dot(nmf_w).T\n nmf2 = NMF(k, init='custom')\n nmf_w = nmf2.fit_transform(data, W=nmf_w, H=H_new)\n H_new = nmf2.components_\n #W_new = W_new/W_new.sum(0)\n # alternatively, use nmf_w and h_new as initializations for another NMF round?\n return nmf_w, H_new"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrun K - fold ensemble topic modeling", "response": "def nmf_kfold(data, k, n_runs=10, **nmf_params):\n \"\"\"\n Runs K-fold ensemble topic modeling (Belford et al. 2017)\n \"\"\"\n # TODO\n nmf = NMF(k)\n W_list = []\n kf = KFold(n_splits=n_runs, shuffle=True)\n # TODO: randomly divide data into n_runs folds\n for train_index, test_index in kf.split(data.T):\n W = nmf.fit_transform(data[:,train_index])\n W_list.append(W)\n W_stacked = np.hstack(W_list)\n nmf_w = nmf.fit_transform(W_stacked)\n nmf_h = nmf.components_\n H_new = data.T.dot(nmf_w).T\n nmf2 = NMF(k, init='custom')\n nmf_w = nmf2.fit_transform(data, W=nmf_w, H=H_new)\n H_new = nmf2.components_\n #W_new = W_new/W_new.sum(0)\n return nmf_w, H_new"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ninitialize the NMF model from the given data.", "response": "def nmf_init(data, clusters, k, init='enhanced'):\n \"\"\"\n runs enhanced NMF initialization from clusterings (Gong 2013)\n\n There are 3 options for init:\n enhanced - uses EIn-NMF from Gong 2013\n basic - uses means for W, assigns H such that the chosen cluster for a given cell has value 0.75 and all others have 0.25/(k-1).\n nmf - uses means for W, and assigns H using the NMF objective while holding W constant.\n \"\"\"\n init_w = np.zeros((data.shape[0], k))\n if sparse.issparse(data):\n for i in range(k):\n if data[:,clusters==i].shape[1]==0:\n point = np.random.randint(0, data.shape[1])\n init_w[:,i] = data[:,point].toarray().flatten()\n else:\n init_w[:,i] = np.array(data[:,clusters==i].mean(1)).flatten()\n else:\n for i in range(k):\n if data[:,clusters==i].shape[1]==0:\n point = np.random.randint(0, data.shape[1])\n init_w[:,i] = data[:,point].flatten()\n else:\n init_w[:,i] = data[:,clusters==i].mean(1)\n init_h = np.zeros((k, data.shape[1]))\n if init == 'enhanced':\n distances = np.zeros((k, data.shape[1]))\n for i in range(k):\n for j in range(data.shape[1]):\n distances[i,j] = np.sqrt(((data[:,j] - init_w[:,i])**2).sum())\n for i in range(k):\n for j in range(data.shape[1]):\n init_h[i,j] = 1/((distances[:,j]/distances[i,j])**(-2)).sum()\n elif init == 'basic':\n init_h = initialize_from_assignments(clusters, k)\n elif init == 'nmf':\n init_h_, _, n_iter = non_negative_factorization(data.T, n_components=k, init='custom', update_H=False, H=init_w.T)\n init_h = init_h_.T\n return init_w, init_h"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef nmf_tsne(data, k, n_runs=10, init='enhanced', **params):\n clusters = []\n nmf = NMF(k)\n tsne = TSNE(2)\n km = KMeans(k)\n for i in range(n_runs):\n w = nmf.fit_transform(data)\n h = nmf.components_\n tsne_wh = tsne.fit_transform(w.dot(h).T)\n clust = km.fit_predict(tsne_wh)\n clusters.append(clust)\n clusterings = np.vstack(clusters)\n consensus = CE.cluster_ensembles(clusterings, verbose=False, N_clusters_max=k)\n nmf_new = NMF(k, init='custom')\n # TODO: find an initialization for the consensus W and H\n init_w, init_h = nmf_init(data, consensus, k, init)\n W = nmf_new.fit_transform(data, W=init_w, H=init_h)\n H = nmf_new.components_\n return W, H", "response": "runs tsne - NMF and tsne + k on all WH matrices and returns the W and H of the NMF run"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ninitializes state estimation using a consensus of several SVD - k - means rounds and fast clustering algorithm.", "response": "def poisson_se_multiclust(data, k, n_runs=10, **se_params):\n \"\"\"\n Initializes state estimation using a consensus of several\n fast clustering/dimensionality reduction algorithms.\n\n It does a consensus of 8 truncated SVD - k-means rounds, and uses the\n basic nmf_init to create starting points.\n \"\"\"\n clusters = []\n norm_data = cell_normalize(data)\n if sparse.issparse(data):\n log_data = data.log1p()\n log_norm = norm_data.log1p()\n else:\n log_data = np.log1p(data)\n log_norm = np.log1p(norm_data)\n tsvd_50 = TruncatedSVD(50)\n tsvd_k = TruncatedSVD(k)\n km = KMeans(k)\n tsvd1 = tsvd_50.fit_transform(data.T)\n tsvd2 = tsvd_k.fit_transform(data.T)\n tsvd3 = tsvd_50.fit_transform(log_data.T)\n tsvd4 = tsvd_k.fit_transform(log_data.T)\n tsvd5 = tsvd_50.fit_transform(norm_data.T)\n tsvd6 = tsvd_k.fit_transform(norm_data.T)\n tsvd7 = tsvd_50.fit_transform(log_norm.T)\n tsvd8 = tsvd_k.fit_transform(log_norm.T)\n tsvd_results = [tsvd1, tsvd2, tsvd3, tsvd4, tsvd5, tsvd6, tsvd7, tsvd8]\n clusters = []\n for t in tsvd_results:\n clust = km.fit_predict(t)\n clusters.append(clust)\n clusterings = np.vstack(clusters)\n consensus = CE.cluster_ensembles(clusterings, verbose=False, N_clusters_max=k)\n init_m, init_w = nmf_init(data, consensus, k, 'basic')\n M, W, ll = poisson_estimate_state(data, k, init_means=init_m, init_weights=init_w, **se_params)\n return M, W, ll"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef poisson_consensus_se(data, k, n_runs=10, **se_params):\n clusters = []\n for i in range(n_runs):\n assignments, means = poisson_cluster(data, k)\n clusters.append(assignments)\n clusterings = np.vstack(clusters)\n consensus = CE.cluster_ensembles(clusterings, verbose=False, N_clusters_max=k)\n init_m, init_w = nmf_init(data, consensus, k, 'basic')\n M, W, ll = poisson_estimate_state(data, k, init_means=init_m, init_weights=init_w, **se_params)\n return M, W, ll", "response": "Initializes Poisson State Estimation using a consensus Poisson clustering."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrun L - EnsNMF on the data.", "response": "def lensNMF(data, k, ks=1):\n \"\"\"\n Runs L-EnsNMF on the data. (Suh et al. 2016)\n \"\"\"\n # TODO: why is this not working\n n_rounds = k/ks\n R_i = data.copy()\n nmf = NMF(ks)\n nmf2 = NMF(ks, init='custom')\n w_is = []\n h_is = []\n rs = []\n w_i = np.zeros((data.shape[0], ks))\n h_i = np.zeros((ks, data.shape[1]))\n for i in range(n_rounds):\n R_i = R_i - w_i.dot(h_i)\n R_i[R_i < 0] = 0\n \"\"\"\n P_r = R_i.sum(1)/R_i.sum()\n print(P_r.shape)\n P_c = R_i.sum(0)/R_i.sum()\n print(P_c.shape)\n row_choice = np.random.choice(range(len(P_r)), p=P_r)\n print(row_choice)\n col_choice = np.random.choice(range(len(P_c)), p=P_c)\n print(col_choice)\n D_r = cosine_similarity(data[row_choice:row_choice+1,:], data)\n D_c = cosine_similarity(data[:,col_choice:col_choice+1].T, data.T)\n D_r = np.diag(D_r.flatten())\n D_c = np.diag(D_c.flatten())\n R_L = D_r.dot(R_i).dot(D_c)\n w_i = nmf.fit_transform(R_L)\n \"\"\"\n w_i = nmf.fit_transform(R_i)\n h_i = nmf.components_\n #nmf2.fit_transform(R_i, W=w_i, H=nmf.components_)\n #h_i = nmf2.components_\n #h_i[h_i < 0] = 0\n w_is.append(w_i)\n h_is.append(h_i)\n rs.append(R_i)\n return np.hstack(w_is), np.vstack(h_is), rs"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_case(family_lines, family_type='ped', vcf_path=None):\n family = None\n LOG.info(\"Parsing family information\")\n \n family_parser = FamilyParser(family_lines, family_type)\n \n families = list(family_parser.families.keys())\n \n LOG.info(\"Found families {0}\".format(', '.join(families)))\n \n if len(families) > 1:\n raise CaseError(\"Only one family per load can be used\")\n \n family = family_parser.families[families[0]]\n \n return family", "response": "Return ped_parser case from a family file"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef update_case(case_obj, existing_case):\n variant_nrs = ['nr_variants', 'nr_sv_variants']\n individuals = [('individuals','_inds'), ('sv_individuals','_sv_inds')]\n \n updated_case = deepcopy(existing_case)\n \n for i,file_name in enumerate(['vcf_path','vcf_sv_path']):\n variant_type = 'snv'\n if file_name == 'vcf_sv_path':\n variant_type = 'sv'\n if case_obj.get(file_name):\n if updated_case.get(file_name):\n LOG.warning(\"VCF of type %s already exists in case\", variant_type)\n raise CaseError(\"Can not replace VCF in existing case\")\n else:\n updated_case[file_name] = case_obj[file_name]\n updated_case[variant_nrs[i]] = case_obj[variant_nrs[i]]\n updated_case[individuals[i][0]] = case_obj[individuals[i][0]]\n updated_case[individuals[i][1]] = case_obj[individuals[i][1]]\n\n return updated_case", "response": "Update an existing case with the contents of the VCF files."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nprint the sequence of states in the SEDML file.", "response": "def main (args):\n \"\"\"Usage: print_sedml input-filename\n \"\"\"\n if len(args) != 2:\n print(main.__doc__)\n sys.exit(1)\n\n doc = libsedml.readSedML(args[1]);\n if ( doc.getErrorLog().getNumFailsWithSeverity(libsedml.LIBSEDML_SEV_ERROR) > 0): \n print doc.getErrorLog().toString();\n sys.exit(2); \n \n print 'The document has {0}\" simulation(s).'.format(doc.getNumSimulations());\n for i in range(0, doc.getNumSimulations()):\n current = doc.getSimulation(i);\n if (current.getTypeCode() == libsedml.SEDML_SIMULATION_UNIFORMTIMECOURSE):\n tc = current;\n kisaoid=\"none\"\n if tc.isSetAlgorithm():\n\t\tkisaoid=tc.getAlgorithm().getKisaoID()\n print \"\\tTimecourse id=\", tc.getId(),\" start=\",tc.getOutputStartTime(),\" end=\",tc.getOutputEndTime(),\" numPoints=\",tc.getNumberOfPoints(),\" kisao=\",kisaoid,\"\\n\";\n else:\n\t print \"\\tUncountered unknown simulation. \",current.getId(),\"\\n\";\n \n print \"\\n\"\n print \"The document has \",doc.getNumModels() , \" model(s).\" , \"\\n\";\n for i in range(0,doc.getNumModels()):\n current = doc.getModel(i);\n print \"\\tModel id=\" , current.getId() , \" language=\" , current.getLanguage() , \" source=\" , current.getSource() , \" numChanges=\" , current.getNumChanges() , \"\\n\";\n \n print \"\\n\";\n print \"The document has \" , doc.getNumTasks() , \" task(s).\" , \"\\n\";\n for i in range(0,doc.getNumTasks()):\n current = doc.getTask(i);\n print \"\\tTask id=\" , current.getId() , \" model=\" , current.getModelReference() , \" sim=\" , current.getSimulationReference() , \"\\n\";\n\n print \"\\n\";\n print \"The document has \" , doc.getNumDataGenerators() , \" datagenerators(s).\" , \"\\n\";\n for i in range( 0, doc.getNumDataGenerators()):\n current = doc.getDataGenerator(i);\n print \"\\tDG id=\" , current.getId() , \" math=\" , libsedml.formulaToString(current.getMath()) , \"\\n\";\n \n print \"\\n\";\n print \"The document has \" , doc.getNumOutputs() , \" output(s).\" , \"\\n\";\n for i in range (0, doc.getNumOutputs()):\n current = doc.getOutput(i);\n tc = current.getTypeCode();\n if tc == libsedml.SEDML_OUTPUT_REPORT:\n r = (current);\n print \"\\tReport id=\" , current.getId() , \" numDataSets=\" , r.getNumDataSets() , \"\\n\";\n elif tc == libsedml.SEDML_OUTPUT_PLOT2D:\n p = (current);\n print \"\\tPlot2d id=\" , current.getId() , \" numCurves=\" , p.getNumCurves() , \"\\n\";\n elif tc == libsedml.SEDML_OUTPUT_PLOT3D:\n p = (current);\n print \"\\tPlot3d id=\" , current.getId() , \" numSurfaces=\" , p.getNumSurfaces() , \"\\n\";\n else: \n print \"\\tEncountered unknown output \" , current.getId() , \"\\n\";"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\nasync def login(self, email: str, password: str) -> bool:\n login_resp = await self._request(\n 'post',\n API_URL_USER,\n json={\n 'version': '1.0',\n 'method': 'Signin',\n 'param': {\n 'Email': email,\n 'Password': password,\n 'CaptchaCode': ''\n },\n 'sourcetype': 0\n })\n\n _LOGGER.debug('Login response: %s', login_resp)\n\n if login_resp.get('Code') != 0:\n return False\n\n self.account_id = login_resp['Json']['gid']\n\n return True", "response": "Login to the profile."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\nasync def packages(\n self, package_state: Union[int, str] = '',\n show_archived: bool = False) -> list:\n \"\"\"Get the list of packages associated with the account.\"\"\"\n packages_resp = await self._request(\n 'post',\n API_URL_BUYER,\n json={\n 'version': '1.0',\n 'method': 'GetTrackInfoList',\n 'param': {\n 'IsArchived': show_archived,\n 'Item': '',\n 'Page': 1,\n 'PerPage': 40,\n 'PackageState': package_state,\n 'Sequence': '0'\n },\n 'sourcetype': 0\n })\n\n _LOGGER.debug('Packages response: %s', packages_resp)\n\n packages = []\n for package in packages_resp.get('Json', []):\n last_event = package.get('FLastEvent')\n if last_event:\n event = json.loads(last_event)\n else:\n event = {}\n\n kwargs = {\n 'destination_country': package.get('FSecondCountry', 0),\n 'friendly_name': package.get('FRemark'),\n 'info_text': event.get('z'),\n 'location': event.get('c'),\n 'origin_country': package.get('FFirstCountry', 0),\n 'package_type': package.get('FTrackStateType', 0),\n 'status': package.get('FPackageState', 0)\n }\n packages.append(Package(package['FTrackNo'], **kwargs))\n return packages", "response": "Get the list of packages associated with the account."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\nasync def summary(self, show_archived: bool = False) -> dict:\n summary_resp = await self._request(\n 'post',\n API_URL_BUYER,\n json={\n 'version': '1.0',\n 'method': 'GetIndexData',\n 'param': {\n 'IsArchived': show_archived\n },\n 'sourcetype': 0\n })\n\n _LOGGER.debug('Summary response: %s', summary_resp)\n\n results = {}\n for kind in summary_resp.get('Json', {}).get('eitem', []):\n results[PACKAGE_STATUS_MAP[kind['e']]] = kind['ec']\n return results", "response": "Get a quick summary of how many packages are in an account."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting up the VeSync switch platform.", "response": "def setup_platform(hass, config, add_entities, discovery_info=None):\n \"\"\"Set up the VeSync switch platform.\"\"\"\n if discovery_info is None:\n return\n\n switches = []\n\n manager = hass.data[DOMAIN]['manager']\n\n if manager.outlets is not None and manager.outlets:\n if len(manager.outlets) == 1:\n count_string = 'switch'\n else:\n count_string = 'switches'\n\n _LOGGER.info(\"Discovered %d VeSync %s\",\n len(manager.outlets), count_string)\n\n if len(manager.outlets) > 1:\n for switch in manager.outlets:\n switch._energy_update_interval = ENERGY_UPDATE_INT\n switches.append(VeSyncSwitchHA(switch))\n _LOGGER.info(\"Added a VeSync switch named '%s'\",\n switch.device_name)\n else:\n switches.append(VeSyncSwitchHA(manager.outlets))\n else:\n _LOGGER.info(\"No VeSync switches found\")\n\n add_entities(switches)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the state attributes of the device.", "response": "def device_state_attributes(self):\n \"\"\"Return the state attributes of the device.\"\"\"\n attr = {}\n attr['active_time'] = self.smartplug.active_time\n attr['voltage'] = self.smartplug.voltage\n attr['active_time'] = self.smartplug.active_time\n attr['weekly_energy_total'] = self.smartplug.weekly_energy_total\n attr['monthly_energy_total'] = self.smartplug.monthly_energy_total\n attr['yearly_energy_total'] = self.smartplug.yearly_energy_total\n return attr"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts a variant to a proper update", "response": "def _get_update(self, variant):\n \"\"\"Convert a variant to a proper update\n \n Args:\n variant(dict)\n \n Returns:\n update(dict)\n \"\"\"\n update = {\n '$inc': {\n 'homozygote': variant.get('homozygote', 0),\n 'hemizygote': variant.get('hemizygote', 0),\n 'observations': 1\n },\n '$set': {\n 'chrom': variant.get('chrom'),\n 'start': variant.get('pos'),\n 'end': variant.get('end'),\n 'ref': variant.get('ref'),\n 'alt': variant.get('alt'),\n }\n }\n if variant.get('case_id'):\n update['$push'] = {\n 'families': {\n '$each': [variant.get('case_id')],\n '$slice': -50\n }\n }\n return update"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef add_variant(self, variant):\n LOG.debug(\"Upserting variant: {0}\".format(variant.get('_id')))\n \n update = self._get_update(variant)\n \n message = self.db.variant.update_one(\n {'_id': variant['_id']},\n update,\n upsert=True\n )\n if message.modified_count == 1:\n LOG.debug(\"Variant %s was updated\", variant.get('_id'))\n else:\n LOG.debug(\"Variant was added to database for first time\")\n return", "response": "Adds a variant to the variant collection"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_variants(self, variants):\n \n operations = []\n nr_inserted = 0\n for i,variant in enumerate(variants, 1):\n # We need to check if there was any information returned\n # The variant could be excluded based on low gq or if no individiual was called\n # in the particular case\n if not variant:\n continue\n nr_inserted += 1\n update = self._get_update(variant)\n operations.append(\n UpdateOne(\n {'_id': variant['_id']},\n update,\n upsert=True\n )\n )\n if i % 10000 == 0:\n self.db.variant.bulk_write(operations, ordered=False)\n operations = []\n \n if len(operations) > 0:\n self.db.variant.bulk_write(operations, ordered=False)\n \n return nr_inserted", "response": "Adds a bulk of variants to the database"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef search_variants(self, variant_ids):\n query = {'_id': {'$in': variant_ids}}\n \n return self.db.variant.find(query)", "response": "Make a batch search for variants in the database"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_variants(self, chromosome=None, start=None, end=None):\n query = {}\n if chromosome:\n query['chrom'] = chromosome\n if start:\n query['start'] = {'$lte': end}\n query['end'] = {'$gte': start}\n LOG.info(\"Find all variants {}\".format(query))\n return self.db.variant.find(query).sort([('start', ASCENDING)])", "response": "Return all variants in the database."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef delete_variant(self, variant):\n mongo_variant = self.get_variant(variant)\n \n if mongo_variant:\n \n if mongo_variant['observations'] == 1:\n LOG.debug(\"Removing variant {0}\".format(\n mongo_variant.get('_id')\n ))\n message = self.db.variant.delete_one({'_id': variant['_id']})\n else:\n LOG.debug(\"Decreasing observations for {0}\".format(\n mongo_variant.get('_id')\n ))\n message = self.db.variant.update_one({\n '_id': mongo_variant['_id']\n },{\n '$inc': {\n 'observations': -1,\n 'homozygote': - (variant.get('homozygote', 0)),\n 'hemizygote': - (variant.get('hemizygote', 0)),\n },\n '$pull': {\n 'families': variant.get('case_id')\n }\n }, upsert=False)\n return", "response": "Delete a single variant in database"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_chromosomes(self, sv=False):\n if sv:\n res = self.db.structural_variant.distinct('chrom')\n else:\n res = self.db.variant.distinct('chrom')\n \n return res", "response": "Return a list of all chromosomes found in the database."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_max_position(self, chrom):\n res = self.db.variant.find({'chrom':chrom}, {'_id':0, 'end':1}).sort([('end', DESCENDING)]).limit(1)\n end = 0\n for variant in res:\n end = variant['end']\n return end", "response": "Get the last position observed on a chromosome in the database"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef restore(ctx, filename):\n filename = filename or background_path\n if not os.path.isfile(filename):\n LOG.warning(\"File {} does not exist. Please point to a valid file\".format(filename))\n ctx.abort()\n \n call = ['mongorestore', '--gzip', '--db', 'loqusdb', '--archive={}'.format(filename)]\n \n LOG.info('Restoring database from %s', filename)\n start_time = datetime.now()\n try:\n completed = subprocess.run(call, check=True)\n except subprocess.CalledProcessError as err:\n LOG.warning(err)\n ctx.abort()\n \n LOG.info('Database restored succesfully')\n LOG.info('Time to restore database: {0}'.format(datetime.now()-start_time))", "response": "Restore the database from a zipped file."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _create_w_objective(m, X, R):\n genes, clusters = m.shape\n cells = X.shape[1]\n R1 = R.reshape((genes, 1)).dot(np.ones((1, cells)))\n def objective(w):\n # convert w into a matrix first... because it's a vector for\n # optimization purposes\n w = w.reshape((m.shape[1], X.shape[1]))\n d = m.dot(w)+eps\n return np.sum((X + R1)*np.log(d + R1) - X*np.log(d))/genes\n def deriv(w):\n # derivative of objective wrt all elements of w\n # for w_{ij}, the derivative is... m_j1+...+m_jn sum over genes minus \n # x_ij\n w2 = w.reshape((m.shape[1], X.shape[1]))\n d = m.dot(w2)+eps\n temp = X/d\n temp2 = (X+R1)/(d+R1)\n m1 = m.T.dot(temp2)\n m2 = m.T.dot(temp)\n deriv = m1 - m2\n return deriv.flatten()/genes\n return objective, deriv", "response": "Creates an objective function and its derivative for W given M and X and R."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef nb_estimate_state(data, clusters, R=None, init_means=None, init_weights=None, max_iters=10, tol=1e-4, disp=True, inner_max_iters=400, normalize=True):\n # TODO: deal with non-NB data... just ignore it? or do something else?\n data_subset = data.copy()\n genes, cells = data_subset.shape\n # 1. use nb_fit to get inital Rs\n if R is None:\n nb_indices = find_nb_genes(data)\n data_subset = data[nb_indices, :]\n if init_means is not None and len(init_means) > sum(nb_indices):\n init_means = init_means[nb_indices, :]\n genes, cells = data_subset.shape\n R = np.zeros(genes)\n P, R = nb_fit(data_subset)\n if init_means is None:\n means, assignments = kmeans_pp(data_subset, clusters)\n else:\n means = init_means.copy()\n clusters = means.shape[1]\n w_init = np.random.random(cells*clusters)\n if init_weights is not None:\n if len(init_weights.shape)==1:\n init_weights = initialize_from_assignments(init_weights, clusters)\n w_init = init_weights.reshape(cells*clusters)\n m_init = means.reshape(genes*clusters)\n ll = np.inf\n # repeat steps 1 and 2 until convergence:\n for i in range(max_iters):\n if disp:\n print('iter: {0}'.format(i))\n w_bounds = [(0, 1.0) for x in w_init]\n m_bounds = [(0, None) for x in m_init]\n # step 1: given M, estimate W\n w_objective, w_deriv = _create_w_objective(means, data_subset, R)\n w_res = minimize(w_objective, w_init, method='L-BFGS-B', jac=w_deriv, bounds=w_bounds, options={'disp':disp, 'maxiter':inner_max_iters})\n w_diff = np.sqrt(np.sum((w_res.x-w_init)**2))/w_init.size\n w_new = w_res.x.reshape((clusters, cells))\n w_init = w_res.x\n # step 2: given W, update M\n m_objective, m_deriv = _create_m_objective(w_new, data_subset, R)\n # method could be 'L-BFGS-B' or 'SLSQP'... SLSQP gives a memory error...\n # or use TNC...\n m_res = minimize(m_objective, m_init, method='L-BFGS-B', jac=m_deriv, bounds=m_bounds, options={'disp':disp, 'maxiter':inner_max_iters})\n m_diff = np.sqrt(np.sum((m_res.x-m_init)**2))/m_init.size\n m_new = m_res.x.reshape((genes, clusters))\n m_init = m_res.x\n ll = m_res.fun\n means = m_new\n if w_diff < tol and m_diff < tol:\n break\n if normalize:\n w_new = w_new/w_new.sum(0)\n return m_new, w_new, R, ll", "response": "Use a Negative Binomial Mixture model to estimate cell states and cell mixing weights for each cell."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns an objective function and its derivative for M given W and X.", "response": "def poisson_objective(X, m, w):\n \"\"\"\n Creates an objective function and its derivative for M, given W and X\n Args:\n w (array): clusters x cells\n X (array): genes x cells\n selected_genes (array): array of ints - genes to be selected\n \"\"\"\n clusters, cells = w.shape\n genes = X.shape[0]\n #m = m.reshape((X.shape[0], w.shape[0]))\n d = m.dot(w)+eps\n #temp = X/d\n #w_sum = w.sum(1)\n #w2 = w.dot(temp.T)\n #deriv = w_sum - w2.T\n return np.sum(d - X*np.log(d))/genes"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrunning LightLDA on the given data and returns the state of the cluster.", "response": "def lightlda_estimate_state(data, k, input_folder=\"data1/LightLDA_input\", threads=8, max_iters=250, prepare_data=True, init_means=None, init_weights=None, lightlda_folder=None, data_capacity=1000):\n \"\"\"\n Runs LDA on the given dataset (can be an 2-D array of any form - sparse\n or dense, as long as it can be indexed). If the data has not already been\n prepared into LDA format, set \"prepare_data\" to TRUE. If \"prepare_data\" is\n FALSE, the method assumes that the data has already been preprocessed into\n LightLDA format and is located at the given \"input_folder\".\n \"\"\"\n if lightlda_folder is None:\n lightlda_folder = LIGHTLDA_FOLDER\n if prepare_data:\n prepare_lightlda_data(data, input_folder, lightlda_folder)\n\n # Check if initializations for M/W were provided.\n if ((init_means is not None) and (init_weights is None)) or ((init_means is None) and (init_weights is not None)):\n raise ValueError(\"LightLDA requires that either both M and W be initialized, or neither. You initialized one but not the other.\")\n \n warm_start = False\n\n # If we have initial M/W matrices, write to the model and doc-topic files\n if (init_means is not None) and (init_weights is not None):\n warm_start = True\n init_means = init_means/init_means.sum(0)\n init_weights = init_weights/init_weights.sum(0)\n create_model_file(\"server_0_table_0.model\", init_means)\n create_model_file(\"doc_topic.0\", init_weights.T)\n print(init_means)\n print(\"init_means\")\n\n # Run LightLDA\n print(\"TRAINING\")\n # TODO: argument for data capacity\n train_args = (os.path.join(lightlda_folder, \"bin/lightlda\"), \"-num_vocabs\", str(data.shape[0]), \"-num_topics\",\n str(k), \"-num_iterations\", str(max_iters), \"-alpha\", \"0.05\", \"-beta\", \"0.01\", \"-mh_steps\", \"2\",\n \"-num_local_workers\", str(threads), \"-num_blocks\", \"1\", \"-max_num_document\", str(data.shape[1]),\n \"-input_dir\", input_folder, \"-data_capacity\", str(data_capacity))\n if warm_start:\n print(\"warm start\")\n train_args = train_args + (\"-warm_start\",)\n\n # Call LightLDA\n subprocess.call(train_args)\n \n # Parse final model and doc-topic files to obtain M/W\n print(\"data shape\")\n print(data.shape)\n\n M = parse_model_file(\"server_0_table_0.model\", k, data.shape[0])\n W = parse_result_file(\"doc_topic.0\", k)\n \n # Not sure if normalization is correct\n M = M * (np.mean(data) / np.mean(M))\n W = W/W.sum(0)\n print(\"shapes\")\n print(M.shape)\n print(W.shape)\n # TODO: poisson_objective doesn't work for sparse matrices\n if sparse.issparse(data):\n ll = 0\n else:\n ll = poisson_objective(data, M, W)\n #M = M * (5./np.mean(M))\n return M, W, ll"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nloads all files in a directory.", "response": "def cli(ctx, directory, uri, verbose, count):\n \"\"\"Load all files in a directory.\"\"\"\n # configure root logger to print to STDERR\n loglevel = \"INFO\"\n if verbose:\n loglevel = \"DEBUG\"\n coloredlogs.install(level=loglevel)\n \n p = Path(directory)\n \n if not p.is_dir():\n LOG.warning(\"{0} is not a valid directory\".format(directory))\n ctx.abort()\n \n start_time = datetime.now()\n # Make sure that the database is indexed\n index_call = ['loqusdb', 'index']\n base_call = ['loqusdb']\n if uri:\n base_call.append('--uri')\n base_call.append(uri)\n index_call.append('--uri')\n index_call.append(uri)\n \n subprocess.run(index_call)\n base_call.append('load')\n \n nr_files = 0\n for nr_files,file_name in enumerate(list(p.glob('*.vcf')),1):\n call = deepcopy(base_call)\n case_id = file_name.stem.split('.')[0]\n call.append('--sv-variants')\n call.append(str(file_name))\n call.append('--case-id')\n call.append(case_id)\n if count:\n continue\n try:\n subprocess.run(call, check=True)\n except subprocess.CalledProcessError as err:\n LOG.warning(err)\n LOG.warning(\"Failed to load file %s\", filename)\n LOG.info(\"Continue with files...\")\n \n if nr_files % 100:\n LOG.info(\"%s files loaded\", nr_files)\n \n LOG.info(\"%s files inserted\", nr_files)\n LOG.info(\"Time to insert files: {}\".format(datetime.now()-start_time))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngenerating initial M and W given a data set and an array of cluster labels.", "response": "def nmf_init(data, clusters, k, init='enhanced'):\n \"\"\"\n Generates initial M and W given a data set and an array of cluster labels.\n\n There are 3 options for init:\n enhanced - uses EIn-NMF from Gong 2013\n basic - uses means for M, assigns W such that the chosen cluster for a given cell has value 0.75 and all others have 0.25/(k-1).\n nmf - uses means for M, and assigns W using the NMF objective while holding M constant.\n \"\"\"\n init_m = np.zeros((data.shape[0], k))\n if sparse.issparse(data):\n for i in range(k):\n if data[:,clusters==i].shape[1]==0:\n point = np.random.randint(0, data.shape[1])\n init_m[:,i] = data[:,point].toarray().flatten()\n else:\n init_m[:,i] = np.array(data[:,clusters==i].mean(1)).flatten()\n else:\n for i in range(k):\n if data[:,clusters==i].shape[1]==0:\n point = np.random.randint(0, data.shape[1])\n init_m[:,i] = data[:,point].flatten()\n else:\n init_m[:,i] = data[:,clusters==i].mean(1)\n init_w = np.zeros((k, data.shape[1]))\n if init == 'enhanced':\n distances = np.zeros((k, data.shape[1]))\n for i in range(k):\n for j in range(data.shape[1]):\n distances[i,j] = np.sqrt(((data[:,j] - init_m[:,i])**2).sum())\n for i in range(k):\n for j in range(data.shape[1]):\n init_w[i,j] = 1/((distances[:,j]/distances[i,j])**(-2)).sum()\n elif init == 'basic':\n init_w = initialize_from_assignments(clusters, k)\n elif init == 'nmf':\n init_w_, _, n_iter = non_negative_factorization(data.T, n_components=k, init='custom', update_W=False, W=init_m.T)\n init_w = init_w_.T\n return init_m, init_w"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nlogging Normalization of a NMF file.", "response": "def log_norm_nmf(data, k, normalize_w=True, return_cost=True, init_weights=None, init_means=None, write_progress_file=None, **kwargs):\n \"\"\"\n Args:\n data (array): dense or sparse array with shape (genes, cells)\n k (int): number of cell types\n normalize_w (bool, optional): True if W should be normalized (so that each column sums to 1). Default: True\n return_cost (bool, optional): True if the NMF objective value (squared error) should be returned. Default: True\n init_weights (array, optional): Initial value for W. Default: None\n init_means (array, optional): Initial value for M. Default: None\n **kwargs: misc arguments to NMF\n\n Returns:\n Two matrices M of shape (genes, k) and W of shape (k, cells). They correspond to M and M in Poisson state estimation. If return_cost is True (which it is by default), then the cost will also be returned. This might be prohibitably costly\n \"\"\"\n init = None\n data = log1p(cell_normalize(data))\n if init_weights is not None or init_means is not None:\n init = 'custom'\n if init_weights is None:\n init_weights_, _, n_iter = non_negative_factorization(data.T, n_components=k, init='custom', update_W=False, W=init_means.T)\n init_weights = init_weights_.T\n elif init_means is None:\n init_means, _, n_iter = non_negative_factorization(data, n_components=k, init='custom', update_W=False, W=init_weights)\n init_means = init_means.copy(order='C')\n init_weights = init_weights.copy(order='C')\n nmf = NMF(k, init=init, **kwargs)\n if write_progress_file is not None:\n progress = open(write_progress_file, 'w')\n progress.write(str(0))\n progress.close()\n M = nmf.fit_transform(data, W=init_means, H=init_weights)\n W = nmf.components_\n if normalize_w:\n W = W/W.sum(0)\n if return_cost:\n cost = 0\n if sparse.issparse(data):\n ws = sparse.csr_matrix(M)\n hs = sparse.csr_matrix(W)\n cost = 0.5*((data - ws.dot(hs)).power(2)).sum()\n else:\n cost = 0.5*((data - M.dot(W))**2).sum()\n return M, W, cost\n else:\n return M, W"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef check_par(chrom, pos):\n par = False\n \n for interval in PAR.get(chrom,[]):\n if (pos >= interval[0] and pos <= interval[1]):\n par = True\n \n return par", "response": "Check if a coordinate is in the PAR region\n \nAttributeNames."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_variant_id(variant):\n variant_id = '_'.join([\n str(variant.CHROM),\n str(variant.POS),\n str(variant.REF),\n str(variant.ALT[0])\n ]\n )\n return variant_id", "response": "Get a variant id on the format chrom_pos_ref_alt"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck if a position is greater than b.", "response": "def is_greater(a,b):\n \"\"\"Check if position a is greater than position b\n This will look at chromosome and position.\n \n For example a position where chrom = 2 and pos = 300 is greater than a position where\n chrom = 1 and pos = 1000\n \n If any of the chromosomes is outside [1-22,X,Y,MT] we can not say which is biggest.\n \n Args:\n a,b(Position)\n \n Returns:\n bool: True if a is greater than b\n \"\"\"\n \n a_chrom = CHROM_TO_INT.get(a.chrom,0)\n b_chrom = CHROM_TO_INT.get(b.chrom,0)\n \n if (a_chrom == 0 or b_chrom == 0):\n return False\n \n if a_chrom > b_chrom:\n return True\n \n if a_chrom == b_chrom:\n if a.pos > b.pos:\n return True\n \n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_coords(variant):\n coordinates = {\n 'chrom': None,\n 'end_chrom': None,\n 'sv_length': None,\n 'sv_type': None,\n 'pos': None,\n 'end': None,\n }\n chrom = variant.CHROM\n if chrom.startswith(('chr', 'CHR', 'Chr')):\n chrom = chrom[3:]\n coordinates['chrom'] = chrom\n end_chrom = chrom\n \n pos = int(variant.POS)\n alt = variant.ALT[0]\n\n # Get the end position\n # This will be None for non-svs\n end_pos = variant.INFO.get('END')\n if end_pos:\n end = int(end_pos)\n else:\n end = int(variant.end)\n coordinates['end'] = end\n \n sv_type = variant.INFO.get('SVTYPE')\n length = variant.INFO.get('SVLEN')\n if length:\n sv_len = abs(length)\n else:\n sv_len = end - pos\n\n # Translocations will sometimes have a end chrom that differs from chrom\n if sv_type == 'BND':\n other_coordinates = alt.strip('ACGTN[]').split(':')\n end_chrom = other_coordinates[0]\n if end_chrom.startswith(('chr', 'CHR', 'Chr')):\n end_chrom = end_chrom[3:]\n\n end = int(other_coordinates[1])\n\n #Set 'infinity' to length if translocation\n sv_len = float('inf')\n\n # Insertions often have length 0 in VCF\n if (sv_len == 0 and alt != ''):\n sv_len = len(alt)\n\n if (pos == end) and (sv_len > 0):\n end = pos + sv_len\n\n position = Position(chrom, pos)\n end_position = Position(end_chrom, end)\n \n # If 'start' is greater than 'end', switch positions\n if is_greater(position, end_position):\n end_chrom = position.chrom\n end = position.pos\n \n chrom = end_position.chrom\n pos = end_position.pos\n \n coordinates['end_chrom'] = end_chrom\n coordinates['pos'] = pos\n coordinates['end'] = end\n coordinates['sv_length'] = sv_len\n coordinates['sv_type'] = sv_type\n \n return coordinates", "response": "Returns a dictionary with position information for the current VCF record."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nbuild a Variant object from a Cyvcf2 formated variant line and return a dictionary of Variant objects.", "response": "def build_variant(variant, case_obj, case_id=None, gq_treshold=None):\n \"\"\"Return a Variant object\n\n Take a cyvcf2 formated variant line and return a models.Variant.\n\n If criterias are not fullfilled, eg. variant have no gt call or quality\n is below gq treshold then return None.\n\n Args:\n variant(cyvcf2.Variant)\n case_obj(Case): We need the case object to check individuals sex\n case_id(str): The case id\n gq_treshold(int): Genotype Quality treshold\n \n Return:\n formated_variant(models.Variant): A variant dictionary\n \"\"\"\n variant_obj = None\n\n sv = False\n # Let cyvcf2 tell if it is a Structural Variant or not\n if variant.var_type == 'sv':\n sv = True\n\n # chrom_pos_ref_alt\n variant_id = get_variant_id(variant)\n\n ref = variant.REF\n # ALT is an array in cyvcf2\n # We allways assume splitted and normalized VCFs\n alt = variant.ALT[0]\n\n coordinates = get_coords(variant)\n chrom = coordinates['chrom']\n pos = coordinates['pos']\n\n # These are integers that will be used when uploading\n found_homozygote = 0\n found_hemizygote = 0\n\n # Only look at genotypes for the present individuals\n if sv:\n found_variant = True\n else:\n found_variant = False\n for ind_obj in case_obj['individuals']:\n ind_id = ind_obj['ind_id']\n # Get the index position for the individual in the VCF\n ind_pos = ind_obj['ind_index']\n gq = int(variant.gt_quals[ind_pos])\n if (gq_treshold and gq < gq_treshold):\n continue\n\n genotype = GENOTYPE_MAP[variant.gt_types[ind_pos]]\n\n if genotype in ['het', 'hom_alt']:\n LOG.debug(\"Found variant\")\n found_variant = True\n\n # If variant in X or Y and individual is male,\n # we need to check hemizygosity\n if chrom in ['X','Y'] and ind_obj['sex'] == 1:\n if not check_par(chrom, pos):\n LOG.debug(\"Found hemizygous variant\")\n found_hemizygote = 1\n\n if genotype == 'hom_alt':\n LOG.debug(\"Found homozygote alternative variant\")\n found_homozygote = 1\n\n if found_variant:\n \n variant_obj = Variant(\n variant_id=variant_id,\n chrom=chrom,\n pos=pos,\n end=coordinates['end'],\n ref=ref,\n alt=alt,\n end_chrom=coordinates['end_chrom'],\n sv_type = coordinates['sv_type'],\n sv_len = coordinates['sv_length'],\n case_id = case_id,\n homozygote = found_homozygote,\n hemizygote = found_hemizygote,\n is_sv = sv,\n id_column = variant.ID,\n )\n\n return variant_obj"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nmigrate an old loqusdb instance to 1. 0", "response": "def migrate(ctx,):\n \"\"\"Migrate an old loqusdb instance to 1.0\n \"\"\"\n adapter = ctx.obj['adapter']\n \n start_time = datetime.now()\n \n nr_updated = migrate_database(adapter)\n \n LOG.info(\"All variants updated, time to complete migration: {}\".format(\n datetime.now() - start_time))\n LOG.info(\"Nr variants that where updated: %s\", nr_updated)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef update(ctx, variant_file, sv_variants, family_file, family_type, skip_case_id, gq_treshold, \n case_id, ensure_index, max_window):\n \"\"\"Load the variants of a case\n\n A variant is loaded if it is observed in any individual of a case\n If no family file is provided all individuals in vcf file will be considered.\n \"\"\"\n if not (family_file or case_id):\n LOG.warning(\"Please provide a family file or a case id\")\n ctx.abort()\n \n if not (variant_file or sv_variants):\n LOG.warning(\"Please provide a VCF file\")\n ctx.abort()\n \n\n variant_path = None\n if variant_file:\n variant_path = os.path.abspath(variant_file)\n \n variant_sv_path = None\n if sv_variants:\n variant_sv_path = os.path.abspath(sv_variants)\n \n adapter = ctx.obj['adapter']\n\n start_inserting = datetime.now()\n \n \n try:\n nr_inserted = update_database(\n adapter=adapter,\n variant_file=variant_path,\n sv_file=variant_sv_path,\n family_file=family_file,\n family_type=family_type,\n skip_case_id=skip_case_id,\n case_id=case_id,\n gq_treshold=gq_treshold,\n max_window=max_window,\n )\n except (SyntaxError, CaseError, IOError, VcfError) as error:\n LOG.warning(error)\n ctx.abort()\n\n LOG.info(\"Nr variants inserted: %s\", nr_inserted)\n LOG.info(\"Time to insert variants: {0}\".format(\n datetime.now() - start_inserting))\n \n if ensure_index:\n adapter.ensure_indexes()\n else:\n adapter.check_indexes()", "response": "Load the variants of a single case and store them in the database."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nexporting the variants of a loqus db Shipment to a vcf file", "response": "def export(ctx, outfile, variant_type):\n \"\"\"Export the variants of a loqus db\n \n The variants are exported to a vcf file\n \"\"\"\n adapter = ctx.obj['adapter']\n version = ctx.obj['version']\n \n LOG.info(\"Export the variants from {0}\".format(adapter))\n nr_cases = 0\n\n is_sv = variant_type == 'sv'\n existing_chromosomes = set(adapter.get_chromosomes(sv=is_sv))\n \n ordered_chromosomes = []\n for chrom in CHROMOSOME_ORDER:\n if chrom in existing_chromosomes:\n ordered_chromosomes.append(chrom)\n existing_chromosomes.remove(chrom)\n for chrom in existing_chromosomes:\n ordered_chromosomes.append(chrom)\n \n nr_cases = adapter.cases().count()\n LOG.info(\"Found {0} cases in database\".format(nr_cases))\n\n head = HeaderParser()\n head.add_fileformat(\"VCFv4.3\")\n head.add_meta_line(\"NrCases\", nr_cases)\n head.add_info(\"Obs\", '1', 'Integer', \"The number of observations for the variant\")\n head.add_info(\"Hom\", '1', 'Integer', \"The number of observed homozygotes\")\n head.add_info(\"Hem\", '1', 'Integer', \"The number of observed hemizygotes\")\n head.add_version_tracking(\"loqusdb\", version, datetime.now().strftime(\"%Y-%m-%d %H:%M\"))\n \n if variant_type == 'sv':\n head.add_info(\"END\", '1', 'Integer', \"End position of the variant\")\n head.add_info(\"SVTYPE\", '1', 'String', \"Type of structural variant\")\n head.add_info(\"SVLEN\", '1', 'Integer', \"Length of structural variant\")\n \n \n for chrom in ordered_chromosomes:\n length = adapter.get_max_position(chrom)\n head.add_contig(contig_id=chrom, length=str(length))\n\n print_headers(head, outfile=outfile)\n \n for chrom in ordered_chromosomes:\n if variant_type == 'snv':\n LOG.info(\"Collecting all SNV variants\")\n variants = adapter.get_variants(chromosome=chrom)\n else:\n LOG.info(\"Collecting all SV variants\")\n variants = adapter.get_sv_variants(chromosome=chrom)\n LOG.info(\"{} variants found\".format(variants.count()))\n for variant in variants:\n variant_line = format_variant(variant, variant_type=variant_type)\n # chrom = variant['chrom']\n # pos = variant['start']\n # ref = variant['ref']\n # alt = variant['alt']\n # observations = variant['observations']\n # homozygotes = variant['homozygote']\n # hemizygotes = variant['hemizygote']\n # info = \"Obs={0}\".format(observations)\n # if homozygotes:\n # info += \";Hom={0}\".format(homozygotes)\n # if hemizygotes:\n # info += \";Hem={0}\".format(hemizygotes)\n # variant_line = \"{0}\\t{1}\\t.\\t{2}\\t{3}\\t.\\t.\\t{4}\\n\".format(\n # chrom, pos, ref, alt, info)\n print_variant(variant_line=variant_line, outfile=outfile)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef load_database(adapter, variant_file=None, sv_file=None, family_file=None,\n family_type='ped', skip_case_id=False, gq_treshold=None,\n case_id=None, max_window = 3000, profile_file=None,\n hard_threshold=0.95, soft_threshold=0.9):\n \"\"\"Load the database with a case and its variants\n\n Args:\n adapter: Connection to database\n variant_file(str): Path to variant file\n sv_file(str): Path to sv variant file\n family_file(str): Path to family file\n family_type(str): Format of family file\n skip_case_id(bool): If no case information should be added to variants\n gq_treshold(int): If only quality variants should be considered\n case_id(str): If different case id than the one in family file should be used\n max_window(int): Specify the max size for sv windows\n check_profile(bool): Does profile check if True\n hard_threshold(float): Rejects load if hamming distance above this is found\n soft_threshold(float): Stores similar samples if hamming distance above this is found\n\n Returns:\n nr_inserted(int)\n \"\"\"\n vcf_files = []\n\n nr_variants = None\n vcf_individuals = None\n if variant_file:\n vcf_info = check_vcf(variant_file)\n nr_variants = vcf_info['nr_variants']\n variant_type = vcf_info['variant_type']\n vcf_files.append(variant_file)\n # Get the indivuduals that are present in vcf file\n vcf_individuals = vcf_info['individuals']\n\n nr_sv_variants = None\n sv_individuals = None\n if sv_file:\n vcf_info = check_vcf(sv_file, 'sv')\n nr_sv_variants = vcf_info['nr_variants']\n vcf_files.append(sv_file)\n sv_individuals = vcf_info['individuals']\n\n profiles = None\n matches = None\n if profile_file:\n profiles = get_profiles(adapter, profile_file)\n ###Check if any profile already exists\n matches = profile_match(adapter,\n profiles,\n hard_threshold=hard_threshold,\n soft_threshold=soft_threshold)\n\n # If a gq treshold is used the variants needs to have GQ\n for _vcf_file in vcf_files:\n # Get a cyvcf2.VCF object\n vcf = get_vcf(_vcf_file)\n\n if gq_treshold:\n if not vcf.contains('GQ'):\n LOG.warning('Set gq-treshold to 0 or add info to vcf {0}'.format(_vcf_file))\n raise SyntaxError('GQ is not defined in vcf header')\n\n # Get a ped_parser.Family object from family file\n family = None\n family_id = None\n if family_file:\n LOG.info(\"Loading family from %s\", family_file)\n with open(family_file, 'r') as family_lines:\n family = get_case(\n family_lines=family_lines,\n family_type=family_type\n )\n family_id = family.family_id\n\n # There has to be a case_id or a family at this stage.\n case_id = case_id or family_id\n # Convert infromation to a loqusdb Case object\n case_obj = build_case(\n case=family,\n case_id=case_id,\n vcf_path=variant_file,\n vcf_individuals=vcf_individuals,\n nr_variants=nr_variants,\n vcf_sv_path=sv_file,\n sv_individuals=sv_individuals,\n nr_sv_variants=nr_sv_variants,\n profiles=profiles,\n matches=matches,\n profile_path=profile_file\n )\n # Build and load a new case, or update an existing one\n load_case(\n adapter=adapter,\n case_obj=case_obj,\n )\n\n nr_inserted = 0\n # If case was succesfully added we can store the variants\n for file_type in ['vcf_path','vcf_sv_path']:\n variant_type = 'snv'\n if file_type == 'vcf_sv_path':\n variant_type = 'sv'\n if case_obj.get(file_type) is None:\n continue\n\n vcf_obj = get_vcf(case_obj[file_type])\n try:\n nr_inserted += load_variants(\n adapter=adapter,\n vcf_obj=vcf_obj,\n case_obj=case_obj,\n skip_case_id=skip_case_id,\n gq_treshold=gq_treshold,\n max_window=max_window,\n variant_type=variant_type,\n )\n except Exception as err:\n # If something went wrong do a rollback\n LOG.warning(err)\n delete(\n adapter=adapter,\n case_obj=case_obj,\n )\n raise err\n return nr_inserted", "response": "Load the database with a case and its variants"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef load_case(adapter, case_obj, update=False):\n # Check if the case already exists in database.\n existing_case = adapter.case(case_obj)\n if existing_case:\n if not update:\n raise CaseError(\"Case {0} already exists in database\".format(case_obj['case_id']))\n case_obj = update_case(case_obj, existing_case)\n\n # Add the case to database\n try:\n adapter.add_case(case_obj, update=update)\n except CaseError as err:\n raise err\n\n return case_obj", "response": "Load a case to the database\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nloading variants for a family into the database.", "response": "def load_variants(adapter, vcf_obj, case_obj, skip_case_id=False, gq_treshold=None,\n max_window=3000, variant_type='snv'):\n \"\"\"Load variants for a family into the database.\n\n Args:\n adapter (loqusdb.plugins.Adapter): initialized plugin\n case_obj(Case): dict with case information\n nr_variants(int)\n skip_case_id (bool): whether to include the case id on variant level\n or not\n gq_treshold(int)\n max_window(int): Specify the max size for sv windows\n variant_type(str): 'sv' or 'snv'\n\n Returns:\n nr_inserted(int)\n \"\"\"\n if variant_type == 'snv':\n nr_variants = case_obj['nr_variants']\n else:\n nr_variants = case_obj['nr_sv_variants']\n\n nr_inserted = 0\n case_id = case_obj['case_id']\n if skip_case_id:\n case_id = None\n # Loop over the variants in the vcf\n with click.progressbar(vcf_obj, label=\"Inserting variants\",length=nr_variants) as bar:\n\n variants = (build_variant(variant,case_obj,case_id, gq_treshold) for variant in bar)\n\n if variant_type == 'sv':\n for sv_variant in variants:\n if not sv_variant:\n continue\n adapter.add_structural_variant(variant=sv_variant, max_window=max_window)\n nr_inserted += 1\n\n if variant_type == 'snv':\n nr_inserted = adapter.add_variants(variants)\n\n LOG.info(\"Inserted %s variants of type %s\", nr_inserted, variant_type)\n\n return nr_inserted"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncalculating the mean and variance of a sparse matrix.", "response": "def sparse_mean_var(data):\n \"\"\"\n Calculates the variance for each row of a sparse matrix,\n using the relationship Var = E[x^2] - E[x]^2.\n\n Returns:\n pair of matrices mean, variance.\n \"\"\"\n data = sparse.csc_matrix(data)\n return sparse_means_var_csc(data.data,\n data.indices,\n data.indptr,\n data.shape[1],\n data.shape[0])"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef cell_normalize(data):\n if sparse.issparse(data):\n data = sparse.csc_matrix(data.astype(float))\n # normalize in-place\n sparse_cell_normalize(data.data,\n data.indices,\n data.indptr,\n data.shape[1],\n data.shape[0])\n return data\n data_norm = data.astype(float)\n total_umis = []\n for i in range(data.shape[1]):\n di = data_norm[:,i]\n total_umis.append(di.sum())\n di /= total_umis[i]\n med = np.median(total_umis)\n data_norm *= med\n return data_norm", "response": "Normalizes the data where the expression is equal to the total count per cell."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the log1p of the data.", "response": "def log1p(data):\n \"\"\"\n Returns ln(data+1), whether the original data is dense or sparse.\n \"\"\"\n if sparse.issparse(data):\n return data.log1p()\n else:\n return np.log1p(data)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a dictionary with individual positions", "response": "def get_individual_positions(individuals):\n \"\"\"Return a dictionary with individual positions\n\n Args:\n individuals(list): A list with vcf individuals in correct order\n\n Returns:\n ind_pos(dict): Map from ind_id -> index position\n \"\"\"\n ind_pos = {}\n if individuals:\n for i, ind in enumerate(individuals):\n ind_pos[ind] = i\n return ind_pos"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef build_case(case, vcf_individuals=None, case_id=None, vcf_path=None, sv_individuals=None,\n vcf_sv_path=None, nr_variants=None, nr_sv_variants=None, profiles=None,\n matches=None, profile_path=None):\n \"\"\"Build a Case from the given information\n\n Args:\n case(ped_parser.Family): A family object\n vcf_individuals(list): Show the order of inds in vcf file\n case_id(str): If another name than the one in family file should be used\n vcf_path(str)\n sv_individuals(list): Show the order of inds in vcf file\n vcf_sv_path(str)\n nr_variants(int)\n nr_sv_variants(int)\n profiles(dict): The profiles for each sample in vcf\n matches(dict(list)): list of similar samples for each sample in vcf.\n\n Returns:\n case_obj(models.Case)\n \"\"\"\n # Create a dict that maps the ind ids to the position they have in vcf\n individual_positions = get_individual_positions(vcf_individuals)\n sv_individual_positions = get_individual_positions(sv_individuals)\n\n family_id = None\n if case:\n if not case.affected_individuals:\n LOG.warning(\"No affected individuals could be found in ped file\")\n family_id = case.family_id\n\n # If case id is given manually we use that one\n case_id = case_id or family_id\n if case_id is None:\n raise CaseError\n\n case_obj = Case(\n case_id=case_id,\n )\n\n if vcf_path:\n case_obj['vcf_path'] = vcf_path\n case_obj['nr_variants'] = nr_variants\n\n if vcf_sv_path:\n case_obj['vcf_sv_path'] = vcf_sv_path\n case_obj['nr_sv_variants'] = nr_sv_variants\n\n if profile_path:\n case_obj['profile_path'] = profile_path\n\n ind_objs = []\n if case:\n if individual_positions:\n _ind_pos = individual_positions\n else:\n _ind_pos = sv_individual_positions\n\n for ind_id in case.individuals:\n individual = case.individuals[ind_id]\n try:\n #If a profile dict exists, get the profile for ind_id\n profile = profiles[ind_id] if profiles else None\n #If matching samples are found, get these samples for ind_id\n similar_samples = matches[ind_id] if matches else None\n ind_obj = Individual(\n ind_id=ind_id,\n case_id=case_id,\n ind_index=_ind_pos[ind_id],\n sex=individual.sex,\n profile=profile,\n similar_samples=similar_samples\n )\n ind_objs.append(dict(ind_obj))\n except KeyError:\n raise CaseError(\"Ind %s in ped file does not exist in VCF\", ind_id)\n else:\n # If there where no family file we can create individuals from what we know\n for ind_id in individual_positions:\n\n profile = profiles[ind_id] if profiles else None\n similar_samples = matches[ind_id] if matches else None\n ind_obj = Individual(\n ind_id = ind_id,\n case_id = case_id,\n ind_index=individual_positions[ind_id],\n profile=profile,\n similar_samples=similar_samples\n )\n ind_objs.append(dict(ind_obj))\n\n # Add individuals to the correct variant type\n for ind_obj in ind_objs:\n if vcf_sv_path:\n case_obj['sv_individuals'].append(dict(ind_obj))\n case_obj['_sv_inds'][ind_obj['ind_id']] = dict(ind_obj)\n if vcf_path:\n case_obj['individuals'].append(dict(ind_obj))\n case_obj['_inds'][ind_obj['ind_id']] = dict(ind_obj)\n\n return case_obj", "response": "Build a Case object from the given information."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef generate_poisson_data(centers, n_cells, cluster_probs=None):\n genes, clusters = centers.shape\n output = np.zeros((genes, n_cells))\n if cluster_probs is None:\n cluster_probs = np.ones(clusters)/clusters\n labels = []\n for i in range(n_cells):\n c = np.random.choice(range(clusters), p=cluster_probs)\n labels.append(c)\n output[:,i] = np.random.poisson(centers[:,c])\n return output, np.array(labels)", "response": "Generates poisson - distributed data given a set of means for each cluster."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef generate_zip_data(M, L, n_cells, cluster_probs=None):\n genes, clusters = M.shape\n output = np.zeros((genes, n_cells))\n if cluster_probs is None:\n cluster_probs = np.ones(clusters)/clusters\n zip_p = np.random.random((genes, n_cells))\n labels = []\n for i in range(n_cells):\n c = np.random.choice(range(clusters), p=cluster_probs)\n labels.append(c)\n output[:,i] = np.where(zip_p[:,i] < L[:,c], 0, np.random.poisson(M[:,c]))\n return output, np.array(labels)", "response": "Generates zero - inflated poisson - distributed data given a set of means and zero probs for each cluster."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef generate_state_data(means, weights):\n x_true = np.dot(means, weights)\n sample = np.random.poisson(x_true)\n return sample.astype(float)", "response": "Generates data according to the Poisson Convex Mixture Model."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngenerate data according to Zero - inflated Poisson Convex Mixture Model.", "response": "def generate_zip_state_data(means, weights, z):\n \"\"\"\n Generates data according to the Zero-inflated Poisson Convex Mixture Model.\n\n Args:\n means (array): Cell types- genes x clusters\n weights (array): Cell cluster assignments- clusters x cells\n z (float): zero-inflation parameter\n\n Returns:\n data matrix - genes x cells\n \"\"\"\n x_true = np.dot(means, weights)\n sample = np.random.poisson(x_true)\n random = np.random.random(x_true.shape)\n x_true[random < z] = 0\n return sample.astype(float)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngenerates data according to the Negative Binomial Convex Mixture Model.", "response": "def generate_nb_state_data(means, weights, R):\n \"\"\"\n Generates data according to the Negative Binomial Convex Mixture Model.\n\n Args:\n means (array): Cell types- genes x clusters\n weights (array): Cell cluster assignments- clusters x cells\n R (array): dispersion parameter - 1 x genes\n\n Returns:\n data matrix - genes x cells\n \"\"\"\n cells = weights.shape[1]\n # x_true = true means\n x_true = np.dot(means, weights)\n # convert means into P\n R_ = np.tile(R, (cells, 1)).T\n P_true = x_true/(R_ + x_true)\n sample = np.random.negative_binomial(np.tile(R, (cells, 1)).T, P_true)\n return sample.astype(float)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef generate_nb_states(n_states, n_cells, n_genes):\n W = np.random.dirichlet([1]*n_states, size=(n_cells,))\n W = W.T\n M = np.random.random((n_genes, n_states))*100\n R = np.random.randint(1, 100, n_genes)\n return M, W, R", "response": "Generates the means and weights for the Negative Binomial Mixture Model."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef generate_poisson_states(n_states, n_cells, n_genes):\n W = np.random.dirichlet([1]*n_states, size=(n_cells,))\n W = W.T\n M = np.random.random((n_genes, n_states))*100\n return M, W", "response": "Generates the Poisson states for a given set of genes and number of cells."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef generate_poisson_lineage(n_states, n_cells_per_cluster, n_genes, means=300):\n # means...\n M = np.random.random((n_genes, n_states))*means\n center = M.mean(1)\n W = np.zeros((n_states, n_cells_per_cluster*n_states))\n # TODO\n # start at a center where all the clusters have equal probability, and for\n # each cluster, interpolate linearly towards the cluster.\n index = 0\n means = np.array([1.0/n_states]*n_states)\n for c in range(n_states):\n for i in range(n_cells_per_cluster):\n w = np.copy(means)\n new_value = w[c] + i*(1.0 - 1.0/n_states)/n_cells_per_cluster\n w[:] = (1.0 - new_value)/(n_states - 1.0)\n w[c] = new_value\n W[:, index] = w\n index += 1\n return M, W", "response": "Generates a lineage for each state."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngenerating negative binomial data for a set of cells in a cluster.", "response": "def generate_nb_data(P, R, n_cells, assignments=None):\n \"\"\"\n Generates negative binomial data\n\n Args:\n P (array): genes x clusters\n R (array): genes x clusters\n n_cells (int): number of cells\n assignments (list): cluster assignment of each cell. Default:\n random uniform\n\n Returns:\n data array with shape genes x cells\n labels - array of cluster labels\n \"\"\"\n genes, clusters = P.shape\n output = np.zeros((genes, n_cells))\n if assignments is None:\n cluster_probs = np.ones(clusters)/clusters\n labels = []\n for i in range(n_cells):\n if assignments is None:\n c = np.random.choice(range(clusters), p=cluster_probs)\n else:\n c = assignments[i]\n labels.append(c)\n # because numpy's negative binomial, r is the number of successes\n output[:,i] = np.random.negative_binomial(R[:,c], 1.0-P[:,c])\n return output, np.array(labels)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nvisualizing a Poisson W.", "response": "def visualize_poisson_w(w, labels, filename, method='pca', figsize=(18,10), title='', **scatter_options):\n \"\"\"\n Saves a scatter plot of a visualization of W, the result from Poisson SE.\n \"\"\"\n if method == 'pca':\n pca = PCA(2)\n r_dim_red = pca.fit_transform(w.T).T\n elif method == 'tsne':\n pass\n else:\n print(\"Method is not available. use 'pca' (default) or 'tsne'.\")\n return\n visualize_dim_red(r_dim_red, labels, filename, figsize, title, **scatter_options)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nvisualize the dimred matrix r.", "response": "def visualize_dim_red(r, labels, filename=None, figsize=(18,10), title='', legend=True, label_map=None, label_scale=False, label_color_map=None, **scatter_options):\n \"\"\"\n Saves a scatter plot of a (2,n) matrix r, where each column is a cell.\n\n Args:\n r (array): (2,n) matrix\n labels (array): (n,) array of ints/strings or floats. Can be None.\n filename (string): string to save the output graph. If None, then this just displays the plot.\n figsize (tuple): Default: (18, 10)\n title (string): graph title\n legend (bool): Default: True\n label_map (dict): map of labels to label names. Default: None\n label_scale (bool): True if labels is should be treated as floats. Default: False\n label_color_map (array): (n,) array or list of colors for each label.\n \"\"\"\n fig = plt.figure(figsize=figsize)\n plt.cla()\n if not label_scale:\n for i in set(labels):\n label = i\n if label_map is not None:\n label = label_map[i]\n if label_color_map is not None:\n c = label_color_map[i]\n plt.scatter(r[0, labels==i], r[1, labels==i], label=label, c=c, **scatter_options)\n else:\n plt.scatter(r[0, labels==i], r[1, labels==i], label=label, **scatter_options)\n else:\n if labels is None:\n plt.scatter(r[0,:], r[1,:], **scatter_options)\n else:\n plt.scatter(r[0,:], r[1,:], c=labels/labels.max(), **scatter_options)\n plt.title(title)\n if legend:\n plt.legend()\n if filename is not None:\n plt.savefig(filename, dpi=100)\n plt.close()\n return fig"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nrun a pre - processing + clustering experiment on a list of methods.", "response": "def run_experiment(methods, data, n_classes, true_labels, n_runs=10, use_purity=True, use_nmi=False, use_ari=False, use_nne=False, consensus=False):\n \"\"\"\n runs a pre-processing + clustering experiment...\n\n exactly one of use_purity, use_nmi, or use_ari can be true\n\n Args:\n methods: list of 2-tuples. The first element is either a single Preprocess object or a list of Preprocess objects, to be applied in sequence to the data. The second element is either a single Cluster object, a list of Cluster objects, or a list of lists, where each list is a sequence of Preprocess objects with the final element being a Cluster object.\n data: genes x cells array\n true_labels: 1d array of length cells\n consensus: if true, runs a consensus on cluster results for each method at the very end.\n use_purity, use_nmi, use_ari, use_nne: which error metric to use (at most one can be True)\n\n Returns:\n purities (list of lists)\n names (list of lists)\n other (dict): keys: timing, preprocessing, clusterings\n \"\"\"\n results = []\n names = []\n clusterings = {}\n other_results = {}\n other_results['timing'] = {}\n other_results['preprocessing'] = {}\n if use_purity:\n purity_method = purity\n elif use_nmi:\n purity_method = nmi\n elif use_ari:\n purity_method = ari\n elif use_nne:\n purity_method = nne\n for i in range(n_runs):\n print('run {0}'.format(i))\n purities = []\n r = 0\n method_index = 0\n for preproc, cluster in methods:\n t0 = time.time()\n if isinstance(preproc, Preprocess):\n preprocessed, ll = preproc.run(data)\n output_names = preproc.output_names\n else:\n # if the input is a list, only use the first preproc result\n p1 = data\n output_names = ['']\n for p in preproc:\n p1, ll = p.run(p1)\n p1 = p1[0]\n if output_names[0] != '':\n output_names[0] = output_names[0] + '_' + p.output_names[0]\n else:\n output_names[0] = p.output_names[0]\n preprocessed = [p1]\n t1 = time.time() - t0\n for name, pre in zip(output_names, preprocessed):\n starting_index = method_index\n if isinstance(cluster, Cluster):\n #try:\n t0 = time.time()\n labels = cluster.run(pre)\n t2 = t1 + time.time() - t0\n if use_nne:\n purities.append(purity_method(pre, true_labels))\n else:\n purities.append(purity_method(labels, true_labels))\n if i==0:\n names.append(name + '_' + cluster.name)\n clusterings[names[-1]] = []\n other_results['timing'][names[-1]] = []\n print(names[r])\n clusterings[names[r]].append(labels)\n print('time: ' + str(t2))\n other_results['timing'][names[r]].append(t2)\n print(purities[-1])\n r += 1\n method_index += 1\n #except:\n # print('failed to do clustering')\n elif type(cluster) == list:\n for c in cluster:\n if isinstance(c, list):\n t2 = t1\n name2 = name\n sub_data = pre.copy()\n for subproc in c[:-1]:\n t0 = time.time()\n subproc_out, ll = subproc.run(sub_data)\n sub_data = subproc_out[0]\n name2 = name2 + '_' + subproc.output_names[0]\n t2 += time.time() - t0\n t0 = time.time()\n labels = c[-1].run(sub_data)\n t2 += time.time() - t0\n if use_nne:\n purities.append(purity_method(sub_data, true_labels))\n else:\n purities.append(purity_method(labels, true_labels))\n if i==0:\n names.append(name2 + '_' + c[-1].name)\n clusterings[names[-1]] = []\n other_results['timing'][names[-1]] = []\n print(names[r])\n clusterings[names[r]].append(labels)\n other_results['timing'][names[r]].append(t2)\n print('time: ' + str(t2))\n print(purities[-1])\n r += 1\n method_index += 1\n else:\n try:\n t0 = time.time()\n labels = c.run(pre)\n t2 = t1 + time.time() - t0\n if i==0:\n names.append(name + '_' + c.name)\n clusterings[names[-1]] = []\n other_results['timing'][names[-1]] = []\n if use_nne:\n purities.append(purity_method(pre, true_labels))\n else:\n purities.append(purity_method(labels, true_labels))\n print(names[r])\n clusterings[names[r]].append(labels)\n other_results['timing'][names[r]].append(t2)\n print('time: ' + str(t2))\n print(purities[-1])\n r += 1\n method_index += 1\n except:\n print('failed to do clustering')\n # find the highest purity for the pre-processing method\n # save the preprocessing result with the highest NMI\n num_clustering_results = method_index - starting_index\n clustering_results = purities[-num_clustering_results:]\n if i > 0 and len(clustering_results) > 0:\n old_clustering_results = results[-1][starting_index:method_index]\n if max(old_clustering_results) < max(clustering_results):\n other_results['preprocessing'][name] = pre\n else:\n other_results['preprocessing'][name] = pre\n print('\\t'.join(names))\n print('purities: ' + '\\t'.join(map(str, purities)))\n results.append(purities)\n consensus_purities = []\n if consensus:\n other_results['consensus'] = {}\n k = len(np.unique(true_labels))\n for name, clusts in clusterings.items():\n print(name)\n clusts = np.vstack(clusts)\n consensus_clust = CE.cluster_ensembles(clusts, verbose=False, N_clusters_max=k)\n other_results['consensus'][name] = consensus_clust\n if use_purity:\n consensus_purity = purity(consensus_clust.flatten(), true_labels)\n print('consensus purity: ' + str(consensus_purity))\n consensus_purities.append(consensus_purity)\n if use_nmi:\n consensus_nmi = nmi(true_labels, consensus_clust)\n print('consensus NMI: ' + str(consensus_nmi))\n consensus_purities.append(consensus_nmi)\n if use_ari:\n consensus_ari = ari(true_labels, consensus_clust)\n print('consensus ARI: ' + str(consensus_ari))\n consensus_purities.append(consensus_ari)\n print('consensus results: ' + '\\t'.join(map(str, consensus_purities)))\n other_results['clusterings'] = clusterings\n return results, names, other_results"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef generate_visualizations(methods, data, true_labels, base_dir = 'visualizations',\n figsize=(18,10), **scatter_options):\n \"\"\"\n Generates visualization scatters for all the methods.\n\n Args:\n methods: follows same format as run_experiments. List of tuples.\n data: genes x cells\n true_labels: array of integers\n base_dir: base directory to save all the plots\n figsize: tuple of ints representing size of figure\n scatter_options: options for plt.scatter\n \"\"\"\n plt.figure(figsize=figsize)\n for method in methods:\n preproc= method[0]\n if isinstance(preproc, Preprocess):\n preprocessed, ll = preproc.run(data)\n output_names = preproc.output_names\n else:\n # if the input is a list, only use the first preproc result\n p1 = data\n output_names = ['']\n for p in preproc:\n p1, ll = p.run(p1)\n p1 = p1[0]\n output_names[0] = output_names[0] + p.output_names[0]\n preprocessed = [p1]\n for r, name in zip(preprocessed, output_names):\n # TODO: cluster labels\n print(name)\n # if it's 2d, just display it... else, do tsne to reduce to 2d\n if r.shape[0]==2:\n r_dim_red = r\n else:\n # sometimes the data is too big to do tsne... (for sklearn)\n if sparse.issparse(r) and r.shape[0] > 100:\n name = 'tsvd_' + name\n tsvd = TruncatedSVD(50)\n r_dim_red = tsvd.fit_transform(r.T)\n try:\n tsne = TSNE(2)\n r_dim_red = tsne.fit_transform(r_dim_red).T\n name = 'tsne_' + name\n except:\n tsvd2 = TruncatedSVD(2)\n r_dim_red = tsvd2.fit_transform(r_dim_red).T\n else:\n name = 'tsne_' + name\n tsne = TSNE(2)\n r_dim_red = tsne.fit_transform(r.T).T\n if isinstance(method[1], list):\n for clustering_method in method[1]:\n try:\n cluster_labels = clustering_method.run(r)\n except:\n print('clustering failed')\n continue\n output_path = base_dir + '/{0}_{1}_labels.png'.format(name, clustering_method.name)\n visualize_dim_red(r_dim_red, cluster_labels, output_path, **scatter_options)\n else:\n clustering_method = method[1]\n try:\n cluster_labels = clustering_method.run(r)\n except:\n print('clustering failed')\n continue\n output_path = base_dir + '/{0}_{1}_labels.png'.format(name, clustering_method.name)\n visualize_dim_red(r_dim_red, cluster_labels, output_path, **scatter_options)\n output_path = base_dir + '/{0}_true_labels.png'.format(name)\n visualize_dim_red(r_dim_red, true_labels, output_path, **scatter_options)", "response": "Generates visualizations for all the methods in the cluster."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nrunning the Poisson EstimateState method.", "response": "def run(self, data):\n \"\"\"\n Returns:\n list of W, M*W\n ll\n \"\"\"\n if self.normalize_data:\n data = cell_normalize(data)\n M, W, ll = poisson_estimate_state(data, **self.params)\n outputs = []\n if self.return_w:\n outputs.append(W)\n if self.return_m:\n outputs.append(M)\n if self.return_mw:\n outputs.append(M.dot(W))\n if self.return_mds:\n X = dim_reduce(M, W, 2)\n outputs.append(X.T.dot(W))\n return outputs, ll"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef calculate_hex(hex_string):\n hex_conv = hex_string.split(':')\n converted_hex = (int(hex_conv[0], 16) + int(hex_conv[1], 16))/8192\n\n return converted_hex", "response": "Credit for conversion to itsnotlupus / vesync_wsproxy"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef resolve_updates(orig_list, updated_list):\n\n if updated_list is not None and updated_list:\n if orig_list is None:\n orig_list = updated_list\n else:\n # Add new devices not in list but found in the update\n for new_device in updated_list:\n was_found = False\n\n for device in orig_list:\n if new_device.cid == device.cid:\n was_found = True\n break\n\n if not was_found:\n orig_list.append(new_device)\n\n # Remove old devices in the list not found in the update\n for device in orig_list:\n should_remove = True\n\n for new_device in updated_list:\n if device.cid == new_device.cid:\n should_remove = False\n break\n\n if should_remove:\n orig_list.remove(device)\n\n # Call update on each device in the list\n [device.update() for device in orig_list]\n\n return orig_list", "response": "Merges changes from one list of devices against another"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef main (args):\n if (len(args) != 2):\n print(main.__doc__)\n sys.exit(1);\n\n # create the document\n doc = libsedml.SedDocument();\n doc.setLevel(1);\n doc.setVersion(1);\n\n # create a first model referencing an sbml file\n model = doc.createModel();\n model.setId(\"model1\");\n model.setSource(\"file.xml\");\n model.setLanguage(\"urn:sedml:sbml\");\n\n # create a second model modifying a variable of that other sbml file\n model = doc.createModel();\n model.setId(\"model2\");\n model.setSource(\"model1\");\n model.setLanguage(\"urn:sedml:sbml\");\n\n # change a paramerter 'k' to 0.1\n change = model.createChangeAttribute();\n change.setTarget(\"/sbml:sbml/sbml:model/sbml:listOfParameters/sbml:parameter[@id='k']/@value\");\n change.setNewValue(\"0.1\");\n\n # remove species 's1'\n remove = model.createRemoveXML();\n remove.setTarget(\"/sbml:sbml/sbml:model/sbml:listOfSpecies/sbml:species[@id='S1']\");\n\n # now for something tricky we want to update the initialConcentration of 'S2' to be\n # half what it was in the original model\n compute = model.createComputeChange();\n compute.setTarget(\"/sbml:sbml/sbml:model/sbml:listOfSpecies/sbml:species[@id="S2"]/@initialConcentration\");\n variable = compute.createVariable();\n variable.setId(\"S2\");\n variable.setModelReference(\"model1\");\n variable.setTarget(\"/sbml:sbml/sbml:model/sbml:listOfSpecies/sbml:species[@id='S2']\");\n compute.setMath(libsedml.parseFormula(\"S2 / 2\")); \n\n # create simulation\n tc = doc.createUniformTimeCourse();\n tc.setId(\"sim1\");\n tc.setInitialTime(0.0);\n tc.setOutputStartTime(0.0);\n tc.setOutputEndTime(10.0);\n tc.setNumberOfPoints(1000);\n # need to set the correct KISAO Term\n alg = tc.createAlgorithm();\n alg.setKisaoID(\"KISAO:0000019\");\n\n # create a task that uses the simulation and the model above\n task = doc.createTask();\n task.setId(\"task1\");\n task.setModelReference(\"model1\");\n task.setSimulationReference(\"sim1\");\n\n # add a DataGenerator to hold the output for time\n dg = doc.createDataGenerator();\n dg.setId(\"time\");\n dg.setName(\"time\");\n var = dg.createVariable();\n var.setId(\"v0\");\n var.setName(\"time\");\n var.setTaskReference(\"task1\");\n var.setSymbol(\"urn:sedml:symbol:time\");\n dg.setMath(libsedml.parseFormula(\"v0\"));\n\n # and one for S1\n dg = doc.createDataGenerator();\n dg.setId(\"S1\");\n dg.setName(\"S1\");\n var = dg.createVariable();\n var.setId(\"v1\");\n var.setName(\"S1\");\n var.setTaskReference(\"task1\");\n var.setTarget(\"/sbml:sbml/sbml:model/sbml:listOfSpecies/sbml:species[@id='S1']\");\n dg.setMath(libsedml.parseFormula(\"v1\"));\n\n # add a report\n report = doc.createReport();\n report.setId(\"r1\");\n report.setName(\"report 1\");\n set = report.createDataSet();\n set.setId(\"ds1\");\n set.setLabel(\"time\");\n set.setDataReference(\"time\");\n set = report.createDataSet();\n set.setId(\"ds2\");\n set.setLabel(\"S1\");\n set.setDataReference(\"S1\");\n\n # add a 2d plot\n plot = doc.createPlot2D();\n plot.setId(\"p1\");\n plot.setName(\"S1 Timecourse\");\n curve = plot.createCurve();\n curve.setId(\"c1\");\n curve.setName(\"S1\");\n curve.setLogX(False);\n curve.setLogY(False);\n curve.setXDataReference(\"time\");\n curve.setYDataReference(\"S1\");\n\n # add a 3D Plot\n plot2 = doc.createPlot3D();\n plot2.setId(\"p2\");\n plot2.setName(\"dunno\");\n surf = plot2.createSurface();\n surf.setId(\"surf1\");\n surf.setName(\"S1\");\n surf.setLogX(False);\n surf.setLogY(False);\n surf.setLogZ(False);\n surf.setXDataReference(\"time\");\n surf.setYDataReference(\"S1\");\n surf.setZDataReference(\"S1\");\n\n # write the document\n libsedml.writeSedML(doc, args[1]);", "response": "create_sedml output - filename\n"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngives a vcf file return a dictionary of profile strings for each sample in the vcf", "response": "def get_profiles(adapter, vcf_file):\n\n \"\"\"Given a vcf, get a profile string for each sample in the vcf\n based on the profile variants in the database\n\n Args:\n adapter(MongoAdapter): Adapter to mongodb\n vcf_file(str): Path to vcf file\n\n Returns:\n profiles (dict(str)): The profiles (given as strings) for each sample\n in vcf.\n\n \"\"\"\n vcf = get_file_handle(vcf_file)\n individuals = vcf.samples\n profiles = {individual: [] for individual in individuals}\n\n for profile_variant in adapter.profile_variants():\n\n ref = profile_variant['ref']\n alt = profile_variant['alt']\n\n pos = profile_variant['pos']\n end = pos + 1\n chrom = profile_variant['chrom']\n\n region = f\"{chrom}:{pos}-{end}\"\n\n #Find variants in region\n\n found_variant = False\n for variant in vcf(region):\n\n variant_id = get_variant_id(variant)\n\n #If variant id i.e. chrom_pos_ref_alt matches\n if variant_id == profile_variant['_id']:\n found_variant = True\n #find genotype for each individual in vcf\n for i, individual in enumerate(individuals):\n\n genotype = GENOTYPE_MAP[variant.gt_types[i]]\n if genotype == 'hom_alt':\n gt_str = f\"{alt}{alt}\"\n elif genotype == 'het':\n gt_str = f\"{ref}{alt}\"\n else:\n gt_str = f\"{ref}{ref}\"\n\n #Append genotype to profile string of individual\n profiles[individual].append(gt_str)\n\n #Break loop if variant is found in region\n break\n\n #If no call was found for variant, give all samples a hom ref genotype\n if not found_variant:\n for individual in individuals: profiles[individual].append(f\"{ref}{ref}\")\n\n\n return profiles"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a list of similar samples for each sample in the DB.", "response": "def profile_match(adapter, profiles, hard_threshold=0.95, soft_threshold=0.9):\n\n \"\"\"\n given a dict of profiles, searches through all the samples in the DB\n for a match. If a matching sample is found an exception is raised,\n and the variants will not be loaded into the database.\n\n Args:\n adapter (MongoAdapter): Adapter to mongodb\n profiles (dict(str)): The profiles (given as strings) for each sample in vcf.\n hard_threshold(float): Rejects load if hamming distance above this is found\n soft_threshold(float): Stores similar samples if hamming distance above this is found\n\n Returns:\n matches(dict(list)): list of similar samples for each sample in vcf.\n \"\"\"\n matches = {sample: [] for sample in profiles.keys()}\n for case in adapter.cases():\n\n for individual in case['individuals']:\n\n for sample in profiles.keys():\n\n if individual.get('profile'):\n\n similarity = compare_profiles(\n profiles[sample], individual['profile']\n )\n\n if similarity >= hard_threshold:\n\n msg = (\n f\"individual {sample} has a {similarity} similarity \"\n f\"with individual {individual['ind_id']} in case \"\n f\"{case['case_id']}\"\n )\n LOG.critical(msg)\n\n #Raise some exception\n raise ProfileError\n\n if similarity >= soft_threshold:\n\n match = f\"{case['case_id']}.{individual['ind_id']}\"\n matches[sample].append(match)\n\n return matches"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngives two profiles, determine the ratio of similarity, i.e. the hamming distance between the strings. Args: profile1/2 (str): profile string Returns: similarity_ratio (float): the ratio of similiarity (0-1)", "response": "def compare_profiles(profile1, profile2):\n\n \"\"\"\n Given two profiles, determine the ratio of similarity, i.e.\n the hamming distance between the strings.\n\n Args:\n profile1/2 (str): profile string\n Returns:\n similarity_ratio (float): the ratio of similiarity (0-1)\n \"\"\"\n\n length = len(profile1)\n\n profile1 = np.array(list(profile1))\n profile2 = np.array(list(profile2))\n\n similarity_array = profile1 == profile2\n\n matches = np.sum(similarity_array)\n\n similarity_ratio = matches/length\n\n return similarity_ratio"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nupdating the profile string for all samples in the specified vcf", "response": "def update_profiles(adapter):\n\n \"\"\"\n For all cases having vcf_path, update the profile string for the samples\n\n Args:\n adapter (MongoAdapter): Adapter to mongodb\n\n \"\"\"\n\n for case in adapter.cases():\n\n #If the case has a vcf_path, get the profiles and update the\n #case with new profiled individuals.\n if case.get('profile_path'):\n\n profiles = get_profiles(adapter, case['profile_path'])\n profiled_individuals = deepcopy(case['individuals'])\n\n for individual in profiled_individuals:\n ind_id = individual['ind_id']\n try:\n profile = profiles[ind_id]\n individual['profile'] = profile\n\n except KeyError:\n LOG.warning(f\"sample IDs in vcf does not match for case {case['case_id']}\")\n\n updated_case = deepcopy(case)\n\n updated_case['individuals'] = profiled_individuals\n\n adapter.add_case(updated_case, update=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a dictionary with the number of hamming distances for all the sample profiles in the database.", "response": "def profile_stats(adapter, threshold = 0.9):\n\n \"\"\"\n Compares the pairwise hamming distances for all the sample profiles in\n the database. Returns a table of the number of distances within given\n ranges.\n\n Args:\n adapter (MongoAdapter): Adapter to mongodb\n threshold (float): If any distance is found above this threshold\n a warning will be given, stating the two matching samples.\n\n Returns:\n distance_dict (dict): dictionary with ranges as keys, and the number\n of distances that are within these ranges as values.\n\n \"\"\"\n profiles = []\n samples = []\n\n #Instatiate the distance dictionary with a count 0 for all the ranges\n distance_dict = {key: 0 for key in HAMMING_RANGES.keys()}\n\n for case in adapter.cases():\n\n for individual in case['individuals']:\n\n if individual.get('profile'):\n #Make sample name .\n sample_id = f\"{case['case_id']}.{individual['ind_id']}\"\n ind_profile = individual['profile']\n\n #Numpy array to hold all the distances for this samples profile\n distance_array = np.array([], dtype=np.float)\n\n for sample, profile in zip(samples, profiles):\n\n #Get distance and append to distance array\n distance = compare_profiles(ind_profile, profile)\n distance_array = np.append(distance_array, distance)\n\n #Issue warning if above threshold\n if distance >= threshold:\n LOG.warning(f\"{sample_id} is {distance} similar to {sample}\")\n\n #Check number of distances in each range and add to distance_dict\n for key,range in HAMMING_RANGES.items():\n\n #Calculate the number of hamming distances found within the\n #range for current individual\n distance_dict[key] += np.sum(\n (distance_array >= range[0]) & (distance_array < range[1])\n )\n\n #Append profile and sample_id for this sample for the next\n #iteration\n profiles.append(ind_profile)\n samples.append(sample_id)\n\n return distance_dict"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncalculates the purity score for the given labels.", "response": "def purity(labels, true_labels):\n \"\"\"\n Calculates the purity score for the given labels.\n\n Args:\n labels (array): 1D array of integers\n true_labels (array): 1D array of integers - true labels\n\n Returns:\n purity score - a float bewteen 0 and 1. Closer to 1 is better.\n \"\"\"\n purity = 0.0\n for i in set(labels):\n indices = (labels==i)\n true_clusters = true_labels[indices]\n if len(true_clusters)==0:\n continue\n counts = Counter(true_clusters)\n lab, count = counts.most_common()[0]\n purity += count\n return float(purity)/len(labels)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef mdl(ll, k, data):\n\n \"\"\"\n N - no. of genes\n n - no. of cells \n k - no. of cell types\n R - sum(Dataset) i.e. total no. of reads\n\n function TotCost = TotBits(N,m,p,R,C)\n # C is the cost from the cost function\n TotCost = C + (N*m + m*p)*(log(R/(N*p)));\n \"\"\"\n N, m = data.shape\n cost = ll + (N*m + m*k)*(np.log(data.sum()/(N*k)))\n return cost", "response": "Returns the minimum description length score of the model given its log - likelihood and k the number of cell types."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef find_nb_genes(data):\n data_means = data.mean(1)\n data_vars = data.var(1)\n nb_indices = data_means < 0.9*data_vars\n return nb_indices", "response": "Finds the indices of all genes in the dataset that have a mean < 0. 9 variance. Returns an array of booleans."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef log_ncr(a, b):\n val = gammaln(a+1) - gammaln(a-b+1) - gammaln(b+1)\n return val", "response": "Returns the log of the n - cr function given that a < b."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef nb_ll(data, P, R):\n # TODO: include factorial...\n #data = data + eps\n genes, cells = data.shape\n clusters = P.shape[1]\n lls = np.zeros((cells, clusters))\n for c in range(clusters):\n P_c = P[:,c].reshape((genes, 1))\n R_c = R[:,c].reshape((genes, 1))\n # don't need constant factors...\n ll = gammaln(R_c + data) - gammaln(R_c) #- gammaln(data + 1)\n ll += data*np.log(P_c) + xlog1py(R_c, -P_c)\n #new_ll = np.sum(nbinom.logpmf(data, R_c, P_c), 0)\n lls[:,c] = ll.sum(0)\n return lls", "response": "Returns the negative binomial log - likelihood of the data."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef zinb_ll(data, P, R, Z):\n lls = nb_ll(data, P, R)\n clusters = P.shape[1]\n for c in range(clusters):\n pass\n return lls", "response": "Returns the zero - inflated negative binomial log - likelihood of the data."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef nb_ll_row(params, data_row):\n p = params[0]\n r = params[1]\n n = len(data_row)\n ll = np.sum(gammaln(data_row + r)) - np.sum(gammaln(data_row + 1))\n ll -= n*gammaln(r)\n ll += np.sum(data_row)*np.log(p)\n ll += n*r*np.log(1-p)\n return -ll", "response": "Returns the negative LL of a single row."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef nb_fit(data, P_init=None, R_init=None, epsilon=1e-8, max_iters=100):\n means = data.mean(1)\n variances = data.var(1)\n if (means > variances).any():\n raise ValueError(\"For NB fit, means must be less than variances\")\n genes, cells = data.shape\n # method of moments\n P = 1.0 - means/variances\n R = means*(1-P)/P\n for i in range(genes):\n result = minimize(nb_ll_row, [P[i], R[i]], args=(data[i,:],),\n bounds = [(0, 1), (eps, None)])\n params = result.x\n P[i] = params[0]\n R[i] = params[1]\n #R[i] = fsolve(nb_r_deriv, R[i], args = (data[i,:],))\n #P[i] = data[i,:].mean()/(data[i,:].mean() + R[i])\n return P,R", "response": "Fits the NB distribution to data using method of moments."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef nb_cluster(data, k, P_init=None, R_init=None, assignments=None, means=None, max_iters=10):\n genes, cells = data.shape\n if P_init is None:\n P_init = np.random.random((genes, k))\n if R_init is None:\n R_init = np.random.randint(1, data.max(), (genes, k))\n R_init = R_init.astype(float)\n if assignments is None:\n _, assignments = kmeans_pp(data, k, means)\n means = np.zeros((genes, k))\n #assignments = np.array([np.random.randint(0,k) for i in range(cells)])\n old_assignments = np.copy(assignments)\n # If mean > variance, then fall back to Poisson, since NB\n # distribution can't handle that case.\n for i in range(max_iters):\n # estimate params from assigned cells\n nb_gene_indices = fit_cluster(data, assignments, k, P_init, R_init, means)\n # re-calculate assignments\n lls = nb_ll(data[nb_gene_indices, :], P_init[nb_gene_indices,:], R_init[nb_gene_indices,:])\n lls += pois_ll.poisson_ll(data[~nb_gene_indices,:], means[~nb_gene_indices,:])\n # set NB params to failure values\n P_init[~nb_gene_indices,:] = 0\n R_init[~nb_gene_indices,:] = np.inf\n for c in range(cells):\n assignments[c] = np.argmax(lls[c,:])\n if np.equal(assignments,old_assignments).all():\n break\n old_assignments = np.copy(assignments)\n return assignments, P_init, R_init", "response": "This function performs negative binomial clustering on the given data."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef fit_cluster(data, assignments, k, P_init, R_init, means):\n for c in range(k):\n if data[:,assignments==c].shape[1] == 0:\n _, assignments = kmeans_pp(data, k)\n genes, cells = data.shape\n nb_gene_indices = np.array([True for i in range(genes)])\n for c in range(k):\n c_data = data[:,assignments==c]\n nb_gene_indices = nb_gene_indices & find_nb_genes(c_data)\n for c in range(k):\n c_data = data[:,assignments==c]\n nb_genes = c_data[nb_gene_indices,:]\n poisson_genes = c_data[~nb_gene_indices, :]\n P_init[nb_gene_indices, c], R_init[nb_gene_indices, c] = nb_fit(nb_genes)\n means[~nb_gene_indices, c] = poisson_genes.mean(1)\n return nb_gene_indices", "response": "Fits NB / poisson params to a cluster."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalculating the zero - inflated Poisson log - likelihood for each cell in the cluster.", "response": "def zip_ll(data, means, M):\n \"\"\"\n Calculates the zero-inflated Poisson log-likelihood.\n\n Args:\n data (array): genes x cells\n means (array): genes x k\n M (array): genes x k - this is the zero-inflation parameter.\n\n Returns:\n cells x k array of log-likelihood for each cell/cluster pair.\n \"\"\"\n genes, cells = data.shape\n clusters = means.shape[1]\n ll = np.zeros((cells, clusters))\n d0 = (data==0)\n d1 = (data>0)\n for i in range(clusters):\n means_i = np.tile(means[:,i], (cells, 1))\n means_i = means_i.transpose()\n L_i = np.tile(M[:,i], (cells, 1))\n L_i = L_i.transpose()\n ll_0 = np.log(L_i + (1 - L_i)*np.exp(-means_i))\n ll_0 = np.where((L_i==0) & (means_i==0), -means_i, ll_0)\n # not including constant factors\n ll_1 = np.log(1 - L_i) + xlogy(data, means_i) - means_i\n ll_0 = np.where(d0, ll_0, 0.0)\n ll_1 = np.where(d1, ll_1, 0.0)\n ll[:,i] = np.sum(ll_0 + ll_1, 0)\n return ll"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the negative log - likelihood of a row given ZIP data.", "response": "def zip_ll_row(params, data_row):\n \"\"\"\n Returns the negative log-likelihood of a row given ZIP data.\n\n Args:\n params (list): [lambda zero-inf]\n data_row (array): 1d array\n\n Returns:\n negative log-likelihood\n \"\"\"\n l = params[0]\n pi = params[1]\n d0 = (data_row==0)\n likelihood = d0*pi + (1-pi)*poisson.pmf(data_row, l)\n return -np.log(likelihood+eps).sum()"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nmigrates an old loqusdb instance to 1. 0 WorkItem", "response": "def migrate_database(adapter):\n \"\"\"Migrate an old loqusdb instance to 1.0\n \n Args:\n adapter\n \n Returns:\n nr_updated(int): Number of variants that where updated\n \"\"\"\n \n all_variants = adapter.get_variants()\n nr_variants = all_variants.count()\n nr_updated = 0\n with progressbar(all_variants, label=\"Updating variants\", length=nr_variants) as bar:\n for variant in bar:\n # Do not update if the variants have the correct format\n if 'chrom' in variant:\n continue\n nr_updated += 1\n splitted_id = variant['_id'].split('_')\n \n chrom = splitted_id[0]\n start = int(splitted_id[1])\n ref = splitted_id[2]\n alt = splitted_id[3]\n \n # Calculate end\n end = start + (max(len(ref), len(alt)) - 1)\n \n adapter.db.variant.find_one_and_update(\n {'_id': variant['_id']},\n {\n '$set': {\n 'chrom': chrom,\n 'start': start,\n 'end': end\n }\n }\n )\n \n return nr_updated"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncalculating the bounding box of each node in each cluster.", "response": "def calculate_bounding_box(data):\n \"\"\"\n Returns a 2 x m array indicating the min and max along each\n dimension.\n \"\"\"\n mins = data.min(0)\n maxes = data.max(0)\n return mins, maxes"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncalculate the gap score and sk for a given set of clustered income classes.", "response": "def calculate_gap(data, clustering, km, B=50, **kwargs):\n \"\"\"\n See: https://datasciencelab.wordpress.com/2013/12/27/finding-the-k-in-k-means-clustering/\n\n https://web.stanford.edu/~hastie/Papers/gap.pdf\n\n Returns two results: the gap score, and s_k.\n \"\"\"\n k = len(set(clustering))\n Wk = km.inertia_\n mins, maxes = calculate_bounding_box(data)\n Wk_est = []\n for i in range(B):\n data_sample = (maxes-mins)*np.random.random(data.shape) + mins\n km_b = KMeans(k)\n km_b.fit_predict(data_sample)\n Wk_est.append(km_b.inertia_)\n Wk_est = np.log(np.array(Wk_est))\n Wk_mean = np.mean(Wk_est)\n Wk_std = np.std(Wk_est)\n gap = Wk_mean - np.log(Wk)\n sk = np.sqrt(1 + 1.0/B)*Wk_std\n return gap, sk"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nruns gap score for all k from k_min to k_max.", "response": "def run_gap_k_selection(data, k_min=1, k_max=50, B=5,\n skip=5, **kwargs):\n \"\"\"\n Runs gap score for all k from k_min to k_max.\n \"\"\"\n if k_min == k_max:\n return k_min\n gap_vals = []\n sk_vals = []\n k_range = list(range(k_min, k_max, skip))\n min_k = 0\n min_i = 0\n for i, k in enumerate(k_range):\n km = KMeans(k)\n clusters = km.fit_predict(data)\n gap, sk = calculate_gap(data, clusters, km, B=B)\n if len(gap_vals) > 1:\n if gap_vals[-1] >= gap - (skip+1)*sk:\n min_i = i\n min_k = k_range[i-1]\n break\n #return k_range[-1], gap_vals, sk_vals\n gap_vals.append(gap)\n sk_vals.append(sk)\n if min_k == 0:\n min_k = k_max\n if skip == 1:\n return min_k, gap_vals, sk_vals\n gap_vals = []\n sk_vals = []\n for k in range(min_k - skip, min_k + skip):\n km = KMeans(k)\n clusters = km.fit_predict(data)\n gap, sk = calculate_gap(data, clusters, km, B=B)\n if len(gap_vals) > 1:\n if gap_vals[-1] >= gap - sk:\n min_k = k-1\n return min_k, gap_vals, sk_vals\n gap_vals.append(gap)\n sk_vals.append(sk)\n return k, gap_vals, sk_vals"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\nasync def _request(\n self,\n method: str,\n url: str,\n *,\n headers: dict = None,\n params: dict = None,\n json: dict = None) -> dict:\n \"\"\"Make a request against the RainMachine device.\"\"\"\n if not headers:\n headers = {}\n\n try:\n async with self._websession.request(method, url, headers=headers,\n params=params,\n json=json) as resp:\n resp.raise_for_status()\n data = await resp.json(content_type=None)\n return data\n except ClientError as err:\n raise RequestError(\n 'Error requesting data from {}: {}'.format(url, err))", "response": "Make a request against the RainMachine device."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_devices(self) -> list:\n\n if not self.enabled:\n return None\n\n self.in_process = True\n\n response, _ = helpers.call_api(\n '/cloud/v1/deviceManaged/devices',\n 'post',\n headers=helpers.req_headers(self),\n json=helpers.req_body(self, 'devicelist')\n )\n\n if response and helpers.check_response(response, 'get_devices'):\n if 'result' in response and 'list' in response['result']:\n device_list = response['result']['list']\n outlets, switches, fans = self.process_devices(device_list)\n else:\n logger.error('Device list in response not found')\n else:\n logger.error('Error retrieving device list')\n\n self.in_process = False\n\n return (outlets, switches, fans)", "response": "Return list of VeSync devices"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nlogging in with username and password.", "response": "def login(self) -> bool:\n \"\"\"Return True if log in request succeeds\"\"\"\n user_check = isinstance(self.username, str) and len(self.username) > 0\n pass_check = isinstance(self.password, str) and len(self.password) > 0\n\n if user_check and pass_check:\n response, _ = helpers.call_api(\n '/cloud/v1/user/login',\n 'post',\n json=helpers.req_body(self, 'login')\n )\n\n if response and helpers.check_response(response, 'login'):\n self.token = response['result']['token']\n self.account_id = response['result']['accountID']\n self.enabled = True\n\n return True\n else:\n logger.error('Error logging in with username and password')\n return False\n\n else:\n if user_check is False:\n logger.error('Username invalid')\n if pass_check is False:\n logger.error('Password invalid')\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfetch updated information about devices", "response": "def update(self):\n \"\"\"Fetch updated information about devices\"\"\"\n\n if self.device_time_check():\n\n if not self.in_process:\n outlets, switches, fans = self.get_devices()\n\n self.outlets = helpers.resolve_updates(self.outlets, outlets)\n self.switches = helpers.resolve_updates(\n self.switches, switches)\n self.fans = helpers.resolve_updates(self.fans, fans)\n\n self.last_update_ts = time.time()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfetch updated energy information about devices", "response": "def update_energy(self, bypass_check=False):\n \"\"\"Fetch updated energy information about devices\"\"\"\n for outlet in self.outlets:\n outlet.update_energy(bypass_check)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngiving a data matrix, this returns the per-gene fit error for the Poisson, Normal, and Log-Normal distributions. Args: Dat (array): numpy array with shape (genes, cells) Returns: d (dict): 'poiss', 'norm', 'lognorm' give the fit error for each distribution.", "response": "def DistFitDataset(Dat):\n \"\"\"\n Given a data matrix, this returns the per-gene fit error for the\n Poisson, Normal, and Log-Normal distributions.\n\n Args:\n Dat (array): numpy array with shape (genes, cells)\n\n Returns:\n d (dict): 'poiss', 'norm', 'lognorm' give the fit error for each distribution.\n \"\"\"\n #Assumes data to be in the form of a numpy matrix \n (r,c) = Dat.shape\n Poiss = np.zeros(r)\n Norm = np.zeros(r)\n LogNorm = np.zeros(r)\n for i in range(r):\n temp = GetDistFitError(Dat[i])\n Poiss[i] = temp['poiss']\n Norm[i] = temp['norm']\n LogNorm[i] = temp['lognorm']\n d = {}\n d['poiss'] = Poiss\n d['norm'] = Norm\n d['lognorm'] = LogNorm\n return d"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndeletes a case and all of its variants from the database.", "response": "def delete(adapter, case_obj, update=False, existing_case=False):\n \"\"\"Delete a case and all of it's variants from the database.\n \n Args:\n adapter: Connection to database\n case_obj(models.Case)\n update(bool): If we are in the middle of an update\n existing_case(models.Case): If something failed during an update we need to revert\n to the original case\n \n \"\"\"\n # This will overwrite the updated case with the previous one\n if update:\n adapter.add_case(existing_case)\n else:\n adapter.delete_case(case_obj)\n\n for file_type in ['vcf_path','vcf_sv_path']:\n if not case_obj.get(file_type):\n continue\n variant_file = case_obj[file_type]\n # Get a cyvcf2.VCF object\n vcf_obj = get_vcf(variant_file)\n\n delete_variants(\n adapter=adapter,\n vcf_obj=vcf_obj,\n case_obj=case_obj,\n )"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndeleting variants for a given VCF file.", "response": "def delete_variants(adapter, vcf_obj, case_obj, case_id=None):\n \"\"\"Delete variants for a case in the database\n \n Args:\n adapter(loqusdb.plugins.Adapter)\n vcf_obj(iterable(dict))\n ind_positions(dict)\n case_id(str)\n \n Returns:\n nr_deleted (int): Number of deleted variants\n \"\"\"\n case_id = case_id or case_obj['case_id']\n nr_deleted = 0\n start_deleting = datetime.now()\n chrom_time = datetime.now()\n current_chrom = None\n new_chrom = None\n \n for variant in vcf_obj:\n formated_variant = build_variant(\n variant=variant,\n case_obj=case_obj,\n case_id=case_id,\n )\n \n if not formated_variant:\n continue\n \n new_chrom = formated_variant.get('chrom')\n adapter.delete_variant(formated_variant)\n nr_deleted += 1\n \n if not current_chrom:\n LOG.info(\"Start deleting chromosome {}\".format(new_chrom))\n current_chrom = new_chrom\n chrom_time = datetime.now()\n continue\n \n if new_chrom != current_chrom:\n LOG.info(\"Chromosome {0} done\".format(current_chrom))\n LOG.info(\"Time to delete chromosome {0}: {1}\".format(\n current_chrom, datetime.now()-chrom_time))\n LOG.info(\"Start deleting chromosome {0}\".format(new_chrom))\n current_chrom = new_chrom\n\n\n return nr_deleted"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nannotates the variants in a VCF", "response": "def annotate(ctx, variant_file, sv):\n \"\"\"Annotate the variants in a VCF\n\n \"\"\"\n adapter = ctx.obj['adapter']\n\n variant_path = os.path.abspath(variant_file)\n\n expected_type = 'snv'\n if sv:\n expected_type = 'sv'\n\n if 'sv':\n nr_cases = adapter.nr_cases(sv_cases=True)\n else:\n nr_cases = adapter.nr_cases(snv_cases=True)\n LOG.info(\"Found {0} {1} cases in database\".format(nr_cases, expected_type))\n\n vcf_obj = get_file_handle(variant_path)\n add_headers(vcf_obj, nr_cases=nr_cases, sv=sv)\n # Print the headers\n for header_line in vcf_obj.raw_header.split('\\n'):\n if len(header_line) == 0:\n continue\n click.echo(header_line)\n\n start_inserting = datetime.now()\n \n if sv:\n annotated_variants = annotate_svs(adapter, vcf_obj)\n else:\n annotated_variants = annotate_snvs(adapter, vcf_obj)\n # try:\n for variant in annotated_variants:\n click.echo(str(variant).rstrip())"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\nasync def find(self, *tracking_numbers: str) -> list:\n data = {'data': [{'num': num} for num in tracking_numbers]}\n tracking_resp = await self._request('post', API_URL_TRACK, json=data)\n\n print(tracking_resp)\n\n if not tracking_resp.get('dat'):\n raise InvalidTrackingNumberError('Invalid data')\n\n packages = []\n for info in tracking_resp['dat']:\n package_info = info.get('track', {})\n\n if not package_info:\n continue\n\n kwargs = {\n 'destination_country': package_info.get('c'),\n 'info_text': package_info.get('z0', {}).get('z'),\n 'location': package_info.get('z0', {}).get('c'),\n 'origin_country': package_info.get('b'),\n 'package_type': package_info.get('d', 0),\n 'status': package_info.get('e', 0),\n 'tracking_info_language': package_info.get('ln1', 'Unknown')\n }\n packages.append(Package(info['no'], **kwargs))\n return packages", "response": "Get tracking info for one or more tracking numbers."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef binarize(qualitative):\n thresholds = qualitative.min(1) + (qualitative.max(1) - qualitative.min(1))/2.0\n binarized = qualitative > thresholds.reshape((len(thresholds), 1)).repeat(8,1)\n return binarized.astype(int)", "response": "binarizes an expression dataset."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndoing qualNorm but returns a filtered gene set based on a p - value threshold.", "response": "def qualNorm_filter_genes(data, qualitative, pval_threshold=0.05, smoothing=1e-5, eps=1e-5):\n \"\"\"\n Does qualNorm but returns a filtered gene set, based on a p-value threshold.\n \"\"\"\n genes, cells = data.shape\n clusters = qualitative.shape[1]\n output = np.zeros((genes, clusters))\n missing_indices = []\n genes_included = []\n qual_indices = []\n thresholds = qualitative.min(1) + (qualitative.max(1) - qualitative.min(1))/2.0\n pvals = np.zeros(genes)\n for i in range(genes):\n if qualitative[i,:].max() == -1 and qualitative[i,:].min() == -1:\n missing_indices.append(i)\n continue\n qual_indices.append(i)\n threshold = thresholds[i]\n data_i = data[i,:]\n if sparse.issparse(data):\n data_i = data_i.toarray().flatten()\n assignments, means = poisson_cluster(data_i.reshape((1, cells)), 2)\n means = means.flatten()\n high_i = 1\n low_i = 0\n if means[0]>means[1]:\n high_i = 0\n low_i = 1\n # do a p-value test\n p_val = poisson_test(data_i[assignments==low_i], data_i[assignments==high_i], smoothing=smoothing)\n pvals[i] = p_val\n if p_val <= pval_threshold:\n genes_included.append(i)\n else:\n continue\n high_mean = np.median(data_i[assignments==high_i])\n low_mean = np.median(data_i[assignments==low_i]) + eps\n for k in range(clusters):\n if qualitative[i,k]>threshold:\n output[i,k] = high_mean\n else:\n output[i,k] = low_mean\n output = output[genes_included,:]\n pvals = pvals[genes_included]\n return output, pvals, genes_included"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef qualNorm(data, qualitative):\n genes, cells = data.shape\n clusters = qualitative.shape[1]\n output = np.zeros((genes, clusters))\n missing_indices = []\n qual_indices = []\n thresholds = qualitative.min(1) + (qualitative.max(1) - qualitative.min(1))/2.0\n for i in range(genes):\n if qualitative[i,:].max() == -1 and qualitative[i,:].min() == -1:\n missing_indices.append(i)\n continue\n qual_indices.append(i)\n threshold = thresholds[i]\n data_i = data[i,:]\n if sparse.issparse(data):\n data_i = data_i.toarray().flatten()\n assignments, means = poisson_cluster(data_i.reshape((1, cells)), 2)\n means = means.flatten()\n high_i = 1\n low_i = 0\n if means[0]>means[1]:\n high_i = 0\n low_i = 1\n high_mean = np.median(data_i[assignments==high_i])\n low_mean = np.median(data_i[assignments==low_i])\n for k in range(clusters):\n if qualitative[i,k]>threshold:\n output[i,k] = high_mean\n else:\n output[i,k] = low_mean\n if missing_indices:\n assignments, means = poisson_cluster(data[qual_indices, :], clusters, output[qual_indices, :], max_iters=1)\n for ind in missing_indices:\n for k in range(clusters):\n if len(assignments==k)==0:\n output[ind, k] = data[ind,:].mean()\n else:\n output[ind, k] = data[ind, assignments==k].mean()\n return output", "response": "Generates starting positions for state estimation or qualitative data."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef qualNormGaussian(data, qualitative):\n genes, cells = data.shape\n clusters = qualitative.shape[1]\n output = np.zeros((genes, clusters))\n missing_indices = []\n qual_indices = []\n for i in range(genes):\n if qualitative[i,:].max() == -1 and qualitative[i,:].min() == -1:\n missing_indices.append(i)\n continue\n qual_indices.append(i)\n threshold = (qualitative[i,:].max() - qualitative[i,:].min())/2.0\n kmeans = KMeans(n_clusters = 2).fit(data[i,:].reshape((1, cells)))\n assignments = kmeans.labels_\n means = kmeans.cluster_centers_\n high_mean = means.max()\n low_mean = means.min()\n for k in range(clusters):\n if qualitative[i,k]>threshold:\n output[i,k] = high_mean\n else:\n output[i,k] = low_mean\n if missing_indices:\n #generating centers for missing indices \n M_init = output[qual_indices, :]\n kmeans = KMeans(n_clusters = 2, init = M_init, max_iter = 1).fit(data[qual_indices, :])\n assignments = kmeans.labels_\n #assignments, means = poisson_cluster(data[qual_indices, :], clusters, output[qual_indices, :], max_iters=1)\n for ind in missing_indices:\n for k in range(clusters):\n output[ind, k] = np.mean(data[ind, assignments==k])\n # TODO: assign to closest\n return output", "response": "Generates starting positions for state estimation or qualitative data for a given gene."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget the directory of the current script s contents.", "response": "def script_dir(pyobject, follow_symlinks=True):\n \"\"\"Get current script's directory\n\n Args:\n pyobject (Any): Any Python object in the script\n follow_symlinks (Optional[bool]): Follow symlinks or not. Defaults to True.\n\n Returns:\n str: Current script's directory\n \"\"\"\n if getattr(sys, 'frozen', False): # py2exe, PyInstaller, cx_Freeze\n path = abspath(sys.executable)\n else:\n path = inspect.getabsfile(pyobject)\n if follow_symlinks:\n path = realpath(path)\n return dirname(path)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef script_dir_plus_file(filename, pyobject, follow_symlinks=True):\n return join(script_dir(pyobject, follow_symlinks), filename)", "response": "Get current script s directory and then append a filename"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef identity(ctx, variant_id):\n if not variant_id:\n LOG.warning(\"Please provide a variant id\")\n ctx.abort()\n\n adapter = ctx.obj['adapter']\n version = ctx.obj['version']\n \n LOG.info(\"Search variants {0}\".format(adapter))\n \n result = adapter.get_clusters(variant_id)\n if result.count() == 0:\n LOG.info(\"No hits for variant %s\", variant_id)\n return\n \n for res in result:\n click.echo(res)", "response": "Check how well SVs are working in the database"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef registration_authority_entity_id(self):\n\n if ATTR_ENTITY_REGISTRATION_AUTHORITY in self.raw:\n try:\n return self.raw[\n ATTR_ENTITY_REGISTRATION_AUTHORITY][\n ATTR_ENTITY_REGISTRATION_AUTHORITY_ENTITY_ID][\n ATTR_DOLLAR_SIGN]\n except KeyError:\n pass", "response": "Return the register entity id for the given resource."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the ELF - code for the specified resource.", "response": "def legal_form(self):\n \"\"\"In some cases, the legal form is stored in the JSON-data.\n In other cases, an ELF-code, consisting of mix of exactly\n four letters and numbers are stored. This ELF-code\n can be looked up in a registry where a code maps to\n a organizational type. ELF-codes are not unique,\n it can reoccur under different names in different\n countries\"\"\"\n\n if ATTR_ENTITY_LEGAL_FORM in self.raw:\n try:\n return LEGAL_FORMS[self.legal_jurisdiction][\n self.raw[ATTR_ENTITY_LEGAL_FORM][\n ATTR_ENTITY_LEGAL_FORM_CODE][ATTR_DOLLAR_SIGN]\n ]\n except KeyError:\n legal_form = self.raw[\n ATTR_ENTITY_LEGAL_FORM][\n ATTR_ENTITY_LEGAL_FORM_CODE][ATTR_DOLLAR_SIGN]\n\n if len(legal_form) == 4:\n # If this is returned, the ELF should\n # be added to the constants.\n return 'ELF code: ' + legal_form\n else:\n return legal_form"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef valid_child_records(self):\n child_lei = list()\n\n \"\"\"Loop through data to find a valid record. Return list of LEI.\"\"\"\n for d in self.raw['data']:\n\n # We're not very greedy here, but it seems some records have\n # lapsed even through the issuer is active\n if d['attributes']['relationship']['status'] in ['ACTIVE']:\n child_lei.append(\n d['attributes']['relationship']['startNode']['id'])\n\n return child_lei", "response": "Loop through data to find a valid record. Return list of LEI."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef annotate_variant(variant, var_obj=None):\n if var_obj:\n \n variant.INFO['Obs'] = var_obj['observations']\n if var_obj.get('homozygote'):\n variant.INFO['Hom'] = var_obj['homozygote']\n if var_obj.get('hemizygote'):\n variant.INFO['Hem'] = var_obj['hemizygote']\n \n return variant", "response": "Annotate a cyvcf variant with observations\n \n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nannotating an SNV variant in a new CNV record", "response": "def annotate_snv(adpter, variant):\n \"\"\"Annotate an SNV/INDEL variant\n \n Args:\n adapter(loqusdb.plugin.adapter)\n variant(cyvcf2.Variant)\n \"\"\"\n variant_id = get_variant_id(variant)\n variant_obj = adapter.get_variant(variant={'_id':variant_id})\n \n annotated_variant = annotated_variant(variant, variant_obj)\n return annotated_variant"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nannotate all SV variants in a VCF", "response": "def annotate_svs(adapter, vcf_obj):\n \"\"\"Annotate all SV variants in a VCF\n \n Args:\n adapter(loqusdb.plugin.adapter)\n vcf_obj(cyvcf2.VCF)\n \n Yields:\n variant(cyvcf2.Variant)\n \"\"\"\n for nr_variants, variant in enumerate(vcf_obj, 1):\n variant_info = get_coords(variant)\n match = adapter.get_structural_variant(variant_info)\n if match:\n annotate_variant(variant, match)\n yield variant"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef annotate_snvs(adapter, vcf_obj):\n variants = {}\n \n for nr_variants, variant in enumerate(vcf_obj, 1):\n # Add the variant to current batch\n variants[get_variant_id(variant)] = variant\n # If batch len == 1000 we annotate the batch\n if (nr_variants % 1000) == 0:\n \n for var_obj in adapter.search_variants(list(variants.keys())):\n var_id = var_obj['_id']\n if var_id in variants:\n annotate_variant(variants[var_id], var_obj)\n\n for variant_id in variants:\n yield variants[variant_id]\n \n variants = {}\n \n for var_obj in adapter.search_variants(list(variants.keys())):\n var_id = var_obj['_id']\n if var_id in variants:\n annotate_variant(variants[var_id], var_obj)\n\n for variant_id in variants:\n yield variants[variant_id]", "response": "Annotate all variants in a VCF in a VCF\n \n "} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates a Geometry from a GeoJSON object.", "response": "def from_geojson(geojson, srid=4326):\n \"\"\"\n Create a Geometry from a GeoJSON. The SRID can be overridden from the\n expected 4326.\n \"\"\"\n type_ = geojson[\"type\"].lower()\n if type_ == \"geometrycollection\":\n geometries = []\n for geometry in geojson[\"geometries\"]:\n geometries.append(Geometry.from_geojson(geometry, srid=None))\n return GeometryCollection(geometries, srid)\n elif type_ == \"point\":\n return Point(geojson[\"coordinates\"], srid=srid)\n elif type_ == \"linestring\":\n return LineString(geojson[\"coordinates\"], srid=srid)\n elif type_ == \"polygon\":\n return Polygon(geojson[\"coordinates\"], srid=srid)\n elif type_ == \"multipoint\":\n geometries = _MultiGeometry._multi_from_geojson(geojson, Point)\n return MultiPoint(geometries, srid=srid)\n elif type_ == \"multilinestring\":\n geometries = _MultiGeometry._multi_from_geojson(geojson, LineString)\n return MultiLineString(geometries, srid=srid)\n elif type_ == \"multipolygon\":\n geometries = _MultiGeometry._multi_from_geojson(geojson, Polygon)\n return MultiPolygon(geometries, srid=srid)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a Geometry from a Shapely geometry and the specified SRID.", "response": "def from_shapely(sgeom, srid=None):\n \"\"\"\n Create a Geometry from a Shapely geometry and the specified SRID.\n\n The Shapely geometry will not be modified.\n \"\"\"\n if SHAPELY:\n WKBWriter.defaults[\"include_srid\"] = True\n if srid:\n lgeos.GEOSSetSRID(sgeom._geom, srid)\n return Geometry(sgeom.wkb_hex)\n else:\n raise DependencyError(\"Shapely\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the geometry as an E ) WKB.", "response": "def wkb(self):\n \"\"\"\n Get the geometry as an (E)WKB.\n \"\"\"\n return self._to_wkb(use_srid=True, dimz=self.dimz, dimm=self.dimm)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef postgis_type(self):\n dimz = \"Z\" if self.dimz else \"\"\n dimm = \"M\" if self.dimm else \"\"\n if self.srid:\n return \"geometry({}{}{},{})\".format(self.type, dimz, dimm, self.srid)\n else:\n return \"geometry({}{}{})\".format(self.type, dimz, dimm)", "response": "Return the type of the geometry in PostGIS format including additional additional\n dimensions and SRID."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef poisson_ll(data, means):\n if sparse.issparse(data):\n return sparse_poisson_ll(data, means)\n genes, cells = data.shape\n clusters = means.shape[1]\n ll = np.zeros((cells, clusters))\n for i in range(clusters):\n means_i = np.tile(means[:,i], (cells, 1))\n means_i = means_i.transpose() + eps\n #ll[:,i] = np.sum(xlogy(data, means_i) - gammaln(data+1) - means_i, 0)\n ll[:,i] = np.sum(xlogy(data, means_i) - means_i, 0)\n return ll", "response": "Calculates the Poisson log - likelihood for each cell in the cluster pair."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef poisson_ll_2(p1, p2):\n p1_1 = p1 + eps\n p2_1 = p2 + eps\n return np.sum(-p2_1 + p1_1*np.log(p2_1))", "response": "Calculates Poisson LL of two points p1 and p2."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncalculates the Poisson distance between two vectors.", "response": "def poisson_dist(p1, p2):\n \"\"\"\n Calculates the Poisson distance between two vectors.\n\n p1 can be a sparse matrix, while p2 has to be a dense matrix.\n \"\"\"\n # ugh...\n p1_ = p1 + eps\n p2_ = p2 + eps\n return np.dot(p1_-p2_, np.log(p1_/p2_))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef delete(ctx, family_file, family_type, case_id):\n if not (family_file or case_id):\n LOG.error(\"Please provide a family file\")\n ctx.abort()\n\n adapter = ctx.obj['adapter']\n\n # Get a ped_parser.Family object from family file\n family = None\n family_id = None\n if family_file:\n with open(family_file, 'r') as family_lines:\n family = get_case(\n family_lines=family_lines,\n family_type=family_type\n )\n family_id = family.family_id\n\n # There has to be a case_id or a family at this stage.\n case_id = case_id or family_id\n\n if not case_id:\n LOG.warning(\"Please provide a case id\")\n ctx.abort()\n\n existing_case = adapter.case({'case_id': case_id})\n if not existing_case:\n LOG.warning(\"Case %s does not exist in database\" %case_id)\n context.abort\n\n\n start_deleting = datetime.now()\n try:\n delete_command(\n adapter=adapter,\n case_obj=existing_case,\n )\n except (CaseError, IOError) as error:\n LOG.warning(error)\n ctx.abort()", "response": "Delete the variants of a case."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef update_energy(self, bypass_check: bool = False):\n if bypass_check or (not bypass_check and self.update_time_check):\n self.get_weekly_energy()\n if 'week' in self.energy:\n self.get_monthly_energy()\n self.get_yearly_energy()\n if not bypass_check:\n self.update_energy_ts = time.time()", "response": "Builds weekly monthly and yearly dictionaries"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef main (args):\n if len(args) != 3:\n print(main.__doc__)\n sys.exit(1)\n\n d = libsedml.readSedML(args[1]);\n if ( d.getErrorLog().getNumFailsWithSeverity(libsedml.LIBSEDML_SEV_ERROR) > 0):\n print (d.getErrorLog().toString());\n else:\n libsedml.writeSedML(d, args[2]);\n\n return 0;", "response": "A simple wrapper for the echo_sedml command."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates the aiohttp session and run the example.", "response": "async def main() -> None:\n \"\"\"Create the aiohttp session and run the example.\"\"\"\n logging.basicConfig(level=logging.INFO)\n\n async with ClientSession() as websession:\n try:\n client = Client(websession)\n\n await client.profile.login('', '')\n _LOGGER.info('Account ID: %s', client.profile.account_id)\n\n summary = await client.profile.summary()\n _LOGGER.info('Account Summary: %s', summary)\n\n packages = await client.profile.packages()\n _LOGGER.info('Package Summary: %s', packages)\n except SeventeenTrackError as err:\n print(err)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef like_button_js_tag(context):\n if FACEBOOK_APP_ID is None:\n log.warning(\"FACEBOOK_APP_ID isn't setup correctly in your settings\")\n\n # make sure FACEBOOK_APP_ID is setup correct and user is authenticated\n if FACEBOOK_APP_ID:\n\n request = context.get('request', None)\n if request:\n return {\"LIKE_BUTTON_IS_VALID\": True,\n \"facebook_app_id\": FACEBOOK_APP_ID,\n \"channel_base_url\": request.get_host()}\n\n # if it is here, it isn't a valid setup, return False to not show the tag.\n return {\"LIKE_BUTTON_IS_VALID\": False}", "response": "This function will check if the user has set FACEBOOK_LIKE_APP_ID correctly in the django settings and if so will return True and the intercom_tag template to be displayed."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef like_button_tag(context):\n\n if FACEBOOK_APP_ID is None:\n log.warning(\"FACEBOOK_APP_ID isn't setup correctly in your settings\")\n\n # make sure INTERCOM_APPID is setup correct and user is authenticated\n if FACEBOOK_APP_ID:\n\n request = context.get('request', None)\n if request:\n path_to_like = (\n \"http://\" + request.get_host() + request.get_full_path())\n\n show_send = true_false_converter(FACEBOOK_SHOW_SEND)\n like_width = FACEBOOK_LIKE_WIDTH\n show_faces = true_false_converter(FACEBOOK_SHOW_FACES)\n font = FACEBOOK_FONT\n\n return {\"LIKE_BUTTON_IS_VALID\": True,\n \"path_to_like\": path_to_like,\n \"show_send\": show_send,\n \"like_width\": like_width,\n \"show_faces\": show_faces,\n \"font\": font,\n \"like_layout\": FACEBOOK_LIKE_LAYOUT}\n\n # if it is here, it isn't a valid setup, return False to not show the tag.\n return {\"LIKE_BUTTON_IS_VALID\": False}", "response": "This function will check if the user has set the FACEBOOK_APP_ID and if so will check if they have the FACEBOOK_APP_ID setup correctly in the django settings and then will pass the data\n along to the intercom_tag template to be displayed."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nadd a single variant to the structural variants collection", "response": "def add_structural_variant(self, variant, max_window = 3000):\n \"\"\"Add a variant to the structural variants collection\n \n The process of adding an SV variant differs quite a bit from the \n more straight forward case of SNV/INDEL.\n \n Variants are represented in the database by clusters that are two intervals,\n one interval for start(pos) and one for end. The size of the intervals changes\n according to the size of the variants. The maximum window size is a parameter.\n \n Here we need to search if the variant matches any of the existing\n clusters. Then we need to choose the closest cluster and update\n the boundaries for that cluster.\n \n Args:\n variant (dict): A variant dictionary\n max_window(int): Specify the maximum window size for large svs\n \n \n -\n - -\n - -\n - -\n - -\n - -\n |-----| |-----|\n /\\ /\\\n | |\n pos - interval_size end + interval_size\n \"\"\"\n # This will return the cluster most similar to variant or None\n cluster = self.get_structural_variant(variant)\n # If there was no matcing cluster we need to create a new cluster\n if cluster is None:\n # The cluster will be populated with information later.\n cluster = {\n 'chrom': variant['chrom'],\n 'end_chrom': variant['end_chrom'],\n 'sv_type': variant['sv_type'],\n 'pos_sum': 0,\n 'end_sum': 0,\n 'observations': 0,\n 'length': 0,\n 'sv_type': variant['sv_type'],\n 'families': [],\n \n }\n # Insert variant to get a _id\n _id = self.db.structural_variant.insert_one(cluster).inserted_id\n \n cluster['_id'] = _id\n \n case_id = variant.get('case_id')\n if case_id:\n # If the variant is already added for this case we continue\n # One case will only give duplicated information\n if case_id in cluster['families']:\n return\n else:\n # Insert the new case in the beginning of array\n cluster['families'].insert(0,case_id)\n # Make sure array does not grow out of bounds\n cluster['families'] = cluster['families'][:50]\n\n # Update number of times we have seen the event\n nr_events = cluster['observations'] + 1\n \n # -\n # - -\n # - -\n # - -\n # - -\n # - -\n # |--.--| |--.--|\n \n # This indicates the center for each of the end points for the event AFTER the new variant\n # is added to the cluster\n # i.e. the dots in the picture above\n pos_mean = int((cluster['pos_sum'] + variant['pos']) // (nr_events))\n end_mean = int((cluster['end_sum'] + variant['end']) // (nr_events))\n \n # We need to calculate the new cluster length\n # Handle translocation as a special case\n if cluster['sv_type'] != 'BND':\n cluster_len = end_mean - pos_mean\n # We need to adapt the interval size depending on the size of the cluster\n divider = 10\n if cluster_len < 1000:\n # We allow intervals for smaller variants to be relatively larger\n divider = 2\n elif cluster_len < 1000:\n # We allow intervals for smaller variants to be relatively larger\n divider = 5\n interval_size = int(min(round(cluster_len/divider, -2), max_window))\n else:\n # We need to treat translocations as a special case.\n # Set length to a huge number that mongodb can handle, float('inf') would not work.\n cluster_len = 10e10\n # This number seems large, if compared with SV size it is fairly small.\n interval_size = max_window * 2\n \n # If the length of SV is shorter than 500 the variant \n # is considered precise\n # Otherwise the interval size is closest whole 100 number\n res = self.db.structural_variant.find_one_and_update(\n {'_id': cluster['_id']},\n {\n '$inc': {\n 'observations': 1,\n 'pos_sum': variant['pos'],\n 'end_sum': variant['end'],\n },\n \n '$set': {\n 'pos_left': max(pos_mean - interval_size, 0),\n 'pos_right': pos_mean + interval_size,\n 'end_left': max(end_mean - interval_size, 0),\n 'end_right': end_mean + interval_size,\n 'families': cluster['families'],\n 'length': cluster_len,\n }\n }\n )\n \n # Insert an identity object to link cases to variants and clusters\n identity_obj = Identity(cluster_id=cluster['_id'], variant_id=variant['id_column'], \n case_id=case_id)\n self.db.identity.insert_one(identity_obj)\n \n return"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the structural variant that is found in the database", "response": "def get_structural_variant(self, variant):\n \"\"\"Check if there are any overlapping sv clusters\n\n Search the sv variants with chrom start end_chrom end and sv_type\n\n Args:\n variant (dict): A variant dictionary\n\n Returns:\n variant (dict): A variant dictionary\n \"\"\"\n # Create a query for the database\n # This will include more variants than we want\n # The rest of the calculations will be done in python\n query = {\n 'chrom': variant['chrom'],\n 'end_chrom': variant['end_chrom'],\n 'sv_type': variant['sv_type'],\n '$and': [\n {'pos_left': {'$lte': variant['pos']}},\n {'pos_right': {'$gte': variant['pos']}},\n ]\n }\n\n res = self.db.structural_variant.find(query).sort('pos_left',1)\n match = None\n distance = None\n closest_hit = None\n # First we check that the coordinates are correct\n # Then we count the distance to mean on both ends to see which variant is closest\n for hit in res:\n # We know from the query that the variants position is larger than the left most part of\n # the cluster. \n # If the right most part of the cluster is smaller than the variant position they do\n # not overlap\n if hit['end_left'] > variant['end']:\n continue\n if hit['end_right'] < variant['end']:\n continue\n\n # We need to calculate the distance to see what cluster that was closest to the variant\n distance = (abs(variant['pos'] - (hit['pos_left'] + hit['pos_right'])/2) + \n abs(variant['end'] - (hit['end_left'] + hit['end_right'])/2))\n\n # If we have no cluster yet we set the curent to be the hit\n if closest_hit is None:\n match = hit\n closest_hit = distance\n continue\n\n # If the distance is closer than previous we choose current cluster\n if distance < closest_hit:\n # Set match to the current closest hit\n match = hit\n # Update the closest distance\n closest_hit = distance\n\n return match"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn all structural variants in the database", "response": "def get_sv_variants(self, chromosome=None, end_chromosome=None, sv_type=None, \n pos=None, end=None):\n \"\"\"Return all structural variants in the database\n\n Args:\n chromosome (str)\n end_chromosome (str)\n sv_type (str)\n pos (int): Left position of SV\n end (int): Right position of SV\n\n Returns:\n variants (Iterable(Variant))\n \"\"\"\n query = {}\n \n if chromosome:\n query['chrom'] = chromosome\n if end_chromosome:\n query['end_chrom'] = end_chromosome\n if sv_type:\n query['sv_type'] = sv_type\n if pos:\n if not '$and' in query:\n query['$and'] = []\n query['$and'].append({'pos_left': {'$lte': pos}})\n query['$and'].append({'pos_right': {'$gte': pos}})\n \n if end:\n if not '$and' in query:\n query['$and'] = []\n query['$and'].append({'end_left': {'$lte': end}})\n query['$and'].append({'end_right': {'$gte': end}})\n \n LOG.info(\"Find all sv variants {}\".format(query))\n \n return self.db.structural_variant.find(query).sort([('chrom', ASCENDING), ('pos_left', ASCENDING)])"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsearch what clusters a variant belongs to", "response": "def get_clusters(self, variant_id):\n \"\"\"Search what clusters a variant belongs to\n \n Args:\n variant_id(str): From ID column in vcf\n \n Returns:\n clusters()\n \"\"\"\n query = {'variant_id':variant_id}\n identities = self.db.identity.find(query)\n return identities"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef run_state_estimation(data, clusters, dist='Poiss', reps=1, **kwargs):\n clusters = int(clusters)\n func = poisson_estimate_state\n dist = dist.lower()\n if dist=='poiss' or dist=='poisson':\n pass\n elif dist=='nb':\n func = nb_estimate_state\n elif dist=='zip':\n func = zip_estimate_state\n elif dist=='lognorm' or dist=='log-normal' or dist=='lognormal':\n func = log_norm_nmf\n elif dist=='gaussian' or dist=='norm' or dist=='normal':\n func = norm_nmf\n else:\n print('dist should be one of Poiss, NB, ZIP, LogNorm, or Gaussian. Using Poiss.')\n # TODO: estimate number of clusters\n if clusters == 0:\n from .gap_score import run_gap_k_selection, preproc_data\n data_tsvd = preproc_data(data, gene_subset=False)\n max_k, gap_vals, sk_vals = run_gap_k_selection(data_tsvd,\n k_min=1, k_max=50, skip=5, B=6)\n clusters = min(max_k, data.shape[0] - 1, data.shape[1] - 1)\n best_ll = np.inf\n best_M = None\n best_W = None\n for i in range(reps):\n results = func(data, clusters, **kwargs)\n M = results[0]\n W = results[1]\n if dist=='NB':\n ll = results[3]\n else:\n ll = results[2]\n if ll < best_ll:\n best_ll = ll\n best_M = M\n best_W = W\n return best_M, best_W, best_ll", "response": "Runs state estimation for multiple initializations returning the result with the highest log - likelihood."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef turn_on(self):\n if self.device_status != 'on':\n body = helpers.req_body(self.manager, 'devicestatus')\n body['uuid'] = self.uuid\n body['status'] = 'on'\n head = helpers.req_headers(self.manager)\n\n r, _ = helpers.call_api('/131airPurifier/v1/device/deviceStatus',\n 'put', json=body, headers=head)\n\n if r is not None and helpers.check_response(r, 'airpur_status'):\n self.device_status = 'on'\n return True\n else:\n return False", "response": "Turn Air Purifier on"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nadjust the fan speed by Specifying 1 2 3 as argument or cycle through speeds increasing by one", "response": "def fan_speed(self, speed: int = None) -> bool:\n \"\"\"Adjust Fan Speed by Specifying 1,2,3 as argument or cycle\n through speeds increasing by one\"\"\"\n body = helpers.req_body(self.manager, 'devicestatus')\n body['uuid'] = self.uuid\n head = helpers.req_headers(self.manager)\n if self.details.get('mode') != 'manual':\n self.mode_toggle('manual')\n else:\n if speed is not None:\n level = int(self.details.get('level'))\n if speed == level:\n return False\n elif speed in [1, 2, 3]:\n body['level'] = speed\n else:\n if (level + 1) > 3:\n body['level'] = 1\n else:\n body['level'] = int(level + 1)\n\n r, _ = helpers.call_api('/131airPurifier/v1/device/updateSpeed',\n 'put', json=body, headers=head)\n\n if r is not None and helpers.check_response(r, 'airpur_status'):\n self.details['level'] = body['level']\n return True\n else:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef mode_toggle(self, mode: str) -> bool:\n head = helpers.req_headers(self.manager)\n body = helpers.req_body(self.manager, 'devicestatus')\n body['uuid'] = self.uuid\n if mode != body['mode'] and mode in ['sleep', 'auto', 'manual']:\n body['mode'] = mode\n if mode == 'manual':\n body['level'] = 1\n\n r, _ = helpers.call_api('/131airPurifier/v1/device/updateMode',\n 'put', json=body, headers=head)\n\n if r is not None and helpers.check_response(r, 'airpur_status'):\n self.details['mode'] = mode\n return True\n\n return False", "response": "Set mode to manual auto or sleep"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef fourier_series(x, *a):\n output = 0\n output += a[0]/2\n w = a[1]\n for n in range(2, len(a), 2):\n n_ = n/2\n val1 = a[n]\n val2 = a[n+1]\n output += val1*np.sin(n_*x*w)\n output += val2*np.cos(n_*x*w)\n return output", "response": "Compute the number of fourier series of a given interval x."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncalculate the distances along a graph from a start node to a new node.", "response": "def graph_distances(start, edges, distances):\n \"\"\"\n Given an undirected adjacency list and a pairwise distance matrix between\n all nodes: calculates distances along graph from start node.\n\n Args:\n start (int): start node\n edges (list): adjacency list of tuples\n distances (array): 2d array of distances between nodes\n\n Returns:\n dict of node to distance from start\n \"\"\"\n # convert adjacency list to adjacency dict\n adj = {x: [] for x in range(len(distances))}\n for n1, n2 in edges:\n adj[n1].append(n2)\n adj[n2].append(n1)\n # run dijkstra's algorithm\n to_visit = []\n new_dist = {}\n for n in adj[start]:\n heapq.heappush(to_visit, (distances[start, n], n))\n while to_visit:\n d, next_node = heapq.heappop(to_visit)\n if next_node not in new_dist:\n new_dist[next_node] = d\n for n in adj[next_node]:\n if n not in new_dist:\n heapq.heappush(to_visit, (d + distances[next_node, n], n))\n return new_dist"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a 1d array containing the pseudotime value of each cell in the tree.", "response": "def pseudotime(starting_node, edges, fitted_vals):\n \"\"\"\n Args:\n starting_node (int): index of the starting node\n edges (list): list of tuples (node1, node2)\n fitted_vals (array): output of lineage (2 x cells)\n\n Returns:\n A 1d array containing the pseudotime value of each cell.\n \"\"\"\n # TODO\n # 1. calculate a distance matrix...\n distances = np.array([[sum((x - y)**2) for x in fitted_vals.T] for y in fitted_vals.T])\n # 2. start from the root node/cell, calculate distance along graph\n distance_dict = graph_distances(starting_node, edges, distances)\n output = []\n for i in range(fitted_vals.shape[1]):\n output.append(distance_dict[i])\n return np.array(output)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _add_countriesdata(cls, iso3, country):\n # type: (str, hxl.Row) -> None\n \"\"\"\n Set up countries data from data in form provided by UNStats and World Bank\n\n Args:\n iso3 (str): ISO3 code for country\n country (hxl.Row): Country information\n\n Returns:\n None\n \"\"\"\n countryname = country.get('#country+name+preferred')\n cls._countriesdata['countrynames2iso3'][countryname.upper()] = iso3\n iso2 = country.get('#country+code+v_iso2')\n if iso2:\n cls._countriesdata['iso2iso3'][iso2] = iso3\n # different types so keys won't clash\n cls._countriesdata['iso2iso3'][iso3] = iso2\n m49 = country.get('#country+code+num+v_m49')\n if m49:\n m49 = int(m49)\n cls._countriesdata['m49iso3'][m49] = iso3\n # different types so keys won't clash\n cls._countriesdata['m49iso3'][iso3] = m49\n cls._countriesdata['aliases'][iso3] = re.compile(country.get('#country+regex'), re.IGNORECASE)\n regionname = country.get('#region+main+name+preferred')\n sub_regionname = country.get('#region+sub+name+preferred')\n intermediate_regionname = country.get('#region+intermediate+name+preferred')\n regionid = country.get('#region+main+code')\n if regionid:\n regionid = int(regionid)\n sub_regionid = country.get('#region+sub+code')\n if sub_regionid:\n sub_regionid = int(sub_regionid)\n intermediate_regionid = country.get('#region+intermediate+code')\n if intermediate_regionid:\n intermediate_regionid = int(intermediate_regionid)\n\n # region, subregion and intermediate region codes do not clash so only need one dict\n def add_country_to_set(colname, idval, iso3):\n value = cls._countriesdata[colname].get(idval)\n if value is None:\n value = set()\n cls._countriesdata['regioncodes2countries'][idval] = value\n value.add(iso3)\n\n if regionname:\n add_country_to_set('regioncodes2countries', regionid, iso3)\n cls._countriesdata['regioncodes2names'][regionid] = regionname\n cls._countriesdata['regionnames2codes'][regionname.upper()] = regionid\n if sub_regionname:\n add_country_to_set('regioncodes2countries', sub_regionid, iso3)\n cls._countriesdata['regioncodes2names'][sub_regionid] = sub_regionname\n cls._countriesdata['regionnames2codes'][sub_regionname.upper()] = sub_regionid\n if intermediate_regionname:\n add_country_to_set('regioncodes2countries', intermediate_regionid, iso3)\n cls._countriesdata['regioncodes2names'][intermediate_regionid] = intermediate_regionname\n cls._countriesdata['regionnames2codes'][intermediate_regionname.upper()] = \\\n intermediate_regionid", "response": "Add countries data from country information to the internal dictionary."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef set_countriesdata(cls, countries):\n # type: (str) -> None\n \"\"\"\n Set up countries data from data in form provided by UNStats and World Bank\n\n Args:\n countries (str): Countries data in HTML format provided by UNStats\n\n Returns:\n None\n \"\"\"\n cls._countriesdata = dict()\n cls._countriesdata['countries'] = dict()\n cls._countriesdata['iso2iso3'] = dict()\n cls._countriesdata['m49iso3'] = dict()\n cls._countriesdata['countrynames2iso3'] = dict()\n cls._countriesdata['regioncodes2countries'] = dict()\n cls._countriesdata['regioncodes2names'] = dict()\n cls._countriesdata['regionnames2codes'] = dict()\n cls._countriesdata['aliases'] = dict()\n\n for country in countries:\n iso3 = country.get('#country+code+v_iso3')\n if not iso3:\n continue\n iso3 = iso3.upper()\n cls._add_countriesdata(iso3, country)\n cls._countriesdata['countries'][iso3] = country.dictionary\n\n def sort_list(colname):\n for idval in cls._countriesdata[colname]:\n cls._countriesdata[colname][idval] = \\\n sorted(list(cls._countriesdata[colname][idval]))\n\n sort_list('regioncodes2countries')", "response": "Sets up countries data from data provided by UNStats and World Bank."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreads countries data from OCHA and stores it in the class s _countriesdata attribute.", "response": "def countriesdata(cls, use_live=True):\n # type: (bool) -> List[Dict[Dict]]\n \"\"\"\n Read countries data from OCHA countries feed (falling back to file)\n\n Args:\n use_live (bool): Try to get use latest data from web rather than file in package. Defaults to True.\n\n Returns:\n List[Dict[Dict]]: Countries dictionaries\n \"\"\"\n if cls._countriesdata is None:\n countries = None\n if use_live:\n try:\n countries = hxl.data(cls._ochaurl)\n except IOError:\n logger.exception('Download from OCHA feed failed! Falling back to stored file.')\n if countries is None:\n countries = hxl.data(\n script_dir_plus_file('Countries & Territories Taxonomy MVP - C&T Taxonomy with HXL Tags.csv',\n Country), allow_local=True)\n cls.set_countriesdata(countries)\n return cls._countriesdata"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsets the World Bank url from which to retrieve countries data.", "response": "def set_ocha_url(cls, url=None):\n # type: (str) -> None\n \"\"\"\n Set World Bank url from which to retrieve countries data\n\n Args:\n url (str): World Bank url from which to retrieve countries data. Defaults to internal value.\n\n Returns:\n None\n \"\"\"\n if url is None:\n url = cls._ochaurl_int\n cls._ochaurl = url"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the country information for a given ISO3 code.", "response": "def get_country_info_from_iso3(cls, iso3, use_live=True, exception=None):\n # type: (str, bool, Optional[ExceptionUpperBound]) -> Optional[Dict[str]]\n \"\"\"Get country information from ISO3 code\n\n Args:\n iso3 (str): ISO3 code for which to get country information\n use_live (bool): Try to get use latest data from web rather than file in package. Defaults to True.\n exception (Optional[ExceptionUpperBound]): An exception to raise if country not found. Defaults to None.\n\n Returns:\n Optional[Dict[str]]: country information\n \"\"\"\n countriesdata = cls.countriesdata(use_live=use_live)\n country = countriesdata['countries'].get(iso3.upper())\n if country is not None:\n return country\n\n if exception is not None:\n raise exception\n return None"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the name of the country from ISO3 code.", "response": "def get_country_name_from_iso3(cls, iso3, use_live=True, exception=None):\n # type: (str, bool, Optional[ExceptionUpperBound]) -> Optional[str]\n \"\"\"Get country name from ISO3 code\n\n Args:\n iso3 (str): ISO3 code for which to get country name\n use_live (bool): Try to get use latest data from web rather than file in package. Defaults to True.\n exception (Optional[ExceptionUpperBound]): An exception to raise if country not found. Defaults to None.\n\n Returns:\n Optional[str]: Country name\n \"\"\"\n countryinfo = cls.get_country_info_from_iso3(iso3, use_live=use_live, exception=exception)\n if countryinfo is not None:\n return countryinfo.get('#country+name+preferred')\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_iso2_from_iso3(cls, iso3, use_live=True, exception=None):\n # type: (str, bool, Optional[ExceptionUpperBound]) -> Optional[str]\n \"\"\"Get ISO2 from ISO3 code\n\n Args:\n iso3 (str): ISO3 code for which to get ISO2 code\n use_live (bool): Try to get use latest data from web rather than file in package. Defaults to True.\n exception (Optional[ExceptionUpperBound]): An exception to raise if country not found. Defaults to None.\n\n Returns:\n Optional[str]: ISO2 code\n \"\"\"\n countriesdata = cls.countriesdata(use_live=use_live)\n iso2 = countriesdata['iso2iso3'].get(iso3.upper())\n if iso2 is not None:\n return iso2\n\n if exception is not None:\n raise exception\n return None", "response": "Get ISO2 code from ISO3 code."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_country_info_from_iso2(cls, iso2, use_live=True, exception=None):\n # type: (str, bool, Optional[ExceptionUpperBound]) -> Optional[Dict[str]]\n \"\"\"Get country name from ISO2 code\n\n Args:\n iso2 (str): ISO2 code for which to get country information\n use_live (bool): Try to get use latest data from web rather than file in package. Defaults to True.\n exception (Optional[ExceptionUpperBound]): An exception to raise if country not found. Defaults to None.\n\n Returns:\n Optional[Dict[str]]: Country information\n \"\"\"\n iso3 = cls.get_iso3_from_iso2(iso2, use_live=use_live, exception=exception)\n if iso3 is not None:\n return cls.get_country_info_from_iso3(iso3, use_live=use_live, exception=exception)\n return None", "response": "Returns the country name from ISO2 code."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_country_name_from_iso2(cls, iso2, use_live=True, exception=None):\n # type: (str, bool, Optional[ExceptionUpperBound]) -> Optional[str]\n \"\"\"Get country name from ISO2 code\n\n Args:\n iso2 (str): ISO2 code for which to get country name\n use_live (bool): Try to get use latest data from web rather than file in package. Defaults to True.\n exception (Optional[ExceptionUpperBound]): An exception to raise if country not found. Defaults to None.\n\n Returns:\n Optional[str]: Country name\n \"\"\"\n iso3 = cls.get_iso3_from_iso2(iso2, use_live=use_live, exception=exception)\n if iso3 is not None:\n return cls.get_country_name_from_iso3(iso3, exception=exception)\n return None", "response": "Returns the name of the country from ISO2 code."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets M49 code from ISO3 code.", "response": "def get_m49_from_iso3(cls, iso3, use_live=True, exception=None):\n # type: (str, bool, Optional[ExceptionUpperBound]) -> Optional[int]\n \"\"\"Get M49 from ISO3 code\n\n Args:\n iso3 (str): ISO3 code for which to get M49 code\n use_live (bool): Try to get use latest data from web rather than file in package. Defaults to True.\n exception (Optional[ExceptionUpperBound]): An exception to raise if country not found. Defaults to None.\n\n Returns:\n Optional[int]: M49 code\n \"\"\"\n countriesdata = cls.countriesdata(use_live=use_live)\n m49 = countriesdata['m49iso3'].get(iso3)\n if m49 is not None:\n return m49\n\n if exception is not None:\n raise exception\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_country_info_from_m49(cls, m49, use_live=True, exception=None):\n # type: (int, bool, Optional[ExceptionUpperBound]) -> Optional[Dict[str]]\n \"\"\"Get country name from M49 code\n\n Args:\n m49 (int): M49 numeric code for which to get country information\n use_live (bool): Try to get use latest data from web rather than file in package. Defaults to True.\n exception (Optional[ExceptionUpperBound]): An exception to raise if country not found. Defaults to None.\n\n Returns:\n Optional[Dict[str]]: Country information\n \"\"\"\n iso3 = cls.get_iso3_from_m49(m49, use_live=use_live, exception=exception)\n if iso3 is not None:\n return cls.get_country_info_from_iso3(iso3, exception=exception)\n return None", "response": "Get country name from M49 code."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_country_name_from_m49(cls, m49, use_live=True, exception=None):\n # type: (int, bool, Optional[ExceptionUpperBound]) -> Optional[str]\n \"\"\"Get country name from M49 code\n\n Args:\n m49 (int): M49 numeric code for which to get country name\n use_live (bool): Try to get use latest data from web rather than file in package. Defaults to True.\n exception (Optional[ExceptionUpperBound]): An exception to raise if country not found. Defaults to None.\n\n Returns:\n Optional[str]: Country name\n \"\"\"\n iso3 = cls.get_iso3_from_m49(m49, use_live=use_live, exception=exception)\n if iso3 is not None:\n return cls.get_country_name_from_iso3(iso3, exception=exception)\n return None", "response": "Returns the country name from M49 code."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef expand_countryname_abbrevs(cls, country):\n # type: (str) -> List[str]\n \"\"\"Expands abbreviation(s) in country name in various ways (eg. FED -> FEDERATED, FEDERAL etc.)\n\n Args:\n country (str): Country with abbreviation(s)to expand\n\n Returns:\n List[str]: Uppercase country name with abbreviation(s) expanded in various ways\n \"\"\"\n def replace_ensure_space(word, replace, replacement):\n return word.replace(replace, '%s ' % replacement).replace(' ', ' ').strip()\n countryupper = country.upper()\n for abbreviation in cls.abbreviations:\n countryupper = replace_ensure_space(countryupper, abbreviation, cls.abbreviations[abbreviation])\n candidates = [countryupper]\n for abbreviation in cls.multiple_abbreviations:\n if abbreviation in countryupper:\n for expanded in cls.multiple_abbreviations[abbreviation]:\n candidates.append(replace_ensure_space(countryupper, abbreviation, expanded))\n return candidates", "response": "Expands abbreviation in country name in various ways."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef simplify_countryname(cls, country):\n # type: (str) -> (str, List[str])\n \"\"\"Simplifies country name by removing descriptive text eg. DEMOCRATIC, REPUBLIC OF etc.\n\n Args:\n country (str): Country name to simplify\n\n Returns:\n Tuple[str, List[str]]: Uppercase simplified country name and list of removed words\n \"\"\"\n countryupper = country.upper()\n words = get_words_in_sentence(countryupper)\n index = countryupper.find(',')\n if index != -1:\n countryupper = countryupper[:index]\n index = countryupper.find(':')\n if index != -1:\n countryupper = countryupper[:index]\n regex = re.compile('\\(.+?\\)')\n countryupper = regex.sub('', countryupper)\n remove = copy.deepcopy(cls.simplifications)\n for simplification1, simplification2 in cls.abbreviations.items():\n countryupper = countryupper.replace(simplification1, '')\n remove.append(simplification2)\n for simplification1, simplifications in cls.multiple_abbreviations.items():\n countryupper = countryupper.replace(simplification1, '')\n for simplification2 in simplifications:\n remove.append(simplification2)\n remove = '|'.join(remove)\n regex = re.compile(r'\\b(' + remove + r')\\b', flags=re.IGNORECASE)\n countryupper = regex.sub('', countryupper)\n countryupper = countryupper.strip()\n countryupper_words = get_words_in_sentence(countryupper)\n if len(countryupper_words) > 1:\n countryupper = countryupper_words[0]\n if countryupper:\n words.remove(countryupper)\n return countryupper, words", "response": "Simplifies a country name by removing descriptive text eg. DEMOCRATIC REPUBLIC OF etc."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn ISO3 code for a given country. Only exact matches or None are returned.", "response": "def get_iso3_country_code(cls, country, use_live=True, exception=None):\n # type: (str, bool, Optional[ExceptionUpperBound]) -> Optional[str]\n \"\"\"Get ISO3 code for cls. Only exact matches or None are returned.\n\n Args:\n country (str): Country for which to get ISO3 code\n use_live (bool): Try to get use latest data from web rather than file in package. Defaults to True.\n exception (Optional[ExceptionUpperBound]): An exception to raise if country not found. Defaults to None.\n\n Returns:\n Optional[str]: ISO3 country code or None\n \"\"\"\n countriesdata = cls.countriesdata(use_live=use_live)\n countryupper = country.upper()\n len_countryupper = len(countryupper)\n if len_countryupper == 3:\n if countryupper in countriesdata['countries']:\n return countryupper\n elif len_countryupper == 2:\n iso3 = countriesdata['iso2iso3'].get(countryupper)\n if iso3 is not None:\n return iso3\n\n iso3 = countriesdata['countrynames2iso3'].get(countryupper)\n if iso3 is not None:\n return iso3\n\n for candidate in cls.expand_countryname_abbrevs(countryupper):\n iso3 = countriesdata['countrynames2iso3'].get(candidate)\n if iso3 is not None:\n return iso3\n\n if exception is not None:\n raise exception\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_iso3_country_code_fuzzy(cls, country, use_live=True, exception=None):\n # type: (str, bool, Optional[ExceptionUpperBound]) -> Tuple[[Optional[str], bool]]\n \"\"\"Get ISO3 code for cls. A tuple is returned with the first value being the ISO3 code and the second\n showing if the match is exact or not.\n\n Args:\n country (str): Country for which to get ISO3 code\n use_live (bool): Try to get use latest data from web rather than file in package. Defaults to True.\n exception (Optional[ExceptionUpperBound]): An exception to raise if country not found. Defaults to None.\n\n Returns:\n Tuple[[Optional[str], bool]]: ISO3 code and if the match is exact or (None, False).\n \"\"\"\n countriesdata = cls.countriesdata(use_live=use_live)\n iso3 = cls.get_iso3_country_code(country,\n use_live=use_live) # don't put exception param here as we don't want it to throw\n\n if iso3 is not None:\n return iso3, True\n\n def remove_matching_from_list(wordlist, word_or_part):\n for word in wordlist:\n if word_or_part in word:\n wordlist.remove(word)\n\n # fuzzy matching\n expanded_country_candidates = cls.expand_countryname_abbrevs(country)\n match_strength = 0\n matches = set()\n for countryname in sorted(countriesdata['countrynames2iso3']):\n for candidate in expanded_country_candidates:\n simplified_country, removed_words = cls.simplify_countryname(candidate)\n if simplified_country in countryname:\n words = get_words_in_sentence(countryname)\n new_match_strength = 0\n if simplified_country:\n remove_matching_from_list(words, simplified_country)\n new_match_strength += 32\n for word in removed_words:\n if word in countryname:\n remove_matching_from_list(words, word)\n new_match_strength += 4\n else:\n if word in cls.major_differentiators:\n new_match_strength -= 16\n else:\n new_match_strength -= 1\n for word in words:\n if word in cls.major_differentiators:\n new_match_strength -= 16\n else:\n new_match_strength -= 1\n iso3 = countriesdata['countrynames2iso3'][countryname]\n if new_match_strength > match_strength:\n match_strength = new_match_strength\n matches = set()\n if new_match_strength == match_strength:\n matches.add(iso3)\n\n if len(matches) == 1 and match_strength > 16:\n return matches.pop(), False\n\n # regex lookup\n for iso3, regex in countriesdata['aliases'].items():\n index = re.search(regex, country.upper())\n if index is not None:\n return iso3, False\n\n if exception is not None:\n raise exception\n return None, False", "response": "Returns the ISO3 code for a given country."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a list of countries in a given region", "response": "def get_countries_in_region(cls, region, use_live=True, exception=None):\n # type: (Union[int,str], bool, Optional[ExceptionUpperBound]) -> List[str]\n \"\"\"Get countries (ISO3 codes) in region\n\n Args:\n region (Union[int,str]): Three digit UNStats M49 region code or region name\n use_live (bool): Try to get use latest data from web rather than file in package. Defaults to True.\n exception (Optional[ExceptionUpperBound]): An exception to raise if region not found. Defaults to None.\n\n Returns:\n List(str): Sorted list of ISO3 country names\n \"\"\"\n countriesdata = cls.countriesdata(use_live=use_live)\n if isinstance(region, int):\n regioncode = region\n else:\n regionupper = region.upper()\n regioncode = countriesdata['regionnames2codes'].get(regionupper)\n\n if regioncode is not None:\n return countriesdata['regioncodes2countries'][regioncode]\n\n if exception is not None:\n raise exception\n return list()"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nloading the profile of samples.", "response": "def load_profile(ctx, variant_file, update, stats, profile_threshold):\n\n \"\"\"\n Command for profiling of samples. User may upload variants used in profiling\n from a vcf, update the profiles for all samples, and get some stats\n from the profiles in the database.\n\n Profiling is used to monitor duplicates in the database. The profile is\n based on the variants in the 'profile_variant' collection, assessing\n the genotypes for each sample at the position of these variants.\n \"\"\"\n\n adapter = ctx.obj['adapter']\n\n if variant_file:\n load_profile_variants(adapter, variant_file)\n\n if update:\n update_profiles(adapter)\n\n if stats:\n\n distance_dict = profile_stats(adapter, threshold=profile_threshold)\n click.echo(table_from_dict(distance_dict))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nadds several variants to the profile_variant collection in the database", "response": "def add_profile_variants(self, profile_variants):\n\n \"\"\"Add several variants to the profile_variant collection in the\n database\n\n Args:\n\n profile_variants(list(models.ProfileVariant))\n\n \"\"\"\n\n results = self.db.profile_variant.insert_many(profile_variants)\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the ZIP parameters that best fit a given data set.", "response": "def zip_fit_params(data):\n \"\"\"\n Returns the ZIP parameters that best fit a given data set.\n\n Args:\n data (array): 2d array of genes x cells belonging to a given cluster\n\n Returns:\n L (array): 1d array of means\n M (array): 1d array of zero-inflation parameter\n \"\"\"\n genes, cells = data.shape\n m = data.mean(1)\n v = data.var(1)\n M = (v-m)/(m**2+v-m)\n #M = v/(v+m**2)\n #M[np.isnan(M)] = 0.0\n M = np.array([min(1.0, max(0.0, x)) for x in M])\n L = m + v/m - 1.0\n #L = (v + m**2)/m\n L[np.isnan(L)] = 0.0\n L = np.array([max(0.0, x) for x in L])\n return L, M"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nperform hard EM clustering using the zero - inflated Poisson distribution.", "response": "def zip_cluster(data, k, init=None, max_iters=100):\n \"\"\"\n Performs hard EM clustering using the zero-inflated Poisson distribution.\n\n Args:\n data (array): A 2d array- genes x cells\n k (int): Number of clusters\n init (array, optional): Initial centers - genes x k array. Default: None, use kmeans++\n max_iters (int, optional): Maximum number of iterations. Default: 100\n\n Returns:\n assignments (array): integer assignments of cells to clusters (length cells)\n L (array): Poisson parameter (genes x k)\n M (array): zero-inflation parameter (genes x k)\n \"\"\"\n genes, cells = data.shape\n init, new_assignments = kmeans_pp(data+eps, k, centers=init)\n centers = np.copy(init)\n M = np.zeros(centers.shape)\n assignments = new_assignments\n for c in range(k):\n centers[:,c], M[:,c] = zip_fit_params_mle(data[:, assignments==c])\n for it in range(max_iters):\n lls = zip_ll(data, centers, M)\n new_assignments = np.argmax(lls, 1)\n if np.equal(assignments, new_assignments).all():\n return assignments, centers, M\n for c in range(k):\n centers[:,c], M[:,c] = zip_fit_params_mle(data[:, assignments==c])\n assignments = new_assignments\n return assignments, centers, M"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef diffusion_mds(means, weights, d, diffusion_rounds=10):\n for i in range(diffusion_rounds):\n weights = weights*weights\n weights = weights/weights.sum(0)\n X = dim_reduce(means, weights, d)\n if X.shape[0]==2:\n return X.dot(weights)\n else:\n return X.T.dot(weights)", "response": "Dimensionality reduction using MDS"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef mds(means, weights, d):\n X = dim_reduce(means, weights, d)\n if X.shape[0]==2:\n return X.dot(weights)\n else:\n return X.T.dot(weights)", "response": "Dimensionality reduction using MDS."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef dim_reduce_data(data, d):\n genes, cells = data.shape\n distances = np.zeros((cells, cells))\n for i in range(cells):\n for j in range(cells):\n distances[i,j] = poisson_dist(data[:,i], data[:,j])\n # do MDS on the distance matrix (procedure from Wikipedia)\n proximity = distances**2\n J = np.eye(cells) - 1./cells\n B = -0.5*np.dot(J, np.dot(proximity, J))\n # B should be symmetric, so we can use eigh\n e_val, e_vec = np.linalg.eigh(B)\n # Note: lam should be ordered to be the largest eigenvalues\n lam = np.diag(e_val[-d:])[::-1]\n #lam = max_or_zero(lam)\n E = e_vec[:,-d:][::-1]\n X = np.dot(E, lam**0.5)\n return X", "response": "Does a MDS on the data directly not on the means."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting a case from the database", "response": "def case(self, case):\n \"\"\"Get a case from the database\n\n Search the cases with the case id\n\n Args:\n case (dict): A case dictionary\n\n Returns:\n mongo_case (dict): A mongo case dictionary\n \"\"\"\n LOG.debug(\"Getting case {0} from database\".format(case.get('case_id')))\n case_id = case['case_id']\n return self.db.case.find_one({'case_id': case_id})"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef nr_cases(self, snv_cases=None, sv_cases=None):\n query = {}\n \n if snv_cases:\n query = {'vcf_path': {'$exists':True}}\n if sv_cases:\n query = {'vcf_sv_path': {'$exists':True}}\n if snv_cases and sv_cases:\n query = None\n\n return self.db.case.count_documents(query)", "response": "Return the number of cases in the database"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nadds a case to the case collection", "response": "def add_case(self, case, update=False):\n \"\"\"Add a case to the case collection\n\n If the case exists and update is False raise error.\n\n Args:\n db (MongoClient): A connection to the mongodb\n case (dict): A case dictionary\n update(bool): If existing case should be updated\n\n Returns:\n mongo_case_id(ObjectId)\n\n \"\"\"\n existing_case = self.case(case)\n if existing_case and not update:\n raise CaseError(\"Case {} already exists\".format(case['case_id']))\n if existing_case:\n self.db.case.find_one_and_replace(\n {'case_id': case['case_id']},\n case,\n )\n else:\n self.db.case.insert_one(case)\n\n return case"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndeletes a case from the database", "response": "def delete_case(self, case):\n \"\"\"Delete case from the database\n\n Delete a case from the database\n\n Args:\n case (dict): A case dictionary\n\n \"\"\"\n mongo_case = self.case(case)\n\n if not mongo_case:\n raise CaseError(\"Tried to delete case {0} but could not find case\".format(\n case.get('case_id')\n ))\n LOG.info(\"Removing case {0} from database\".format(\n mongo_case.get('case_id')\n ))\n self.db.case.delete_one({'_id': mongo_case['_id']})\n\n return"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef make_sink_proxies(drivers):\n ''' Build a list of sink proxies. sink proxies are a two-level ordered\n dictionary. The first level contains the lst of drivers, and the second\n level contains the list of sink proxies for each driver:\n\n drv1-->sink1\n | |->sink2\n |\n drv2-->sink1\n |->sink2\n '''\n sink_proxies = OrderedDict()\n if drivers is not None:\n for driver_name in drivers._fields:\n driver = getattr(drivers, driver_name)\n driver_sink = getattr(driver, 'input')\n if driver_sink is not None:\n driver_sink_proxies = OrderedDict()\n for name in driver_sink._fields:\n driver_sink_proxies[name] = Subject()\n\n sink_proxies[driver_name] = driver.input(**driver_sink_proxies)\n return sink_proxies", "response": "Build a list of sink proxies for each driver."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef build_profile_variant(variant):\n\n chrom = variant.CHROM\n if chrom.startswith(('chr', 'CHR', 'Chr')):\n chrom = chrom[3:]\n\n pos = int(variant.POS)\n\n variant_id = get_variant_id(variant)\n\n ref = variant.REF\n alt = variant.ALT[0]\n\n maf = get_maf(variant)\n\n profile_variant = ProfileVariant(\n variant_id=variant_id,\n chrom=chrom,\n pos=pos,\n ref=ref,\n alt=alt,\n maf=maf,\n id_column = variant.ID\n )\n\n return profile_variant", "response": "Builds a ProfileVariant object from a Cyvcf2. Variant object."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_headers(vcf_obj, nr_cases=None, sv=False):\n\n vcf_obj.add_info_to_header(\n {\n 'ID':\"Obs\",\n 'Number': '1',\n 'Type': 'Integer',\n 'Description': \"The number of observations for the variant\"}\n )\n if not sv:\n vcf_obj.add_info_to_header(\n {\n 'ID':\"Hom\",\n 'Number': '1',\n 'Type': 'Integer',\n 'Description': \"The number of observed homozygotes\"}\n )\n vcf_obj.add_info_to_header(\n {\n 'ID':\"Hem\",\n 'Number': '1',\n 'Type': 'Integer',\n 'Description': \"The number of observed hemizygotes\"}\n )\n if nr_cases:\n case_header = \"##NrCases={}\".format(nr_cases)\n vcf_obj.add_to_header(case_header)\n # head.add_version_tracking(\"loqusdb\", version, datetime.now().strftime(\"%Y-%m-%d %H:%M\"))\n return", "response": "Add loqus specific information to a VCF header"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a VCF object for a given file path.", "response": "def get_file_handle(file_path):\n \"\"\"Return cyvcf2 VCF object\n\n Args:\n file_path(str)\n\n Returns:\n vcf_obj(cyvcf2.VCF)\n \"\"\"\n LOG.debug(\"Check if file end is correct\")\n\n if not os.path.exists(file_path):\n raise IOError(\"No such file:{0}\".format(file_path))\n\n if not os.path.splitext(file_path)[-1] in VALID_ENDINGS:\n raise IOError(\"Not a valid vcf file name: {}\".format(file_path))\n\n vcf_obj = VCF(file_path)\n\n return vcf_obj"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if the vcf file is in correct format.", "response": "def check_vcf(vcf_path, expected_type='snv'):\n \"\"\"Check if there are any problems with the vcf file\n\n Args:\n vcf_path(str)\n expected_type(str): 'sv' or 'snv'\n\n Returns:\n vcf_info(dict): dict like\n {\n 'nr_variants':,\n 'variant_type': in ['snv', 'sv'],\n 'individuals': individual positions in file\n }\n \"\"\"\n LOG.info(\"Check if vcf is on correct format...\")\n\n vcf = VCF(vcf_path)\n individuals = vcf.samples\n variant_type = None\n\n previous_pos = None\n previous_chrom = None\n\n posititon_variants = set()\n\n nr_variants = 0\n for nr_variants,variant in enumerate(vcf,1):\n\n # Check the type of variant\n current_type = 'sv' if variant.var_type == 'sv' else 'snv'\n if not variant_type:\n variant_type = current_type\n\n # Vcf can not include both snvs and svs\n if variant_type != current_type:\n raise VcfError(\"Vcf includes a mix of snvs and svs\")\n\n current_chrom = variant.CHROM\n current_pos = variant.POS\n\n # We start with a simple id that can be used by SV:s\n variant_id = \"{0}_{1}\".format(current_chrom, current_pos)\n # For SNVs we can create a proper variant id with chrom_pos_ref_alt\n if variant_type == 'snv':\n variant_id = get_variant_id(variant)\n\n # Initiate variables\n if not previous_chrom:\n previous_chrom = current_chrom\n previous_pos = current_pos\n posititon_variants = set([variant_id])\n continue\n\n # Update variables if new chromosome\n if current_chrom != previous_chrom:\n previous_chrom = current_chrom\n previous_pos = current_pos\n posititon_variants = set([variant_id])\n continue\n\n if variant_type == 'snv':\n # Check if variant is unique\n if current_pos == previous_pos:\n if variant_id in posititon_variants:\n raise VcfError(\"Variant {0} occurs several times\"\\\n \" in vcf\".format(variant_id))\n else:\n posititon_variants.add(variant_id)\n # Check if vcf is sorted\n else:\n if not current_pos >= previous_pos:\n raise VcfError(\"Vcf if not sorted in a correct way\")\n previous_pos = current_pos\n # Reset posititon_variants since we are on a new position\n posititon_variants = set([variant_id])\n\n if variant_type != expected_type:\n raise VcfError(\"VCF file does not only include {0}s, please check vcf {1}\".format(\n expected_type.upper(), vcf_path))\n\n LOG.info(\"Vcf file %s looks fine\", vcf_path)\n LOG.info(\"Nr of variants in vcf: {0}\".format(nr_variants))\n LOG.info(\"Type of variants in vcf: {0}\".format(variant_type))\n\n vcf_info = {\n 'nr_variants': nr_variants,\n 'variant_type': variant_type,\n 'individuals': individuals,\n }\n\n return vcf_info"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef is_dn(s):\n if s == '':\n return True\n rm = DN_REGEX.match(s)\n return rm is not None and rm.group(0) == s", "response": "Return True if s is a LDAP DN."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nwriting string line as one or more folded lines.", "response": "def _fold_line(self, line):\n \"\"\"Write string line as one or more folded lines.\"\"\"\n if len(line) <= self._cols:\n self._output_file.write(line)\n self._output_file.write(self._line_sep)\n else:\n pos = self._cols\n self._output_file.write(line[0:self._cols])\n self._output_file.write(self._line_sep)\n while pos < len(line):\n self._output_file.write(b' ')\n end = min(len(line), pos + self._cols - 1)\n self._output_file.write(line[pos:end])\n self._output_file.write(self._line_sep)\n pos = end"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _needs_base64_encoding(self, attr_type, attr_value):\n return attr_type.lower() in self._base64_attrs or \\\n isinstance(attr_value, bytes) or \\\n UNSAFE_STRING_RE.search(attr_value) is not None", "response": "Return True if attr_value has to be base - 64 encoded."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nwrites a single attribute type / value pair.", "response": "def _unparse_attr(self, attr_type, attr_value):\n \"\"\"Write a single attribute type/value pair.\"\"\"\n if self._needs_base64_encoding(attr_type, attr_value):\n if not isinstance(attr_value, bytes):\n attr_value = attr_value.encode(self._encoding)\n encoded = base64.encodestring(attr_value)\\\n .replace(b'\\n', b'')\\\n .decode('ascii')\n line = ':: '.join([attr_type, encoded])\n else:\n line = ': '.join([attr_type, attr_value])\n self._fold_line(line.encode('ascii'))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _unparse_entry_record(self, entry):\n for attr_type in sorted(entry.keys()):\n for attr_value in entry[attr_type]:\n self._unparse_attr(attr_type, attr_value)", "response": "Unparse a record from the database."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _unparse_changetype(self, mod_len):\n if mod_len == 2:\n changetype = 'add'\n elif mod_len == 3:\n changetype = 'modify'\n else:\n raise ValueError(\"modlist item of wrong length\")\n\n self._unparse_attr('changetype', changetype)", "response": "Detect and write the changetype."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nwrites an entry or change record to the output file.", "response": "def unparse(self, dn, record):\n \"\"\"Write an entry or change record to the output file.\n\n :type dn: string\n :param dn: distinguished name\n\n :type record: Union[Dict[string, List[string]], List[Tuple]]\n :param record: Either a dictionary holding an entry or a list of\n additions (2-tuple) or modifications (3-tuple).\n \"\"\"\n self._unparse_attr('dn', dn)\n if isinstance(record, dict):\n self._unparse_entry_record(record)\n elif isinstance(record, list):\n self._unparse_change_record(record)\n else:\n raise ValueError(\"Argument record must be dictionary or list\")\n self._output_file.write(self._line_sep)\n self.records_written += 1"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nstripping trailing line separators from s but no other whitespaces.", "response": "def _strip_line_sep(self, s):\n \"\"\"Strip trailing line separators from s, but no other whitespaces.\"\"\"\n if s[-2:] == b'\\r\\n':\n return s[:-2]\n elif s[-1:] == b'\\n':\n return s[:-1]\n else:\n return s"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\niterating input unfoled lines. Skip comments.", "response": "def _iter_unfolded_lines(self):\n \"\"\"Iter input unfoled lines. Skip comments.\"\"\"\n line = self._input_file.readline()\n while line:\n self.line_counter += 1\n self.byte_counter += len(line)\n\n line = self._strip_line_sep(line)\n\n nextline = self._input_file.readline()\n while nextline and nextline[:1] == b' ':\n line += self._strip_line_sep(nextline)[1:]\n nextline = self._input_file.readline()\n\n if not line.startswith(b'#'):\n yield line\n line = nextline"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\niterates input lines in blocks separated by blank lines.", "response": "def _iter_blocks(self):\n \"\"\"Iter input lines in blocks separated by blank lines.\"\"\"\n lines = []\n for line in self._iter_unfolded_lines():\n if line:\n lines.append(line)\n elif lines:\n self.records_read += 1\n yield lines\n lines = []\n if lines:\n self.records_read += 1\n yield lines"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _parse_attr(self, line):\n colon_pos = line.index(b':')\n attr_type = line[0:colon_pos].decode('ascii')\n\n if line[colon_pos:].startswith(b'::'):\n attr_value = base64.decodestring(line[colon_pos + 2:])\n elif line[colon_pos:].startswith(b':<'):\n url = line[colon_pos + 2:].strip()\n attr_value = b''\n if self._process_url_schemes:\n u = urlparse(url)\n if u[0] in self._process_url_schemes:\n attr_value = urlopen(url.decode('ascii')).read()\n else:\n attr_value = line[colon_pos + 1:].strip()\n\n return self._decode_value(attr_type, attr_value)", "response": "Parse a single attribute type and value pair."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _check_dn(self, dn, attr_value):\n if dn is not None:\n self._error('Two lines starting with dn: in one record.')\n if not is_dn(attr_value):\n self._error('No valid string-representation of '\n 'distinguished name %s.' % attr_value)", "response": "Check dn attribute for issues."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nchecking changetype attribute for issues.", "response": "def _check_changetype(self, dn, changetype, attr_value):\n \"\"\"Check changetype attribute for issues.\"\"\"\n if dn is None:\n self._error('Read changetype: before getting valid dn: line.')\n if changetype is not None:\n self._error('Two lines starting with changetype: in one record.')\n if attr_value not in CHANGE_TYPES:\n self._error('changetype value %s is invalid.' % attr_value)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _parse_entry_record(self, lines):\n dn = None\n entry = OrderedDict()\n\n for line in lines:\n attr_type, attr_value = self._parse_attr(line)\n\n if attr_type == 'dn':\n self._check_dn(dn, attr_value)\n dn = attr_value\n elif attr_type == 'version' and dn is None:\n pass # version = 1\n else:\n if dn is None:\n self._error('First line of record does not start '\n 'with \"dn:\": %s' % attr_type)\n if attr_value is not None and \\\n attr_type.lower() not in self._ignored_attr_types:\n if attr_type in entry:\n entry[attr_type].append(attr_value)\n else:\n entry[attr_type] = [attr_value]\n\n return dn, entry", "response": "Parse a single entry record from a list of lines."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _create_w_objective(m, X, Z=None):\n genes, clusters = m.shape\n cells = X.shape[1]\n nonzeros = (X!=0)\n def objective(w):\n # convert w into a matrix first... because it's a vector for\n # optimization purposes\n w = w.reshape((m.shape[1], X.shape[1]))\n d = m.dot(w)+eps\n # derivative of objective wrt all elements of w\n # for w_{ij}, the derivative is... m_j1+...+m_jn sum over genes minus \n # x_ij\n temp = X/d\n m_sum = m.T.dot(nonzeros)\n m2 = m.T.dot(temp)\n deriv = m_sum - m2\n return np.sum(nonzeros*(d - X*np.log(d)))/genes, deriv.flatten()/genes\n return objective", "response": "Create an objective function and its derivative for W given M and X."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nuse a Zero - inflated Poisson Mixture model to estimate cell states and cell mixing weights.", "response": "def zip_estimate_state(data, clusters, init_means=None, init_weights=None, max_iters=10, tol=1e-4, disp=True, inner_max_iters=400, normalize=True):\n \"\"\"\n Uses a Zero-inflated Poisson Mixture model to estimate cell states and\n cell state mixing weights.\n\n Args:\n data (array): genes x cells\n clusters (int): number of mixture components\n init_means (array, optional): initial centers - genes x clusters. Default: kmeans++ initializations\n init_weights (array, optional): initial weights - clusters x cells. Default: random(0,1)\n max_iters (int, optional): maximum number of iterations. Default: 10\n tol (float, optional): if both M and W change by less than tol (in RMSE), then the iteration is stopped. Default: 1e-4\n disp (bool, optional): whether or not to display optimization parameters. Default: True\n inner_max_iters (int, optional): Number of iterations to run in the scipy minimizer for M and W. Default: 400\n normalize (bool, optional): True if the resulting W should sum to 1 for each cell. Default: True.\n\n Returns:\n M: genes x clusters - state centers\n W: clusters x cells - state mixing components for each cell\n ll: final log-likelihood\n \"\"\"\n genes, cells = data.shape\n # TODO: estimate ZIP parameter?\n if init_means is None:\n means, assignments = kmeans_pp(data, clusters)\n else:\n means = init_means.copy()\n clusters = means.shape[1]\n w_init = np.random.random(cells*clusters)\n if init_weights is not None:\n if len(init_weights.shape)==1:\n init_weights = initialize_from_assignments(init_weights, clusters)\n w_init = init_weights.reshape(cells*clusters)\n m_init = means.reshape(genes*clusters)\n # using zero-inflated parameters...\n L, Z = zip_fit_params_mle(data)\n # repeat steps 1 and 2 until convergence:\n ll = np.inf\n for i in range(max_iters):\n if disp:\n print('iter: {0}'.format(i))\n w_bounds = [(0, 1.0) for x in w_init]\n m_bounds = [(0, None) for x in m_init]\n # step 1: given M, estimate W\n w_objective = _create_w_objective(means, data, Z)\n w_res = minimize(w_objective, w_init, method='L-BFGS-B', jac=True, bounds=w_bounds, options={'disp':disp, 'maxiter':inner_max_iters})\n w_diff = np.sqrt(np.sum((w_res.x-w_init)**2))/w_init.size\n w_new = w_res.x.reshape((clusters, cells))\n w_init = w_res.x\n # step 2: given W, update M\n m_objective = _create_m_objective(w_new, data, Z)\n # method could be 'L-BFGS-B' or 'SLSQP'... SLSQP gives a memory error...\n # or use TNC...\n m_res = minimize(m_objective, m_init, method='L-BFGS-B', jac=True, bounds=m_bounds, options={'disp':disp, 'maxiter':inner_max_iters})\n ll = m_res.fun\n m_diff = np.sqrt(np.sum((m_res.x-m_init)**2))/m_init.size\n m_new = m_res.x.reshape((genes, clusters))\n m_init = m_res.x\n means = m_new\n if w_diff < tol and m_diff < tol:\n break\n if normalize:\n w_new = w_new/w_new.sum(0)\n return m_new, w_new, ll"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngenerate k - means ++ initial centers.", "response": "def kmeans_pp(data, k, centers=None):\n \"\"\"\n Generates kmeans++ initial centers.\n\n Args:\n data (array): A 2d array- genes x cells\n k (int): Number of clusters\n centers (array, optional): if provided, these are one or more known cluster centers. 2d array of genes x number of centers (<=k).\n\n Returns:\n centers - a genes x k array of cluster means.\n assignments - a cells x 1 array of cluster assignments\n \"\"\"\n # TODO: what if there is missing data for a given gene?\n # missing data could be if all the entires are -1.\n genes, cells = data.shape\n if sparse.issparse(data) and not sparse.isspmatrix_csc(data):\n data = sparse.csc_matrix(data)\n num_known_centers = 0\n if centers is None:\n centers = np.zeros((genes, k))\n else:\n num_known_centers = centers.shape[1]\n centers = np.concatenate((centers, np.zeros((genes, k-num_known_centers))), 1)\n distances = np.zeros((cells, k))\n distances[:] = np.inf\n if num_known_centers == 0:\n init = np.random.randint(0, cells)\n if sparse.issparse(data):\n centers[:,0] = data[:, init].toarray().flatten()\n else:\n centers[:,0] = data[:, init]\n num_known_centers+=1\n available_cells = list(range(cells))\n for c in range(num_known_centers, k):\n c2 = c-1\n # use different formulation for distance... if sparse, use lls\n # if not sparse, use poisson_dist\n if sparse.issparse(data):\n lls = poisson_ll(data, centers[:,c2:c2+1]).flatten()\n distances[:,c2] = 1 + lls.max() - lls\n distances[:,c2] /= distances[:,c2].max()\n else:\n for cell in range(cells):\n distances[cell, c2] = poisson_dist(data[:,cell], centers[:,c2])\n # choose a new data point as center... probability proportional\n # to distance^2\n min_distances = np.min(distances, 1)\n min_distances = min_distances**2\n min_distances = min_distances[available_cells]\n # should be sampling without replacement\n min_dist = np.random.choice(available_cells,\n p=min_distances/min_distances.sum())\n available_cells.pop(available_cells.index(min_dist))\n if sparse.issparse(data):\n centers[:,c] = data[:, min_dist].toarray().flatten()\n else:\n centers[:,c] = data[:, min_dist]\n lls = poisson_ll(data, centers)\n new_assignments = np.argmax(lls, 1)\n centers[centers==0.0] = eps\n return centers, new_assignments"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef poisson_cluster(data, k, init=None, max_iters=100):\n # TODO: be able to use a combination of fixed and unknown starting points\n # e.g., have init values only for certain genes, have a row of all\n # zeros indicating that kmeans++ should be used for that row.\n genes, cells = data.shape\n #print 'starting: ', centers\n if sparse.issparse(data) and not sparse.isspmatrix_csc(data):\n data = sparse.csc_matrix(data)\n init, assignments = kmeans_pp(data, k, centers=init)\n centers = np.copy(init)\n assignments = np.zeros(cells)\n for it in range(max_iters):\n lls = poisson_ll(data, centers)\n #cluster_dists = np.zeros((cells, k))\n new_assignments = np.argmax(lls, 1)\n if np.equal(assignments, new_assignments).all():\n #print 'ending: ', centers\n return new_assignments, centers\n for c in range(k):\n if sparse.issparse(data):\n if data[:,new_assignments==c].shape[0]==0:\n # re-initialize centers?\n new_c, _ = kmeans_pp(data, k, centers[:,:c])\n centers[:,c] = new_c[:,c]\n else:\n centers[:,c] = np.asarray(data[:,new_assignments==c].mean(1)).flatten()\n else:\n if len(data[:,new_assignments==c])==0:\n new_c, _ = kmeans_pp(data, k, centers[:,:c])\n centers[:,c] = new_c[:,c]\n else:\n centers[:,c] = np.mean(data[:,new_assignments==c], 1)\n assignments = new_assignments\n return assignments, centers", "response": "Performs Poisson hard EM on the given data."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndisplays cases in the database.", "response": "def cases(ctx, case_id, to_json):\n \"\"\"Display cases in the database.\"\"\"\n\n adapter = ctx.obj['adapter']\n cases = []\n\n if case_id:\n case_obj = adapter.case({'case_id':case_id})\n if not case_obj:\n LOG.info(\"Case {0} does not exist in database\".format(case_id))\n return\n case_obj['_id'] = str(case_obj['_id'])\n cases.append(case_obj)\n else:\n cases = adapter.cases()\n if cases.count() == 0:\n LOG.info(\"No cases found in database\")\n ctx.abort()\n\n if to_json:\n click.echo(json.dumps(cases))\n return\n\n click.echo(\"#case_id\\tvcf_path\")\n\n for case_obj in cases:\n click.echo(\"{0}\\t{1}\".format(case_obj.get('case_id'), case_obj.get('vcf_path')))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndisplaying variants in the database.", "response": "def variants(ctx, variant_id, chromosome, end_chromosome, start, end, variant_type,\n sv_type):\n \"\"\"Display variants in the database.\"\"\"\n if sv_type:\n variant_type = 'sv'\n\n adapter = ctx.obj['adapter']\n\n if (start or end):\n if not (chromosome and start and end):\n LOG.warning(\"Regions must be specified with chromosome, start and end\")\n return\n\n if variant_id:\n variant = adapter.get_variant({'_id':variant_id})\n if variant:\n click.echo(variant)\n else:\n LOG.info(\"Variant {0} does not exist in database\".format(variant_id))\n return\n\n if variant_type == 'snv':\n result = adapter.get_variants(\n chromosome=chromosome,\n start=start,\n end=end\n )\n else:\n LOG.info(\"Search for svs\")\n result = adapter.get_sv_variants(\n chromosome=chromosome,\n end_chromosome=end_chromosome,\n sv_type=sv_type,\n pos=start,\n end=end\n )\n\n i = 0\n for variant in result:\n i += 1\n pp(variant)\n\n LOG.info(\"Number of variants found in database: %s\", i)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef dotd(A, B, out=None):\n A = asarray(A, float)\n B = asarray(B, float)\n if A.ndim == 1 and B.ndim == 1:\n if out is None:\n return dot(A, B)\n return dot(A, B, out)\n if out is None:\n out = empty((A.shape[0],), float)\n return einsum(\"ij,ji->i\", A, B, out=out)", "response": "r Diagonal of A \\ mathrm A \\ mathrm B \\ intercal."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef ddot(L, R, left=None, out=None):\n L = asarray(L, float)\n R = asarray(R, float)\n if left is None:\n ok = min(L.ndim, R.ndim) == 1 and max(L.ndim, R.ndim) == 2\n if not ok:\n msg = \"Wrong array layout. One array should have\"\n msg += \" ndim=1 and the other one ndim=2.\"\n raise ValueError(msg)\n left = L.ndim == 1\n if left:\n if out is None:\n out = copy(R)\n L = L.reshape(list(L.shape) + [1] * (R.ndim - 1))\n return multiply(L, R, out=out)\n else:\n if out is None:\n out = copy(L)\n return multiply(L, R, out=out)", "response": "r Dot product of a matrix and a diagonal one."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef nanrankdata(a, axis=-1, inplace=False):\n from scipy.stats import rankdata\n\n if hasattr(a, \"dtype\") and issubdtype(a.dtype, integer):\n raise ValueError(\"Integer type is not supported.\")\n\n if isinstance(a, (tuple, list)):\n if inplace:\n raise ValueError(\"Can't use `inplace=True` for {}.\".format(type(a)))\n a = asarray(a, float)\n\n orig_shape = a.shape\n if a.ndim == 1:\n a = a.reshape(orig_shape + (1,))\n\n if not inplace:\n a = a.copy()\n\n def rank1d(x):\n idx = ~isnan(x)\n x[idx] = rankdata(x[idx])\n return x\n\n a = a.swapaxes(1, axis)\n a = apply_along_axis(rank1d, 0, a)\n a = a.swapaxes(1, axis)\n\n return a.reshape(orig_shape)", "response": "Rank data for arrays contaning NaN values."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef economic_qs(K, epsilon=sqrt(finfo(float).eps)):\n\n (S, Q) = eigh(K)\n\n nok = abs(max(Q[0].min(), Q[0].max(), key=abs)) < epsilon\n nok = nok and abs(max(K.min(), K.max(), key=abs)) >= epsilon\n if nok:\n from scipy.linalg import eigh as sp_eigh\n\n (S, Q) = sp_eigh(K)\n\n ok = S >= epsilon\n nok = logical_not(ok)\n S0 = S[ok]\n Q0 = Q[:, ok]\n Q1 = Q[:, nok]\n return ((Q0, Q1), S0)", "response": "r Economic eigen decomposition for symmetric matrices."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef unique(ar):\n\n import dask.array as da\n\n if isinstance(ar, da.core.Array):\n return da.unique(ar)\n\n return _unique(ar)", "response": "r Find the unique elements of an array."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef lu_slogdet(LU):\n LU = (asarray(LU[0], float), asarray(LU[1], float))\n adet = _sum(log(_abs(LU[0].diagonal())))\n\n s = prod(sign(LU[0].diagonal()))\n\n nrows_exchange = LU[1].size - _sum(LU[1] == arange(LU[1].size, dtype=\"int32\"))\n\n odd = nrows_exchange % 2 == 1\n if odd:\n s *= -1.0\n\n return (s, adet)", "response": "Natural logarithm of a LU decomposition."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef lu_solve(LU, b):\n from scipy.linalg import lu_solve as sp_lu_solve\n\n LU = (asarray(LU[0], float), asarray(LU[1], float))\n b = asarray(b, float)\n return sp_lu_solve(LU, b, check_finite=False)", "response": "r Solve for LU decomposition."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef dotd(A, B):\n A = asarray(A, float)\n B = asarray(B, float)\n if A.ndim == 1 and B.ndim == 1:\n return dot(A, B)\n\n out = empty((A.shape[0],), float)\n out[:] = sum(A * B.T, axis=1)\n\n return out", "response": "r Diagonal of A \\ mathrm A \\ mathrm B \\ intercal."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef economic_svd(G, epsilon=sqrt(finfo(float).eps)):\n from scipy.linalg import svd\n\n G = asarray(G, float)\n (U, S, V) = svd(G, full_matrices=False, check_finite=False)\n ok = S >= epsilon\n S = S[ok]\n U = U[:, ok]\n V = V[ok, :]\n return (U, S, V)", "response": "r Economic Singular Value Decomposition."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef rsolve(A, b, epsilon=_epsilon):\n A = asarray(A, float)\n b = asarray(b, float)\n if A.shape[0] == 0:\n return zeros((A.shape[1],))\n if A.shape[1] == 0:\n return zeros((0,))\n try:\n x = lstsq(A, b, rcond=epsilon)\n r = sum(x[3] > epsilon)\n if r == 0:\n return zeros(A.shape[1])\n return x[0]\n except (ValueError, LinAlgError) as e:\n warnings.warn(str(e), RuntimeWarning)\n return solve(A, b)", "response": "r Robust solve for the linear equations."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncheck if A is a semi - definite positive matrix.", "response": "def check_semidefinite_positiveness(A):\n \"\"\"Check if ``A`` is a semi-definite positive matrix.\n\n Args:\n A (array_like): Matrix.\n\n Returns:\n bool: ``True`` if ``A`` is definite positive; ``False`` otherwise.\n \"\"\"\n B = empty_like(A)\n B[:] = A\n B[diag_indices_from(B)] += sqrt(finfo(float).eps)\n try:\n cholesky(B)\n except LinAlgError:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef check_symmetry(A):\n A = asanyarray(A)\n if A.ndim != 2:\n raise ValueError(\"Checks symmetry only for bi-dimensional arrays.\")\n\n if A.shape[0] != A.shape[1]:\n return False\n\n return abs(A - A.T).max() < sqrt(finfo(float).eps)", "response": "Checks if A is a symmetric matrix."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef cho_solve(L, b):\n from scipy.linalg import cho_solve as sp_cho_solve\n\n L = asarray(L, float)\n b = asarray(b, float)\n if L.size == 0:\n if b.size != 0:\n raise ValueError(\"Dimension mismatch between L and b.\")\n return empty(b.shape)\n return sp_cho_solve((L, True), b, check_finite=False)", "response": "r Solve for Cholesky decomposition."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget filename and check if in getcwd then get from the package resources folder", "response": "def file_or_resource(fname=None):\n '''get filename and check if in getcwd then get from\n the package resources folder\n '''\n if fname is not None:\n filename = os.path.expanduser(fname)\n \n resource_package = opentargets_validator.__name__\n resource_path = os.path.sep.join(('resources', filename))\n\n abs_filename = os.path.join(os.path.abspath(os.getcwd()), filename) \\\n if not os.path.isabs(filename) else filename\n\n return abs_filename if os.path.isfile(abs_filename) \\\n else res.resource_filename(resource_package, resource_path)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef stl(A, b):\n from scipy.linalg import solve_triangular\n\n A = asarray(A, float)\n b = asarray(b, float)\n return solve_triangular(A, b, lower=True, check_finite=False)", "response": "r Solve a lower - triangular matrix A and b."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _windows_resolve(command):\n try:\n import win32api\n except ImportError:\n if (2, 8) < sys.version_info < (3, 5):\n logger.info(\n \"Resolving executable names only supported on Python 2.7 and 3.5+\"\n )\n else:\n logger.warning(\n \"Could not resolve executable name: package win32api missing\"\n )\n return command\n\n if not command or not isinstance(command[0], six.string_types):\n return command\n\n try:\n _, found_executable = win32api.FindExecutable(command[0])\n logger.debug(\"Resolved %s as %s\", command[0], found_executable)\n return (found_executable,) + tuple(command[1:])\n except Exception as e:\n if not hasattr(e, \"winerror\"):\n raise\n # Keep this error message for later in case we fail to resolve the name\n logwarning = getattr(e, \"strerror\", str(e))\n\n if \".\" in command[0]:\n # Special case. The win32api may not properly check file extensions, so\n # try to resolve the executable explicitly.\n for extension in os.getenv(\"PATHEXT\").split(os.pathsep):\n try:\n _, found_executable = win32api.FindExecutable(command[0] + extension)\n logger.debug(\"Resolved %s as %s\", command[0], found_executable)\n return (found_executable,) + tuple(command[1:])\n except Exception as e:\n if not hasattr(e, \"winerror\"):\n raise\n\n logger.warning(\"Error trying to resolve the executable: %s\", logwarning)\n return command", "response": "Resolves the executable name and returns the command array with the resolved name."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nruns a command line and returns a dictionary containing the output of the command line and the exit code.", "response": "def run(\n command,\n timeout=None,\n debug=False,\n stdin=None,\n print_stdout=True,\n print_stderr=True,\n callback_stdout=None,\n callback_stderr=None,\n environment=None,\n environment_override=None,\n win32resolve=True,\n working_directory=None,\n):\n \"\"\"\n Run an external process.\n\n File system path objects (PEP-519) are accepted in the command, environment,\n and working directory arguments.\n\n :param array command: Command line to be run, specified as array.\n :param timeout: Terminate program execution after this many seconds.\n :param boolean debug: Enable further debug messages.\n :param stdin: Optional string that is passed to command stdin.\n :param boolean print_stdout: Pass stdout through to sys.stdout.\n :param boolean print_stderr: Pass stderr through to sys.stderr.\n :param callback_stdout: Optional function which is called for each\n stdout line.\n :param callback_stderr: Optional function which is called for each\n stderr line.\n :param dict environment: The full execution environment for the command.\n :param dict environment_override: Change environment variables from the\n current values for command execution.\n :param boolean win32resolve: If on Windows, find the appropriate executable\n first. This allows running of .bat, .cmd, etc.\n files without explicitly specifying their\n extension.\n :param string working_directory: If specified, run the executable from\n within this working directory.\n :return: A ReturnObject() containing the executed command, stdout and stderr\n (both as bytestrings), and the exitcode. Further values such as\n process runtime can be accessed as dictionary values.\n \"\"\"\n\n time_start = time.strftime(\"%Y-%m-%d %H:%M:%S GMT\", time.gmtime())\n logger.debug(\"Starting external process: %s\", command)\n\n if stdin is None:\n stdin_pipe = None\n else:\n assert sys.platform != \"win32\", \"stdin argument not supported on Windows\"\n stdin_pipe = subprocess.PIPE\n\n start_time = timeit.default_timer()\n if timeout is not None:\n max_time = start_time + timeout\n\n if environment is not None:\n env = {key: _path_resolve(environment[key]) for key in environment}\n else:\n env = os.environ\n if environment_override:\n env = copy.copy(env)\n env.update(\n {\n key: str(_path_resolve(environment_override[key]))\n for key in environment_override\n }\n )\n\n command = tuple(_path_resolve(part) for part in command)\n if win32resolve and sys.platform == \"win32\":\n command = _windows_resolve(command)\n\n p = subprocess.Popen(\n command,\n shell=False,\n cwd=_path_resolve(working_directory),\n env=env,\n stdin=stdin_pipe,\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE,\n )\n\n thread_pipe_pool = []\n notifyee, notifier = Pipe(False)\n thread_pipe_pool.append(notifyee)\n stdout = _NonBlockingStreamReader(\n p.stdout,\n output=print_stdout,\n debug=debug,\n notify=notifier.close,\n callback=callback_stdout,\n )\n notifyee, notifier = Pipe(False)\n thread_pipe_pool.append(notifyee)\n stderr = _NonBlockingStreamReader(\n p.stderr,\n output=print_stderr,\n debug=debug,\n notify=notifier.close,\n callback=callback_stderr,\n )\n if stdin is not None:\n notifyee, notifier = Pipe(False)\n thread_pipe_pool.append(notifyee)\n stdin = _NonBlockingStreamWriter(\n p.stdin, data=stdin, debug=debug, notify=notifier.close\n )\n\n timeout_encountered = False\n\n while (p.returncode is None) and (\n (timeout is None) or (timeit.default_timer() < max_time)\n ):\n if debug and timeout is not None:\n logger.debug(\"still running (T%.2fs)\" % (timeit.default_timer() - max_time))\n\n # wait for some time or until a stream is closed\n try:\n if thread_pipe_pool:\n # Wait for up to 0.5 seconds or for a signal on a remaining stream,\n # which could indicate that the process has terminated.\n try:\n event = thread_pipe_pool[0].poll(0.5)\n except IOError as e:\n # on Windows this raises \"IOError: [Errno 109] The pipe has been ended\"\n # which is for all intents and purposes equivalent to a True return value.\n if e.errno != 109:\n raise\n event = True\n if event:\n # One-shot, so remove stream and watch remaining streams\n thread_pipe_pool.pop(0)\n if debug:\n logger.debug(\"Event received from stream thread\")\n else:\n time.sleep(0.5)\n except KeyboardInterrupt:\n p.kill() # if user pressed Ctrl+C we won't be able to produce a proper report anyway\n # but at least make sure the child process dies with us\n raise\n\n # check if process is still running\n p.poll()\n\n if p.returncode is None:\n # timeout condition\n timeout_encountered = True\n if debug:\n logger.debug(\"timeout (T%.2fs)\" % (timeit.default_timer() - max_time))\n\n # send terminate signal and wait some time for buffers to be read\n p.terminate()\n if thread_pipe_pool:\n thread_pipe_pool[0].poll(0.5)\n if not stdout.has_finished() or not stderr.has_finished():\n time.sleep(2)\n p.poll()\n\n if p.returncode is None:\n # thread still alive\n # send kill signal and wait some more time for buffers to be read\n p.kill()\n if thread_pipe_pool:\n thread_pipe_pool[0].poll(0.5)\n if not stdout.has_finished() or not stderr.has_finished():\n time.sleep(5)\n p.poll()\n\n if p.returncode is None:\n raise RuntimeError(\"Process won't terminate\")\n\n runtime = timeit.default_timer() - start_time\n if timeout is not None:\n logger.debug(\n \"Process ended after %.1f seconds with exit code %d (T%.2fs)\"\n % (runtime, p.returncode, timeit.default_timer() - max_time)\n )\n else:\n logger.debug(\n \"Process ended after %.1f seconds with exit code %d\"\n % (runtime, p.returncode)\n )\n\n stdout = stdout.get_output()\n stderr = stderr.get_output()\n time_end = time.strftime(\"%Y-%m-%d %H:%M:%S GMT\", time.gmtime())\n\n result = ReturnObject(\n {\n \"exitcode\": p.returncode,\n \"command\": command,\n \"stdout\": stdout,\n \"stderr\": stderr,\n \"timeout\": timeout_encountered,\n \"runtime\": runtime,\n \"time_start\": time_start,\n \"time_end\": time_end,\n }\n )\n if stdin is not None:\n result.update(\n {\n \"stdin_bytes_sent\": stdin.bytes_sent(),\n \"stdin_bytes_remain\": stdin.bytes_remaining(),\n }\n )\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef run_process_dummy(command, **kwargs):\n warnings.warn(\n \"procrunner.run_process_dummy() is deprecated\", DeprecationWarning, stacklevel=2\n )\n\n time_start = time.strftime(\"%Y-%m-%d %H:%M:%S GMT\", time.gmtime())\n logger.info(\"run_process is disabled. Requested command: %s\", command)\n\n result = ReturnObject(\n {\n \"exitcode\": 0,\n \"command\": command,\n \"stdout\": \"\",\n \"stderr\": \"\",\n \"timeout\": False,\n \"runtime\": 0,\n \"time_start\": time_start,\n \"time_end\": time_start,\n }\n )\n if kwargs.get(\"stdin\") is not None:\n result.update(\n {\"stdin_bytes_sent\": len(kwargs[\"stdin\"]), \"stdin_bytes_remain\": 0}\n )\n return result", "response": "A stand - in function that returns a valid result dictionary indicating a successful execution of a command."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding a single character to the buffer.", "response": "def add(self, data):\n \"\"\"\n Add a single character to buffer. If one or more full lines are found,\n print them (if desired) and pass to callback function.\n \"\"\"\n data = self._decoder.decode(data)\n if not data:\n return\n self._buffer += data\n if \"\\n\" in data:\n to_print, remainder = self._buffer.rsplit(\"\\n\")\n if self._print:\n try:\n print(to_print)\n except UnicodeEncodeError:\n print(to_print.encode(sys.getdefaultencoding(), errors=\"replace\"))\n if not hasattr(self, \"_warned\"):\n logger.warning(\"output encoding error, characters replaced\")\n setattr(self, \"_warned\", True)\n if self._callback:\n self._callback(to_print)\n self._buffer = remainder"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nprinting and send any remaining data to callback function.", "response": "def flush(self):\n \"\"\"Print/send any remaining data to callback function.\"\"\"\n self._buffer += self._decoder.decode(b\"\", final=True)\n if self._buffer:\n if self._print:\n print(self._buffer)\n if self._callback:\n self._callback(self._buffer)\n self._buffer = \"\""} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nretrieving the stored data in full.", "response": "def get_output(self):\n \"\"\"\n Retrieve the stored data in full.\n This call may block if the reading thread has not yet terminated.\n \"\"\"\n self._closing = True\n if not self.has_finished():\n if self._debug:\n # Main thread overtook stream reading thread.\n underrun_debug_timer = timeit.default_timer()\n logger.warning(\"NBSR underrun\")\n self._thread.join()\n if not self.has_finished():\n if self._debug:\n logger.debug(\n \"NBSR join after %f seconds, underrun not resolved\"\n % (timeit.default_timer() - underrun_debug_timer)\n )\n raise Exception(\"thread did not terminate\")\n if self._debug:\n logger.debug(\n \"NBSR underrun resolved after %f seconds\"\n % (timeit.default_timer() - underrun_debug_timer)\n )\n if self._closed:\n raise Exception(\"streamreader double-closed\")\n self._closed = True\n data = self._buffer.getvalue()\n self._buffer.close()\n return data"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconvert Hiragana to Full - width Katakana.", "response": "def hira2kata(text, ignore=''):\n \"\"\"Convert Hiragana to Full-width (Zenkaku) Katakana.\n\n Parameters\n ----------\n text : str\n Hiragana string.\n ignore : str\n Characters to be ignored in converting.\n\n Return\n ------\n str\n Katakana string.\n\n Examples\n --------\n >>> print(jaconv.hira2kata('\u3068\u3082\u3048\u307e\u307f'))\n \u30c8\u30e2\u30a8\u30de\u30df\n >>> print(jaconv.hira2kata('\u307e\u3069\u307e\u304e', ignore='\u3069'))\n \u30de\u3069\u30de\u30ae\n \"\"\"\n if ignore:\n h2k_map = _exclude_ignorechar(ignore, H2K_TABLE.copy())\n return _convert(text, h2k_map)\n return _convert(text, H2K_TABLE)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef hira2hkata(text, ignore=''):\n if ignore:\n h2hk_map = _exclude_ignorechar(ignore, H2HK_TABLE.copy())\n return _convert(text, h2hk_map)\n return _convert(text, H2HK_TABLE)", "response": "Convert Hiragana to Hankaku"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts Full - width Katakana to Hiragana.", "response": "def kata2hira(text, ignore=''):\n \"\"\"Convert Full-width Katakana to Hiragana\n\n Parameters\n ----------\n text : str\n Full-width Katakana string.\n ignore : str\n Characters to be ignored in converting.\n\n Return\n ------\n str\n Hiragana string.\n\n Examples\n --------\n >>> print(jaconv.kata2hira('\u5df4\u30de\u30df'))\n \u5df4\u307e\u307f\n >>> print(jaconv.kata2hira('\u30de\u30df\u30b5\u30f3', ignore='\u30f3'))\n \u307e\u307f\u3055\u30f3\n \"\"\"\n if ignore:\n k2h_map = _exclude_ignorechar(ignore, K2H_TABLE.copy())\n return _convert(text, k2h_map)\n return _convert(text, K2H_TABLE)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts Hankaku to Zenkaku", "response": "def h2z(text, ignore='', kana=True, ascii=False, digit=False):\n \"\"\"Convert Half-width (Hankaku) Katakana to Full-width (Zenkaku) Katakana\n\n Parameters\n ----------\n text : str\n Half-width Katakana string.\n ignore : str\n Characters to be ignored in converting.\n kana : bool\n Either converting Kana or not.\n ascii : bool\n Either converting ascii or not.\n digit : bool\n Either converting digit or not.\n\n Return\n ------\n str\n Full-width Katakana string.\n\n Examples\n --------\n >>> print(jaconv.h2z('\uff83\uff68\uff9b\uff8c\uff68\uff85\uff70\uff9a'))\n \u30c6\u30a3\u30ed\u30d5\u30a3\u30ca\u30fc\u30ec\n >>> print(jaconv.h2z('\uff83\uff68\uff9b\uff8c\uff68\uff85\uff70\uff9a', ignore='\uff68'))\n \u30c6\uff68\u30ed\u30d5\uff68\u30ca\u30fc\u30ec\n >>> print(jaconv.h2z('abcd', ascii=True))\n \uff21\uff22\uff23\uff24\n >>> print(jaconv.h2z('1234', digit=True))\n \uff11\uff12\uff13\uff14\n \"\"\"\n def _conv_dakuten(text):\n \"\"\"Convert Hankaku Dakuten Kana to Zenkaku Dakuten Kana\n \"\"\"\n text = text.replace(\"\uff76\uff9e\", \"\u30ac\").replace(\"\uff77\uff9e\", \"\u30ae\")\n text = text.replace(\"\uff78\uff9e\", \"\u30b0\").replace(\"\uff79\uff9e\", \"\u30b2\")\n text = text.replace(\"\uff7a\uff9e\", \"\u30b4\").replace(\"\uff7b\uff9e\", \"\u30b6\")\n text = text.replace(\"\uff7c\uff9e\", \"\u30b8\").replace(\"\uff7d\uff9e\", \"\u30ba\")\n text = text.replace(\"\uff7e\uff9e\", \"\u30bc\").replace(\"\uff7f\uff9e\", \"\u30be\")\n text = text.replace(\"\uff80\uff9e\", \"\u30c0\").replace(\"\uff81\uff9e\", \"\u30c2\")\n text = text.replace(\"\uff82\uff9e\", \"\u30c5\").replace(\"\uff83\uff9e\", \"\u30c7\")\n text = text.replace(\"\uff84\uff9e\", \"\u30c9\").replace(\"\uff8a\uff9e\", \"\u30d0\")\n text = text.replace(\"\uff8b\uff9e\", \"\u30d3\").replace(\"\uff8c\uff9e\", \"\u30d6\")\n text = text.replace(\"\uff8d\uff9e\", \"\u30d9\").replace(\"\uff8e\uff9e\", \"\u30dc\")\n text = text.replace(\"\uff8a\uff9f\", \"\u30d1\").replace(\"\uff8b\uff9f\", \"\u30d4\")\n text = text.replace(\"\uff8c\uff9f\", \"\u30d7\").replace(\"\uff8d\uff9f\", \"\u30da\")\n return text.replace(\"\uff8e\uff9f\", \"\u30dd\").replace(\"\uff73\uff9e\", \"\u30f4\")\n\n if ascii:\n if digit:\n if kana:\n h2z_map = H2Z_ALL\n else:\n h2z_map = H2Z_AD\n elif kana:\n h2z_map = H2Z_AK\n else:\n h2z_map = H2Z_A\n elif digit:\n if kana:\n h2z_map = H2Z_DK\n else:\n h2z_map = H2Z_D\n else:\n h2z_map = H2Z_K\n if kana:\n text = _conv_dakuten(text)\n if ignore:\n h2z_map = _exclude_ignorechar(ignore, h2z_map.copy())\n return _convert(text, h2z_map)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef z2h(text, ignore='', kana=True, ascii=False, digit=False):\n if ascii:\n if digit:\n if kana:\n z2h_map = Z2H_ALL\n else:\n z2h_map = Z2H_AD\n elif kana:\n z2h_map = Z2H_AK\n else:\n z2h_map = Z2H_A\n elif digit:\n if kana:\n z2h_map = Z2H_DK\n else:\n z2h_map = Z2H_D\n else:\n z2h_map = Z2H_K\n if ignore:\n z2h_map = _exclude_ignorechar(ignore, z2h_map.copy())\n return _convert(text, z2h_map)", "response": "Convert a string from Zenkaku to Hankaku."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert Hankaku ASCII and DIGIT to Full - width ASCII and DIGIT and Hankaku.", "response": "def normalize(text, mode='NFKC', ignore=''):\n \"\"\"Convert Half-width (Hankaku) Katakana to Full-width (Zenkaku) Katakana,\n Full-width (Zenkaku) ASCII and DIGIT to Half-width (Hankaku) ASCII\n and DIGIT.\n Additionally, Full-width wave dash (\u301c) etc. are normalized\n\n Parameters\n ----------\n text : str\n Source string.\n mode : str\n Unicode normalization mode.\n ignore : str\n Characters to be ignored in converting.\n\n Return\n ------\n str\n Normalized string.\n\n Examples\n --------\n >>> print(jaconv.normalize('\uff83\uff68\uff9b\uff65\u30d5\u30a3\u30ca\u301c\u30ec', 'NFKC'))\n \u30c6\u30a3\u30ed\u30fb\u30d5\u30a3\u30ca\u30fc\u30ec\n \"\"\"\n text = text.replace('\u301c', '\u30fc').replace('\uff5e', '\u30fc')\n text = text.replace(\"\u2019\", \"'\").replace('\u201d', '\"').replace('\u201c', '``')\n text = text.replace('\u2015', '-').replace('\u2010', '-').replace('\u02d7', '-').replace('\u058a', '-')\n text = text.replace('\u2010', '-').replace('\u2011', '-').replace('\u2012', '-').replace('\u2013', '-')\n text = text.replace('\u2043', '-').replace('\u207b', '-').replace('\u208b', '-').replace('\u2212', '-')\n text = text.replace('\ufe63', '\u30fc').replace('\uff0d', '\u30fc').replace('\u2014', '\u30fc').replace('\u2015', '\u30fc')\n text = text.replace('\u2501', '\u30fc').replace('\u2500', '\u30fc')\n return unicodedata.normalize(mode, text)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts Hiragana to hepburn - style alphabets.", "response": "def kana2alphabet(text):\n \"\"\"Convert Hiragana to hepburn-style alphabets\n\n Parameters\n ----------\n text : str\n Hiragana string.\n\n Return\n ------\n str\n Hepburn-style alphabets string.\n\n Examples\n --------\n >>> print(jaconv.kana2alphabet('\u307e\u307f\u3055\u3093'))\n mamisan\n \"\"\"\n text = text.replace('\u304d\u3083', 'kya').replace('\u304d\u3085', 'kyu').replace('\u304d\u3087', 'kyo')\n text = text.replace('\u304e\u3083', 'gya').replace('\u304e\u3085', 'gyu').replace('\u304e\u3087', 'gyo')\n text = text.replace('\u3057\u3083', 'sha').replace('\u3057\u3085', 'shu').replace('\u3057\u3087', 'sho')\n text = text.replace('\u3058\u3083', 'ja').replace('\u3058\u3085', 'ju').replace('\u3058\u3087', 'jo')\n text = text.replace('\u3061\u3083', 'cha').replace('\u3061\u3085', 'chu').replace('\u3061\u3087', 'cho')\n text = text.replace('\u306b\u3083', 'nya').replace('\u306b\u3085', 'nyu').replace('\u306b\u3087', 'nyo')\n text = text.replace('\u3075\u3041', 'fa').replace('\u3075\u3043', 'fi').replace('\u3075\u3047', 'fe')\n text = text.replace('\u3075\u3049', 'fo')\n text = text.replace('\u3072\u3083', 'hya').replace('\u3072\u3085', 'hyu').replace('\u3072\u3087', 'hyo')\n text = text.replace('\u307f\u3083', 'mya').replace('\u307f\u3085', 'myu').replace('\u307f\u3087', 'myo')\n text = text.replace('\u308a\u3083', 'rya').replace('\u308a\u3085', 'ryu').replace('\u308a\u3087', 'ryo')\n text = text.replace('\u3073\u3083', 'bya').replace('\u3073\u3085', 'byu').replace('\u3073\u3087', 'byo')\n text = text.replace('\u3074\u3083', 'pya').replace('\u3074\u3085', 'pyu').replace('\u3074\u3087', 'pyo')\n text = text.replace('\u304c', 'ga').replace('\u304e', 'gi').replace('\u3050', 'gu')\n text = text.replace('\u3052', 'ge').replace('\u3054', 'go').replace('\u3056', 'za')\n text = text.replace('\u3058', 'ji').replace('\u305a', 'zu').replace('\u305c', 'ze')\n text = text.replace('\u305e', 'zo').replace('\u3060', 'da').replace('\u3062', 'ji')\n text = text.replace('\u3065', 'zu').replace('\u3067', 'de').replace('\u3069', 'do')\n text = text.replace('\u3070', 'ba').replace('\u3073', 'bi').replace('\u3076', 'bu')\n text = text.replace('\u3079', 'be').replace('\u307c', 'bo').replace('\u3071', 'pa')\n text = text.replace('\u3074', 'pi').replace('\u3077', 'pu').replace('\u307a', 'pe')\n text = text.replace('\u307d', 'po')\n text = text.replace('\u304b', 'ka').replace('\u304d', 'ki').replace('\u304f', 'ku')\n text = text.replace('\u3051', 'ke').replace('\u3053', 'ko').replace('\u3055', 'sa')\n text = text.replace('\u3057', 'shi').replace('\u3059', 'su').replace('\u305b', 'se')\n text = text.replace('\u305d', 'so').replace('\u305f', 'ta').replace('\u3061', 'chi')\n text = text.replace('\u3064', 'tsu').replace('\u3066', 'te').replace('\u3068', 'to')\n text = text.replace('\u306a', 'na').replace('\u306b', 'ni').replace('\u306c', 'nu')\n text = text.replace('\u306d', 'ne').replace('\u306e', 'no').replace('\u306f', 'ha')\n text = text.replace('\u3072', 'hi').replace('\u3075', 'fu').replace('\u3078', 'he')\n text = text.replace('\u307b', 'ho').replace('\u307e', 'ma').replace('\u307f', 'mi')\n text = text.replace('\u3080', 'mu').replace('\u3081', 'me').replace('\u3082', 'mo')\n text = text.replace('\u3089', 'ra').replace('\u308a', 'ri').replace('\u308b', 'ru')\n text = text.replace('\u308c', 're').replace('\u308d', 'ro')\n text = text.replace('\u3084', 'ya').replace('\u3086', 'yu').replace('\u3088', 'yo')\n text = text.replace('\u308f', 'wa').replace('\u3090', 'wi').replace('\u3092', 'wo')\n text = text.replace('\u3091', 'we')\n text = _convert(text, KANA2HEP)\n while '\u3063' in text:\n text = list(text)\n tsu_pos = text.index('\u3063')\n if len(text) <= tsu_pos + 1:\n return ''.join(text[:-1]) + 'xtsu'\n if tsu_pos == 0:\n text[tsu_pos] = 'xtsu'\n else:\n text[tsu_pos] = text[tsu_pos + 1]\n text = ''.join(text)\n return text"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert alphabets to Kiragana.", "response": "def alphabet2kana(text):\n \"\"\"Convert alphabets to Hiragana\n\n Parameters\n ----------\n text : str\n Alphabets string.\n\n Return\n ------\n str\n Hiragana string.\n\n Examples\n --------\n >>> print(jaconv.alphabet2kana('mamisan'))\n \u307e\u307f\u3055\u3093\n \"\"\"\n text = text.replace('kya', '\u304d\u3083').replace('kyu', '\u304d\u3085').replace('kyo', '\u304d\u3087')\n text = text.replace('gya', '\u304e\u3083').replace('gyu', '\u304e\u3085').replace('gyo', '\u304e\u3087')\n text = text.replace('sha', '\u3057\u3083').replace('shu', '\u3057\u3085').replace('sho', '\u3057\u3087')\n text = text.replace('zya', '\u3058\u3083').replace('zyu', '\u3058\u3085').replace('zyo', '\u3058\u3087')\n text = text.replace('zyi', '\u3058\u3043').replace('zye', '\u3058\u3047')\n text = text.replace('ja', '\u3058\u3083').replace('ju', '\u3058\u3085').replace('jo', '\u3058\u3087')\n text = text.replace('jya', '\u3058\u3083').replace('jyu', '\u3058\u3085').replace('jyo', '\u3058\u3087')\n text = text.replace('cha', '\u3061\u3083').replace('chu', '\u3061\u3085').replace('cho', '\u3061\u3087')\n text = text.replace('tya', '\u3061\u3083').replace('tyu', '\u3061\u3085').replace('tyo', '\u3061\u3087')\n text = text.replace('nya', '\u306b\u3083').replace('nyu', '\u306b\u3085').replace('nyo', '\u306b\u3087')\n text = text.replace('hya', '\u3072\u3083').replace('hyu', '\u3072\u3085').replace('hyo', '\u3072\u3087')\n text = text.replace('mya', '\u307f\u3083').replace('myu', '\u307f\u3085').replace('myo', '\u307f\u3087')\n text = text.replace('rya', '\u308a\u3083').replace('ryu', '\u308a\u3085').replace('ryo', '\u308a\u3087')\n text = text.replace('bya', '\u3073\u3083').replace('byu', '\u3073\u3085').replace('byo', '\u3073\u3087')\n text = text.replace('pya', '\u3074\u3083').replace('pyu', '\u3074\u3085').replace('pyo', '\u3074\u3087')\n text = text.replace('oh', '\u304a\u304a')\n text = text.replace('ga', '\u304c').replace('gi', '\u304e').replace('gu', '\u3050')\n text = text.replace('ge', '\u3052').replace('go', '\u3054').replace('za', '\u3056')\n text = text.replace('ji', '\u3058').replace('zu', '\u305a').replace('ze', '\u305c')\n text = text.replace('zo', '\u305e').replace('da', '\u3060').replace('ji', '\u3062').replace('di', '\u3062')\n text = text.replace('va', '\u3094\u3041').replace('vi', '\u3094\u3043').replace('vu', '\u3094')\n text = text.replace('ve', '\u3094\u3047').replace('vo', '\u3094\u3049').replace('vya', '\u3094\u3083')\n text = text.replace('vyi', '\u3094\u3043').replace('vyu', '\u3094\u3085').replace('vye', '\u3094\u3047')\n text = text.replace('vyo', '\u3094\u3087')\n text = text.replace('zu', '\u3065').replace('de', '\u3067').replace('do', '\u3069')\n text = text.replace('ba', '\u3070').replace('bi', '\u3073').replace('bu', '\u3076')\n text = text.replace('be', '\u3079').replace('bo', '\u307c').replace('pa', '\u3071')\n text = text.replace('pi', '\u3074').replace('pu', '\u3077').replace('pe', '\u307a')\n text = text.replace('po', '\u307d').replace('dha', '\u3067\u3083').replace('dhi', '\u3067\u3043')\n text = text.replace('dhu', '\u3067\u3085').replace('dhe', '\u3067\u3047').replace('dho', '\u3067\u3087')\n text = text.replace('ka', '\u304b').replace('ki', '\u304d').replace('ku', '\u304f')\n text = text.replace('ke', '\u3051').replace('ko', '\u3053').replace('sa', '\u3055')\n text = text.replace('shi', '\u3057').replace('su', '\u3059').replace('se', '\u305b')\n text = text.replace('so', '\u305d').replace('ta', '\u305f').replace('chi', '\u3061')\n text = text.replace('tsu', '\u3064').replace('te', '\u3066').replace('to', '\u3068')\n text = text.replace('na', '\u306a').replace('ni', '\u306b').replace('nu', '\u306c')\n text = text.replace('ne', '\u306d').replace('no', '\u306e').replace('ha', '\u306f')\n text = text.replace('hi', '\u3072').replace('fu', '\u3075').replace('he', '\u3078')\n text = text.replace('ho', '\u307b').replace('ma', '\u307e').replace('mi', '\u307f')\n text = text.replace('mu', '\u3080').replace('me', '\u3081').replace('mo', '\u3082')\n text = text.replace('ra', '\u3089').replace('ri', '\u308a').replace('ru', '\u308b')\n text = text.replace('re', '\u308c').replace('ro', '\u308d')\n text = text.replace('ya', '\u3084').replace('yu', '\u3086').replace('yo', '\u3088')\n text = text.replace('wa', '\u308f').replace('wi', '\u3090').replace('we', '\u3091')\n text = text.replace('wo', '\u3092')\n text = text.replace('nn', '\u3093').replace('tu', '\u3064').replace('hu', '\u3075')\n text = text.replace('fa', '\u3075\u3041').replace('fi', '\u3075\u3043').replace('fe', '\u3075\u3047')\n text = text.replace('fo', '\u3075\u3049').replace('-', '\u30fc')\n text = _convert(text, HEP2KANA)\n ret = []\n for (i, char) in enumerate(text):\n if char in consonants:\n char = '\u3063'\n ret.append(char)\n return ''.join(ret)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef histpoints(x, bins=None, xerr=None, yerr='gamma', normed=False, **kwargs):\n import matplotlib.pyplot as plt\n\n if bins is None:\n bins = calc_nbins(x)\n\n h, bins = np.histogram(x, bins=bins)\n width = bins[1] - bins[0]\n center = (bins[:-1] + bins[1:]) / 2\n area = sum(h * width)\n\n if isinstance(yerr, str):\n yerr = poisson_limits(h, yerr)\n\n if xerr == 'binwidth':\n xerr = width / 2\n\n if normed:\n h = h / area\n yerr = yerr / area\n area = 1.\n\n if not 'color' in kwargs:\n kwargs['color'] = 'black'\n\n if not 'fmt' in kwargs:\n kwargs['fmt'] = 'o'\n\n plt.errorbar(center, h, xerr=xerr, yerr=yerr, **kwargs)\n\n return center, (yerr[0], h, yerr[1]), area", "response": "Plots a histogram of x using individual data points x and y."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef pool(self):\n self._pool = self._pool or eventlet.GreenPool(size=self.pool_size)\n return self._pool", "response": "Get an eventlet pool used to dispatch requests."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef start(self):\n if self.pid is not None:\n\n LOG.error(\n \"The process is already running with pid {0}.\".format(self.pid)\n )\n sys.exit(exit.ALREADY_RUNNING)\n\n self.daemonize()\n\n LOG.info(\"Beginning run loop for process.\")\n try:\n\n self.run()\n\n except Exception:\n\n LOG.exception(\"Uncaught exception in the daemon run() method.\")\n self.stop()\n sys.exit(exit.RUN_FAILURE)", "response": "Start the process with daemonization."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef stop(self):\n if self.pid is None:\n\n return None\n\n try:\n\n while True:\n\n self.send(signal.SIGTERM)\n time.sleep(0.1)\n\n except RuntimeError as err:\n\n if \"No such process\" in str(err):\n\n LOG.info(\"Succesfully stopped the process.\")\n return None\n\n LOG.exception(\"Failed to stop the process:\")\n sys.exit(exit.STOP_FAILED)\n\n except TypeError as err:\n\n if \"an integer is required\" in str(err):\n\n LOG.info(\"Succesfully stopped the process.\")\n return None\n\n LOG.exception(\"Failed to stop the process:\")\n sys.exit(exit.STOP_FAILED)", "response": "Stop the daemonized process."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef handle(self, signum, handler):\n if not isinstance(signum, int):\n\n raise TypeError(\n \"Signals must be given as integers. Got {0}.\".format(\n type(signum),\n ),\n )\n\n if not callable(handler):\n\n raise TypeError(\n \"Signal handlers must be callable.\",\n )\n\n signal.signal(signum, self._handle_signals)\n self._handlers[signum].append(handler)", "response": "Set a function to run when the given signal is recieved."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsends the given signal to the running process.", "response": "def send(self, signum):\n \"\"\"Send the given signal to the running process.\n\n If the process is not running a RuntimeError with a message of \"No such\n process\" should be emitted.\n \"\"\"\n if not isinstance(signum, int):\n\n raise TypeError(\n \"Signals must be given as integers. Got {0}.\".format(\n type(signum),\n ),\n )\n\n try:\n\n os.kill(self.pid, signum)\n\n except OSError as err:\n\n if \"No such process\" in err.strerror:\n\n raise RuntimeError(\"No such process {0}.\".format(self.pid))\n\n raise err"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nhandles all signals which trigger a process stop.", "response": "def shutdown(self, signum):\n \"\"\"Handle all signals which trigger a process stop.\n\n This method should run all appropriate signal handlers registered\n through the 'handle' method. At the end it should cause the process\n to exit with a status code. If any of the handlers raise an exception\n the exit code should be SHUTDOWN_FAILED otherwise SUCCESS.\n \"\"\"\n dirty = False\n for handler in self._handlers[signum]:\n\n try:\n\n handler()\n\n except:\n\n LOG.exception(\"A shutdown handler failed to execute:\")\n dirty = True\n\n del self.pid\n\n if dirty:\n\n sys.exit(exit.SHUTDOWN_FAILED)\n return None\n\n sys.exit(exit.SUCCESS)\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef preprocess_path(path):\n if path.startswith(\"gs://\"):\n fs = gcsfs.GCSFileSystem(token=get_gcloud_token())\n for sub_f in fs.ls(path):\n if sub_f.endswith(\"/\") and sub_f.split(\"/\")[-2] == FILES_FOLDER:\n return \"gs://\" + sub_f\n return path\n if path.startswith(\"hdfs://\"):\n return path\n for sub_f in os.listdir(path):\n sub_f_tot = os.path.join(path, sub_f)\n if os.path.isdir(sub_f_tot) and sub_f == FILES_FOLDER:\n if check_for_dataset(sub_f_tot):\n return sub_f_tot\n else:\n raise ValueError(\"Dataset in {} was not in GMQL format\".format(sub_f_tot))\n # if we are here it means that there is no files folder...so we need to check the root\n if check_for_dataset(path):\n return path\n else:\n raise ValueError(\"Dataset in {} was not in GMQL format\".format(path))", "response": "This function takes a path and checks if the file is in the GMQL format."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef check_for_dataset(files):\n all_files = os.listdir(files)\n meta_files = set(map(lambda y: y[: -9], filter(lambda x: x.endswith(\".gdm.meta\"), all_files)))\n regs_files = set(map(lambda y: y[: -4], filter(lambda x: x.endswith(\".gdm\"), all_files)))\n return meta_files == regs_files", "response": "Check if the path of the dataset contains a gdm dataset."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef load_from_path(local_path=None, parser=None, all_load=False):\n\n from .. import GDataframe\n from .. import GMQLDataset\n pmg = get_python_manager()\n\n local_path = preprocess_path(local_path)\n\n if all_load:\n # load directly the metadata for exploration\n meta = MetaLoaderFile.load_meta_from_path(local_path)\n if isinstance(parser, RegionParser):\n # region data\n regs = RegLoaderFile.load_reg_from_path(local_path, parser)\n else:\n regs = RegLoaderFile.load_reg_from_path(local_path)\n\n return GDataframe.GDataframe(regs=regs, meta=meta)\n else:\n from ...settings import is_metaprofiling_enabled\n if is_metaprofiling_enabled():\n meta_profile = create_metadata_profile(local_path)\n else:\n meta_profile = None\n\n if parser is None:\n # find the parser\n parser = RegLoaderFile.get_parser(local_path)\n elif not isinstance(parser, RegionParser):\n raise ValueError(\"parser must be RegionParser. {} was provided\".format(type(parser)))\n\n source_table = get_source_table()\n id = source_table.search_source(local=local_path)\n if id is None:\n id = source_table.add_source(local=local_path, parser=parser)\n local_sources = [id]\n\n index = pmg.read_dataset(str(id), parser.get_gmql_parser())\n return GMQLDataset.GMQLDataset(index=index, parser=parser,\n location=\"local\", path_or_name=local_path,\n local_sources=local_sources,\n meta_profile=meta_profile)", "response": "Loads the data from a local path into a GMQLDataset."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nloading the data from a remote repository.", "response": "def load_from_remote(remote_name, owner=None):\n \"\"\" Loads the data from a remote repository.\n\n :param remote_name: The name of the dataset in the remote repository\n :param owner: (optional) The owner of the dataset. If nothing is provided, the current user\n is used. For public datasets use 'public'.\n\n :return: A new GMQLDataset or a GDataframe\n \"\"\"\n from .. import GMQLDataset\n pmg = get_python_manager()\n remote_manager = get_remote_manager()\n parser = remote_manager.get_dataset_schema(remote_name, owner)\n\n source_table = get_source_table()\n id = source_table.search_source(remote=remote_name)\n if id is None:\n id = source_table.add_source(remote=remote_name, parser=parser)\n index = pmg.read_dataset(str(id), parser.get_gmql_parser())\n remote_sources = [id]\n return GMQLDataset.GMQLDataset(index=index, location=\"remote\", path_or_name=remote_name,\n remote_sources=remote_sources)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nperform a scan for access points in the area.", "response": "def scan(interface):\n \"\"\"Perform a scan for access points in the area.\n\n Arguments:\n interface - device to use for scanning (e.g. eth1, wlan0).\n \"\"\"\n interface = _get_bytes(interface)\n\n head = ffi.new('wireless_scan_head *')\n\n with iwlib_socket() as sock:\n range = _get_range_info(interface, sock=sock)\n\n if iwlib.iw_scan(sock, interface, range.we_version_compiled, head) != 0:\n errno = ffi.errno\n strerror = \"Error while scanning: %s\" % os.strerror(errno)\n raise OSError(errno, strerror)\n\n results = []\n\n scan = head.result\n\n buf = ffi.new('char []', 1024)\n\n while scan != ffi.NULL:\n parsed_scan = {}\n\n if scan.b.has_mode:\n parsed_scan['Mode'] = ffi.string(iwlib.iw_operation_mode[scan.b.mode])\n\n if scan.b.essid_on:\n parsed_scan['ESSID'] = ffi.string(scan.b.essid)\n else:\n parsed_scan['ESSID'] = b'Auto'\n\n if scan.has_ap_addr:\n iwlib.iw_ether_ntop(\n ffi.cast('struct ether_addr *', scan.ap_addr.sa_data), buf)\n if scan.b.has_mode and scan.b.mode == iwlib.IW_MODE_ADHOC:\n parsed_scan['Cell'] = ffi.string(buf)\n else:\n parsed_scan['Access Point'] = ffi.string(buf)\n\n if scan.has_maxbitrate:\n iwlib.iw_print_bitrate(buf, len(buf), scan.maxbitrate.value)\n parsed_scan['BitRate'] = ffi.string(buf)\n\n if scan.has_stats:\n parsed_scan['stats'] = _parse_stats(scan.stats)\n\n results.append(parsed_scan)\n scan = scan.next\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef login(self, username=None, password=None):\n if (username is None) and (password is None):\n auth_token = self.__login_guest()\n elif (username is not None) and (password is not None):\n auth_token, fullName = self.__login_credentials(username, password)\n self.logger.info(\"You are logged as {}\".format(fullName))\n else:\n raise ValueError(\"you have to specify both username and password or nothing\")\n\n if auth_token is not None:\n self.auth_token = auth_token\n else:\n raise ConnectionError(\"Impossible to retrieve the authentication token\")", "response": "Login to the GMQL serivice."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef logout(self):\n url = self.address + \"/logout\"\n header = self.__check_authentication()\n response = requests.get(url, headers=header)\n if response.status_code != 200:\n raise ValueError(\"Code {}. {}\".format(response.status_code, response.json().get(\"error\")))", "response": "Logout from the remote account\n "} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the list of available datasets for the current user.", "response": "def get_dataset_list(self):\n \"\"\" Returns the list of available datasets for the current user.\n\n :return: a pandas Dataframe\n \"\"\"\n url = self.address + \"/datasets\"\n header = self.__check_authentication()\n response = requests.get(url, headers=header)\n response = response.json()\n datasets = response.get(\"datasets\")\n res = pd.DataFrame.from_dict(datasets)\n return self.process_info_list(res, \"info\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting the list of samples of a specific remote dataset.", "response": "def get_dataset_samples(self, dataset_name, owner=None):\n \"\"\" Get the list of samples of a specific remote dataset.\n\n :param dataset_name: the dataset name\n :param owner: (optional) who owns the dataset. If it is not specified, the current user\n is used. For public dataset use 'public'.\n :return: a pandas Dataframe\n \"\"\"\n if isinstance(owner, str):\n owner = owner.lower()\n dataset_name = owner + \".\" + dataset_name\n\n header = self.__check_authentication()\n\n url = self.address + \"/datasets/\" + dataset_name\n response = requests.get(url, headers=header)\n if response.status_code != 200:\n raise ValueError(\"Code {}: {}\".format(response.status_code, response.json().get(\"error\")))\n response = response.json()\n samples = response.get(\"samples\")\n if len(samples) == 0:\n return None\n res = pd.DataFrame.from_dict(samples)\n return self.process_info_list(res, \"info\")"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngiving a dataset name, it returns a BedParser coherent with the schema of it :param dataset_name: a dataset name on the repository :param owner: (optional) who owns the dataset. If it is not specified, the current user is used. For public dataset use 'public'. :return: a BedParser", "response": "def get_dataset_schema(self, dataset_name, owner=None):\n \"\"\" Given a dataset name, it returns a BedParser coherent with the schema of it\n\n :param dataset_name: a dataset name on the repository\n :param owner: (optional) who owns the dataset. If it is not specified, the current user\n is used. For public dataset use 'public'.\n :return: a BedParser\n \"\"\"\n\n if isinstance(owner, str):\n owner = owner.lower()\n dataset_name = owner + \".\" + dataset_name\n\n url = self.address + \"/datasets/\" + dataset_name+\"/schema\"\n header = self.__check_authentication()\n response = requests.get(url, headers=header)\n if response.status_code != 200:\n raise ValueError(\"Code {}: {}\".format(response.status_code, response.json().get(\"error\")))\n\n response = response.json()\n name = response.get(\"name\")\n schemaType = response.get(\"type\")\n coordinates_system = response.get(\"coordinate_system\")\n fields = response.get(\"fields\")\n\n i = 0\n chrPos, startPos, stopPos, strandPos = None, None, None, None\n otherPos = []\n if schemaType == GTF:\n chrPos = 0 # seqname\n startPos = 3 # start\n stopPos = 4 # end\n strandPos = 6 # strand\n otherPos = [(1, 'source', 'string'), (2, 'feature', 'string'),\n (5, 'score', 'float'), (7, 'frame', 'string')]\n\n for field in fields:\n fieldName = field.get(\"name\")\n fieldType = field.get(\"type\").lower()\n if fieldName.lower() not in {'seqname', 'start', 'end', 'strand',\n 'source', 'feature', 'score', 'frame'}:\n otherPos.append((i, fieldName, fieldType))\n i += 1\n\n else:\n for field in fields:\n fieldName = field.get(\"name\")\n fieldType = field.get(\"type\").lower()\n\n if fieldName.lower() in chr_aliases and chrPos is None:\n chrPos = i\n elif fieldName.lower() in start_aliases and startPos is None:\n startPos = i\n elif fieldName.lower() in stop_aliases and stopPos is None:\n stopPos = i\n elif fieldName.lower() in strand_aliases and strandPos is None:\n strandPos = i\n else: # other positions\n otherPos.append((i, fieldName, fieldType))\n i += 1\n if len(otherPos) == 0:\n otherPos = None\n\n return RegionParser(chrPos=chrPos,\n startPos=startPos,\n stopPos=stopPos,\n strandPos=strandPos,\n otherPos=otherPos,\n schema_format=schemaType,\n coordinate_system=coordinates_system,\n delimiter=\"\\t\", parser_name=name)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef upload_dataset(self, dataset, dataset_name, schema_path=None):\n\n url = self.address + \"/datasets/\" + dataset_name + \"/uploadSample\"\n header = self.__check_authentication()\n\n fields = dict()\n remove = False\n if isinstance(dataset, GDataframe):\n tmp_path = TempFileManager.get_new_dataset_tmp_folder()\n dataset.to_dataset_files(local_path=tmp_path)\n dataset = tmp_path\n remove = True\n\n # a path is provided\n if not isinstance(dataset, str):\n raise TypeError(\"Dataset can be a path or a GDataframe. {} was passed\".format(type(dataset)))\n\n file_paths, schema_path_found = Loader.get_file_paths(dataset)\n if schema_path is None:\n schema_path = schema_path_found\n fields['schema'] = (os.path.basename(schema_path), open(schema_path, \"rb\"), 'application/octet-stream')\n for i, file in enumerate(file_paths):\n fields[\"file\"+str(i + 1)] = (os.path.basename(file), open(file, \"rb\"), 'application/octet-stream')\n\n encoder = MultipartEncoder(fields)\n callback = create_callback(encoder, len(fields))\n\n m_encoder = MultipartEncoderMonitor(encoder, callback)\n header['Content-Type'] = m_encoder.content_type\n\n self.logger.debug(\"Uploading dataset at {} with name {}\".format(dataset, dataset_name))\n\n response = requests.post(url, data=m_encoder,\n headers=header)\n\n # closing files\n for fn in fields.keys():\n _, f, _ = fields[fn]\n f.close()\n\n if response.status_code != 200:\n raise ValueError(\"Code {}: {}\".format(response.status_code, response.content))\n\n if remove:\n TempFileManager.delete_tmp_dataset(dataset)", "response": "Upload a dataset to the repository"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef delete_dataset(self, dataset_name):\n url = self.address + \"/datasets/\" + dataset_name\n header = self.__check_authentication()\n response = requests.delete(url, headers=header)\n if response.status_code != 200:\n raise ValueError(\"Code {}: {}\".format(response.status_code, response.json().get(\"error\")))\n self.logger.debug(\"Dataset {} was deleted from the repository\".format(dataset_name))", "response": "Deletes the dataset with the specified name from the repository."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef download_dataset(self, dataset_name, local_path, how=\"stream\"):\n if not os.path.isdir(local_path):\n os.makedirs(local_path)\n else:\n raise ValueError(\"Path {} already exists!\".format(local_path))\n\n local_path = os.path.join(local_path, FILES_FOLDER)\n os.makedirs(local_path)\n\n if how == 'zip':\n return self.download_as_zip(dataset_name, local_path)\n elif how == 'stream':\n return self.download_as_stream(dataset_name, local_path)\n else:\n raise ValueError(\"how must be {'zip', 'stream'}\")", "response": "This function downloads the dataset from the repository and puts it in the specified local folder."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nexecute a GMQL textual query on the remote server and return a pandas dataframe with the result data.", "response": "def query(self, query, output_path=None, file_name=\"query\", output=\"tab\"):\n \"\"\" Execute a GMQL textual query on the remote server.\n\n :param query: the string containing the query\n :param output_path (optional): where to store the results locally. If specified\n the results are downloaded locally\n :param file_name (optional): the name of the query\n :param output (optional): how to save the results. It can be \"tab\" or \"gtf\"\n :return: a pandas dataframe with the dictionary ids of the results\n \"\"\"\n header = self.__check_authentication()\n header['Content-Type'] = \"text/plain\"\n output = output.lower()\n if output not in ['tab', 'gtf']:\n raise ValueError(\"output must be 'tab' or 'gtf'\")\n url = self.address + \"/queries/run/\" + file_name + '/' + output\n response = requests.post(url, data=query, headers=header)\n if response.status_code != 200:\n raise ValueError(\"Code {}. {}\".format(response.status_code, response.json().get(\"error\")))\n response = response.json()\n jobid = response.get(\"id\")\n self.logger.debug(\"JobId: {}. Waiting for the result\".format(jobid))\n\n status_resp = self._wait_for_result(jobid)\n\n datasets = status_resp.get(\"datasets\")\n return self.__process_result_datasets(datasets, output_path)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget information about the specified remote job.", "response": "def trace_job(self, jobId):\n \"\"\" Get information about the specified remote job\n\n :param jobId: the job identifier\n :return: a dictionary with the information\n \"\"\"\n header = self.__check_authentication()\n status_url = self.address + \"/jobs/\" + jobId + \"/trace\"\n status_resp = requests.get(status_url, headers=header)\n if status_resp.status_code != 200:\n raise ValueError(\"Code {}. {}\".format(status_resp.status_code, status_resp.json().get(\"error\")))\n return status_resp.json()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_mode(how):\n global __mode\n if how == \"local\":\n __mode = how\n elif how == \"remote\":\n __mode = how\n else:\n raise ValueError(\"how must be 'local' or 'remote'\")", "response": "Sets the behavior of the API\n ."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nenable or disables the progress bars for the loading writing and downloading of datasets .", "response": "def set_progress(how):\n \"\"\" Enables or disables the progress bars for the loading, writing and downloading\n of datasets\n\n :param how: True if you want the progress bar, False otherwise\n :return: None\n\n Example::\n\n import gmql as gl\n\n gl.set_progress(True) # abilitates progress bars\n # ....do something...\n gl.set_progress(False) # removes progress bars\n # ....do something...\n \"\"\"\n global __progress_bar\n if isinstance(how, bool):\n __progress_bar = how\n else:\n raise ValueError(\n \"how must be a boolean. {} was found\".format(type(how)))"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nenables or disables the profiling of metadata at the loading of a GMQLDataset .", "response": "def set_meta_profiling(how):\n \"\"\" Enables or disables the profiling of metadata at the loading of a GMQLDataset\n\n :param how: True if you want to analyze the metadata when a GMQLDataset is created\n by a load_from_*. False otherwise. (Default=True)\n :return: None\n \"\"\"\n global __metadata_profiling\n if isinstance(how, bool):\n __metadata_profiling = how\n else:\n raise TypeError(\"how must be boolean. {} was provided\".format(type(how)))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngive a file path it loads it into memory as a Pandas dataframe", "response": "def parse_regions(self, path):\n \"\"\" Given a file path, it loads it into memory as a Pandas dataframe\n\n :param path: file path\n :return: a Pandas Dataframe\n \"\"\"\n if self.schema_format.lower() == GTF.lower():\n res = self._parse_gtf_regions(path)\n else:\n res = self._parse_tab_regions(path)\n return res"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_attributes(self):\n attr = ['chr', 'start', 'stop']\n if self.strandPos is not None:\n attr.append('strand')\n if self.otherPos:\n for i, o in enumerate(self.otherPos):\n attr.append(o[1])\n\n return attr", "response": "Returns the unordered list of attributes"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the ordered list of attributes", "response": "def get_ordered_attributes(self):\n \"\"\" Returns the ordered list of attributes\n\n :return: list of strings\n \"\"\"\n attrs = self.get_attributes()\n attr_arr = np.array(attrs)\n poss = [self.chrPos, self.startPos, self.stopPos]\n if self.strandPos is not None:\n poss.append(self.strandPos)\n if self.otherPos:\n for o in self.otherPos:\n poss.append(o[0])\n\n idx_sort = np.array(poss).argsort()\n return attr_arr[idx_sort].tolist()"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the unordered list of data types that are used in the log file.", "response": "def get_types(self):\n \"\"\" Returns the unordered list of data types\n\n :return: list of data types\n \"\"\"\n types = [str, int, int]\n if self.strandPos is not None:\n types.append(str)\n if self.otherPos:\n for o in self.otherPos:\n types.append(o[2])\n\n return types"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a dictionary of the typenames and data types.", "response": "def get_name_type_dict(self):\n \"\"\" Returns a dictionary of the type\n {'column_name': data_type, ...}\n\n :return: dict\n \"\"\"\n attrs = self.get_attributes()\n types = self.get_types()\n d = dict()\n for i,a in enumerate(attrs):\n d[a] = types[i]\n\n return d"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the ordered list of data types in the order of the positiy.", "response": "def get_ordered_types(self):\n \"\"\" Returns the ordered list of data types\n\n :return: list of data types\n \"\"\"\n types = self.get_types()\n types_arr = np.array(types)\n poss = [self.chrPos, self.startPos, self.stopPos]\n if self.strandPos is not None:\n poss.append(self.strandPos)\n if self.otherPos:\n for o in self.otherPos:\n poss.append(o[0])\n idx_sort = np.array(poss).argsort()\n return types_arr[idx_sort].tolist()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a new object containing the x - means clustering. rst algorithm.", "response": "def xmeans(cls, initial_centers=None, kmax=20, tolerance=0.025, criterion=splitting_type.BAYESIAN_INFORMATION_CRITERION, ccore=False):\n \"\"\"\n Constructor of the x-means clustering.rst algorithm\n\n :param initial_centers: Initial coordinates of centers of clusters that are represented by list: [center1, center2, ...]\n Note: The dimensions of the initial centers should be same as of the dataset.\n :param kmax: Maximum number of clusters that can be allocated.\n :param tolerance: Stop condition for each iteration: if maximum value of change of centers of clusters is less than tolerance than algorithm will stop processing\n :param criterion: Type of splitting creation.\n :param ccore: Defines should be CCORE (C++ pyclustering library) used instead of Python code or not.\n :return: returns the clustering.rst object\n \"\"\"\n model = xmeans(None, initial_centers, kmax, tolerance, criterion, ccore)\n return cls(model)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef clarans(cls, number_clusters, num_local, max_neighbour):\n model = clarans(None, number_clusters, num_local, max_neighbour)\n return cls(model)", "response": "Return a new CLARANS object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef optics(cls, data, eps, minpts, ccore=False):\n data = cls.input_preprocess(data)\n model = optics(data, eps, minpts)\n return cls(model)", "response": "This method is used to create OPTICS clustering. rst objects for the given data."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking if the clustering. rst algorithm belongs to pyclustering", "response": "def is_pyclustering_instance(model):\n \"\"\"\n Checks if the clustering.rst algorithm belongs to pyclustering\n\n :param model: the clustering.rst algorithm model\n :return: the truth value (Boolean)\n \"\"\"\n return any(isinstance(model, i) for i in [xmeans, clarans, rock, optics])"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef fit(self, data=None):\n if self.is_pyclustering_instance(self.model):\n if isinstance(self.model, xmeans):\n data = self.input_preprocess(data)\n self.model._xmeans__pointer_data = data\n elif isinstance(self.model, clarans):\n data = self.input_preprocess(data)\n self.model._clarans__pointer_data = data\n\n self.model.process()\n else:\n self.model.fit(data)\n return self", "response": "Performs clustering.rst\n\n :param data: Data to be fit\n :return: the clustering.rst object"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _labels_from_pyclusters(self):\n clusters = self.model.get_clusters()\n labels = []\n for i in range(0, len(clusters)):\n for j in clusters[i]:\n labels.insert(int(j), i)\n return labels", "response": "Computes and returns the list of labels indicating the data points and the corresponding cluster ids."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nextracts the cluster at the given index from the input dataframe that contains the clusters .", "response": "def retrieve_cluster(self, df, cluster_no):\n \"\"\"\n Extracts the cluster at the given index from the input dataframe\n\n :param df: the dataframe that contains the clusters\n :param cluster_no: the cluster number\n :return: returns the extracted cluster\n \"\"\"\n if self.is_pyclustering_instance(self.model):\n clusters = self.model.get_clusters()\n mask = []\n for i in range(0, df.shape[0]):\n mask.append(i in clusters[cluster_no])\n else:\n mask = self.model.labels_ == cluster_no # a boolean mask\n return df[mask]"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nretrieves the labels of a clustering. rst object.", "response": "def get_labels(obj):\n \"\"\"\n Retrieve the labels of a clustering.rst object\n\n :param obj: the clustering.rst object\n :return: the resulting labels\n \"\"\"\n if Clustering.is_pyclustering_instance(obj.model):\n return obj._labels_from_pyclusters\n else:\n return obj.model.labels_"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncomputing and plots the silhouette score vs number of clusters graph to help selecting the number of clusters visually", "response": "def silhouette_n_clusters(data, k_min, k_max, distance='euclidean'):\n \"\"\"\n Computes and plot the silhouette score vs number of clusters graph to help selecting the number of clusters visually\n\n :param data: The data object\n :param k_min: lowerbound of the cluster range\n :param k_max: upperbound of the cluster range\n :param distance: the distance metric, 'euclidean' by default\n :return:\n \"\"\"\n k_range = range(k_min, k_max)\n\n k_means_var = [Clustering.kmeans(k).fit(data) for k in k_range]\n\n silhouette_scores = [obj.silhouette_score(data=data, metric=distance) for obj in k_means_var]\n\n fig = plt.figure()\n ax = fig.add_subplot(111)\n ax.plot(k_range, silhouette_scores, 'b*-')\n ax.set_ylim((-1, 1))\n plt.grid(True)\n plt.xlabel('n_clusters')\n plt.ylabel('The silhouette score')\n plt.title('Silhouette score vs. k')\n plt.show()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncalculate and plots the variance explained vs. k", "response": "def elbow_method(data, k_min, k_max, distance='euclidean'):\n \"\"\"\n Calculates and plots the plot of variance explained - number of clusters\n Implementation reference: https://github.com/sarguido/k-means-clustering.rst\n\n :param data: The dataset\n :param k_min: lowerbound of the cluster range\n :param k_max: upperbound of the cluster range\n :param distance: the distance metric, 'euclidean' by default\n :return:\n \"\"\"\n # Determine your k range\n k_range = range(k_min, k_max)\n\n # Fit the kmeans model for each n_clusters = k\n k_means_var = [Clustering.kmeans(k).fit(data) for k in k_range]\n\n # Pull out the cluster centers for each model\n centroids = [X.model.cluster_centers_ for X in k_means_var]\n\n # Calculate the Euclidean distance from\n # each point to each cluster center\n k_euclid = [cdist(data, cent, distance) for cent in centroids]\n dist = [np.min(ke, axis=1) for ke in k_euclid]\n\n # Total within-cluster sum of squares\n wcss = [sum(d ** 2) for d in dist]\n\n # The total sum of squares\n tss = sum(pdist(data) ** 2) / data.shape[0]\n\n # The between-cluster sum of squares\n bss = tss - wcss\n\n # elbow curve\n fig = plt.figure()\n ax = fig.add_subplot(111)\n ax.plot(k_range, bss / tss * 100, 'b*-')\n ax.set_ylim((0, 100))\n plt.grid(True)\n plt.xlabel('n_clusters')\n plt.ylabel('Percentage of variance explained')\n plt.title('Variance Explained vs. k')\n plt.show()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef adjusted_mutual_info(self, reference_clusters):\n return adjusted_mutual_info_score(self.get_labels(self), self.get_labels(reference_clusters))", "response": "Calculates the adjusted mutual information score w. r. t. the reference clusters"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef adjusted_rand_score(self, reference_clusters):\n return adjusted_rand_score(self.get_labels(self), self.get_labels(reference_clusters))", "response": "Calculates the adjusted rand score w. r. t. the reference clusters"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate the completeness score w. r. t. the reference clusters", "response": "def completeness_score(self, reference_clusters):\n \"\"\"\n Calculates the completeness score w.r.t. the reference clusters (explicit evaluation)\n\n :param reference_clusters: Clusters that are to be used as reference\n :return: the resulting completeness score\n \"\"\"\n return completeness_score(self.get_labels(self), self.get_labels(reference_clusters))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncalculates the Fowlkes - Mallows index w. r. t. the reference clusters.", "response": "def fowlkes_mallows(self, reference_clusters):\n \"\"\"\n Calculates the Fowlkes-Mallows index (FMI) w.r.t. the reference clusters (explicit evaluation)\n\n :param reference_clusters: Clusters that are to be used as reference\n :return: The resulting Fowlkes-Mallows score.\n \"\"\"\n return fowlkes_mallows_score(self.get_labels(self), self.get_labels(reference_clusters))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalculating the homogeneity score w. r. t. the reference clusters", "response": "def homogeneity_score(self, reference_clusters):\n \"\"\"\n Calculates the homogeneity score w.r.t. the reference clusters (explicit evaluation)\n\n :param reference_clusters: Clusters that are to be used as reference\n :return: The resulting homogeneity score.\n \"\"\"\n return homogeneity_score(self.get_labels(self), self.get_labels(reference_clusters))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate the MI score w. r. t. the reference clusters", "response": "def mutual_info_score(self, reference_clusters):\n \"\"\"\n Calculates the MI (mutual information) w.r.t. the reference clusters (explicit evaluation)\n\n :param reference_clusters: Clusters that are to be used as reference\n :return: The resulting MI score.\n \"\"\"\n return mutual_info_score(self.get_labels(self), self.get_labels(reference_clusters))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef normalized_mutual_info_score(self, reference_clusters):\n return normalized_mutual_info_score(self.get_labels(self), self.get_labels(reference_clusters))", "response": "Calculates the normalized mutual information score w. r. t. the reference clusters"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncompute the mean Silhouette Coefficient of all samples in the cluster", "response": "def silhouette_score(self, data, metric='euclidean', sample_size=None, random_state=None, **kwds):\n \"\"\"\n Computes the mean Silhouette Coefficient of all samples (implicit evaluation)\n\n :param data: The data that the clusters are generated from\n :param metric: the pairwise distance metric\n :param sample_size: the size of the sample to use computing the Silhouette Coefficient\n :param random_state: If an integer is given then it fixes its seed otherwise random.\n :param kwds: any further parameters that are passed to the distance function\n :return: the mean Silhouette Coefficient of all samples\n \"\"\"\n return silhouette_score(data, self.get_labels(self), metric, sample_size, random_state, **kwds)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef materialize(datasets):\n from .. import GMQLDataset\n if isinstance(datasets, dict):\n result = dict()\n for output_path in datasets.keys():\n dataset = datasets[output_path]\n if not isinstance(dataset, GMQLDataset.GMQLDataset):\n raise TypeError(\"The values of the dictionary must be GMQLDataset.\"\n \" {} was given\".format(type(dataset)))\n gframe = dataset.materialize(output_path)\n result[output_path] = gframe\n elif isinstance(datasets, list):\n result = []\n for dataset in datasets:\n if not isinstance(dataset, GMQLDataset.GMQLDataset):\n raise TypeError(\"The values of the list must be GMQLDataset.\"\n \" {} was given\".format(type(dataset)))\n gframe = dataset.materialize()\n result.append(gframe)\n else:\n raise TypeError(\"The input must be a dictionary of a list. \"\n \"{} was given\".format(type(datasets)))\n return result", "response": "Multiple materializations. Enables the user to specify a set of GMQLDataset to be materialized."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nretrieve the bicluster at the given row and column number from the input dataframe.", "response": "def retrieve_bicluster(self, df, row_no, column_no):\n \"\"\"\n Extracts the bicluster at the given row bicluster number and the column bicluster number from the input dataframe.\n\n :param df: the input dataframe whose values were biclustered\n :param row_no: the number of the row bicluster\n :param column_no: the number of the column bicluster\n :return: the extracted bicluster from the dataframe\n \"\"\"\n res = df[self.model.biclusters_[0][row_no]]\n bicluster = res[res.columns[self.model.biclusters_[1][column_no]]]\n return bicluster"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef bicluster_similarity(self, reference_model):\n similarity_score = consensus_score(self.model.biclusters_, reference_model.biclusters_)\n return similarity_score", "response": "Calculates the similarity between the current model of biclusters and the reference model of biclusters."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nloads the multi - referenced mapped data from the file system.", "response": "def load(self, path, genes_uuid, regs=['chr', 'left', 'right', 'strand'], meta=[], values=[], full_load=False):\n \"\"\"\n Loads the multi referenced mapped data from the file system\n \n :param path: The path to the files\n :param genes_uuid: The unique identifier metadata column name to separate the data by the number of references\n :param regs: The region data that are to be analyzed\n :param meta: The metadata that are to be analyzed\n :param values: The values to fill the matrix\n :param full_load: Specifies the method of parsing the data. If False then parser omits the parsing of zero(0) \n values in order to speed up and save memory. However, while creating the matrix, those zero values are going to be put into the matrix.\n (unless a row contains \"all zero columns\". This parsing is strongly recommended for sparse datasets.\n If the full_load parameter is True then all the zero(0) data are going to be read.\n\n \"\"\"\n\n if not full_load:\n warnings.warn(\"\\n\\n You are using the optimized loading technique. \"\n \"All-zero rows are not going to be loaded into memory. \"\n \"To load all the data please set the full_load parameter equal to True.\")\n p = Parser(path)\n all_meta_data = p.parse_meta(meta)\n all_data = p.parse_data(regs, values, full_load)\n\n all_data = pd.pivot_table(all_data,\n values=values, columns=regs, index=['sample'],\n fill_value=0)\n\n group1 = all_meta_data.groupby([genes_uuid]).count()\n for g in group1.index.values:\n series = all_meta_data[genes_uuid] == g\n m = (all_meta_data[series])\n d = (all_data.loc[series]).dropna(axis=1, how='all') # not to show the NaN data\n self.data_model.append(GenometricSpace.from_memory(d, m))\n self.all_meta_data = all_meta_data"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef compact_view(self, merged_data, selected_meta, reference_no):\n meta_names = list(selected_meta)\n meta_index = []\n\n for x in meta_names:\n meta_index.append(self.all_meta_data.ix[merged_data.index.get_level_values(reference_no)][x].values)\n meta_index = np.asarray(meta_index)\n multi_meta_index = pd.MultiIndex.from_arrays(meta_index, names=meta_names)\n merged_data.index = multi_meta_index\n return merged_data", "response": "Creates and returns the compact view where the index of the dataframe is a multi index of the selected metadata."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef prune_by_missing_percent(df, percentage=0.4):\n mask = (df.isnull().sum() / df.shape[0]).map(lambda x: True if x < percentage else False)\n pruned_df = df[df.columns[mask.values]]\n return pruned_df", "response": "This method is used to remove the attributes that are more than a percentage of missing values."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nimputes the missing values by the selected statistical property of each column", "response": "def impute_using_statistics(df, method='min'):\n \"\"\"\n Imputes the missing values by the selected statistical property of each column\n\n :param df: The input dataframe that contains missing values\n :param method: The imputation method (min by default)\n \"zero\": fill missing entries with zeros\n \"mean\": fill with column means\n \"median\" : fill with column medians\n \"min\": fill with min value per column\n \"random\": fill with gaussian noise according to mean/std of column\n :return: the imputed dataframe\n \"\"\"\n sf = SimpleFill(method)\n imputed_matrix = sf.complete(df.values)\n imputed_df = pd.DataFrame(imputed_matrix, df.index, df.columns)\n return imputed_df"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef impute_knn(df, k=3):\n imputed_matrix = KNN(k=k).complete(df.values)\n imputed_df = pd.DataFrame(imputed_matrix, df.index, df.columns)\n return imputed_df", "response": "This function imputes the k nearest neighbour imputations which weights samples using the mean squared difference on features for which two rows both have observed data."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nimputing the missing values by using SVD decomposition", "response": "def impute_svd(df, rank=10, convergence_threshold=0.00001, max_iters=200):\n \"\"\"\n Imputes the missing values by using SVD decomposition\n Based on the following publication: 'Missing value estimation methods for DNA microarrays' by Troyanskaya et. al.\n\n :param df:The input dataframe that contains missing values\n :param rank: Rank value of the truncated SVD\n :param convergence_threshold: The threshold to stop the iterations\n :param max_iters: Max number of iterations\n :return: the imputed dataframe\n \"\"\"\n imputed_matrix = IterativeSVD(rank,convergence_threshold, max_iters).complete(df.values)\n imputed_df = pd.DataFrame(imputed_matrix, df.index, df.columns)\n return imputed_df"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nselecting the desired number of features in the imput dataframe.", "response": "def feature_selection(df, labels, n_features, method='chi2'):\n \"\"\"\n Reduces the number of features in the imput dataframe.\n Ex: labels = gs.meta['biospecimen_sample__sample_type_id'].apply(int).apply(lambda x: 0 if x < 10 else 1)\n chi2_fs(gs.data, labels, 50)\n\n :param df: The input dataframe\n :param labels: Labels for each row in the df. Type: Pandas.Series\n :param no_features: The desired number of features\n :param method: The feature selection method to be employed. It is set to 'chi2' by default\n To select the features using mutual information, the method value should be set to 'mi'\n To select the features using ANOVA, the method value should be set to 'ANOVA'\n :return: Returns the dataframe with the selected features\n\n \"\"\"\n fs_obj = None\n if method == 'chi2':\n fs_obj = chi2\n elif method == 'ANOVA':\n fs_obj = f_classif\n elif method == 'mi':\n fs_obj = mutual_info_classif\n else:\n raise ValueError('The method is not recognized')\n \n fs = SelectKBest(fs_obj, k=n_features)\n fs.fit_transform(df, labels)\n df_reduced = df.loc[:, fs.get_support()]\n return df_reduced"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsetting a spark property as a key value pair of using a dictionary d.", "response": "def set_spark_conf(self, key=None, value=None, d=None):\n \"\"\" Sets a spark property as a ('key', 'value') pair of using a dictionary\n {'key': 'value', ...}\n\n :param key: string\n :param value: string\n :param d: dictionary\n :return: None\n \"\"\"\n if isinstance(d, dict):\n self._properties.update(d)\n elif isinstance(key, str) and isinstance(value, str):\n self._properties[key] = value\n else:\n raise TypeError(\"key, value must be strings\")"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset a java system property as a key value pair of using a dictionary d.", "response": "def set_system_conf(self, key=None, value=None, d=None):\n \"\"\" Sets a java system property as a ('key', 'value') pair of using a dictionary\n {'key': 'value', ...}\n\n :param key: string\n :param value: string\n :param d: dictionary\n :return: None\n \"\"\"\n if isinstance(d, dict):\n self._system.update(d)\n elif isinstance(key, str) and isinstance(value, str):\n self._system[key] = value\n else:\n raise TypeError(\"key, value must be strings\")"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef isin(self, values):\n if not isinstance(values, list):\n raise TypeError(\"Input should be a string. {} was provided\".format(type(values)))\n if not (self.name.startswith(\"(\") and self.name.endswith(\")\")):\n first = True\n new_condition = None\n for v in values:\n if first:\n first = False\n new_condition = self.__eq__(v)\n else:\n new_condition = new_condition.__or__(self.__eq__(v))\n return new_condition\n else:\n raise SyntaxError(\"You cannot use 'isin' with a complex condition\")", "response": "Select the samples having the metadata attribute between the provided values as input\n ."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the flatpages available for the current site and all of the users in the current template context.", "response": "def get_flatpages_i18n(parser, token):\n \"\"\"\n Retrieves all flatpage objects available for the current site and\n visible to the specific user (or visible to all users if no user is\n specified). Populates the template context with them in a variable\n whose name is defined by the ``as`` clause.\n\n An optional ``for`` clause can be used to control the user whose\n permissions are to be used in determining which flatpages are visible.\n\n An optional argument, ``starts_with``, can be applied to limit the\n returned flatpages to those beginning with a particular base URL.\n This argument can be passed as a variable or a string, as it resolves\n from the template context.\n\n Syntax::\n\n {% get_flatpages_i18n ['url_starts_with'] [for user] as context_name %}\n\n Example usage::\n\n {% get_flatpages_i18n as flatpages %}\n {% get_flatpages_i18n for someuser as flatpages %}\n {% get_flatpages_i18n '/about/' as about_pages %}\n {% get_flatpages_i18n prefix as about_pages %}\n {% get_flatpages_i18n '/about/' for someuser as about_pages %}\n {% get_flatpages_i18n containing '/en/' as my_pages %}\n {% get_flatpages_i18n excluding '/en/' as my_pages %}\n \"\"\"\n bits = token.split_contents()\n syntax_message = (\"%(tag_name)s expects a syntax of %(tag_name)s \"\n \"['url_starts_with'] [for user] as context_name\" %\n dict(tag_name=bits[0]))\n\n # Must have at 3-6 bits in the tag\n if 3 <= len(bits) <= 8:\n containing = None\n excluding = None\n\n # If there's an even number of bits, there's no prefix\n if len(bits) % 2 == 0:\n prefix = bits[1]\n else:\n prefix = None\n if bits[1] == \"containing\":\n containing = bits[2]\n else:\n if bits[1] == \"excluding\":\n excluding = bits[2]\n\n # The very last bit must be the context name\n if bits[-2] != 'as':\n raise template.TemplateSyntaxError(syntax_message)\n context_name = bits[-1]\n\n # If there are 5 or 6 bits, there is a user defined\n if len(bits) >= 5:\n if bits[-4] != 'for':\n raise template.TemplateSyntaxError(syntax_message)\n user = bits[-3]\n else:\n user = None\n\n return FlatpageNode(\n context_name, starts_with=prefix, contains=containing,\n excludes=excluding, user=user)\n else:\n raise template.TemplateSyntaxError(syntax_message)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef daemonize(self):\n self._double_fork()\n\n # Write pidfile.\n self.pid = os.getpid()\n\n LOG.info(\n \"Succesfully daemonized process {0}.\".format(self.pid)\n )", "response": "Double fork and set the pid."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _double_fork(self):\n try:\n\n pid = os.fork()\n if pid > 0:\n\n # Exit first parent.\n sys.exit(0)\n return None\n\n except OSError as err:\n\n LOG.exception(\n \"Fork #1 failed: {0} ({1})\".format(\n err.errno,\n err.strerror,\n ),\n )\n sys.exit(exit.DAEMONIZE_FAILED)\n return None\n\n # Decouple from parent environment.\n os.chdir(\"/\")\n os.setsid()\n os.umask(0)\n\n # Do second fork.\n try:\n\n pid = os.fork()\n if pid > 0:\n\n # Exit from second parent.\n sys.exit(0)\n\n except OSError as err:\n\n LOG.exception(\n \"Fork #2 failed: {0} ({1})\".format(\n err.errno,\n err.strerror,\n ),\n )\n sys.exit(exit.DAEMONIZE_FAILED)\n return None", "response": "Do the UNIX double - fork magic."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\noverloading constructor to create the GenometricSpace object from memory data and meta variables.", "response": "def from_memory(cls, data, meta):\n \"\"\"\n Overloaded constructor to create the GenometricSpace object from memory data and meta variables.\n The indexes of the data and meta dataframes should be the same.\n\n\n :param data: The data model\n :param meta: The metadata\n :return: A GenometricSpace object\n \"\"\"\n\n obj = cls()\n obj.data = data\n obj.meta = meta\n return obj"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nparse and loads the data into the instance attributes.", "response": "def load(self, _path, regs=['chr', 'left', 'right', 'strand'], meta=[], values=[], full_load=False, file_extension=\"gdm\"):\n \"\"\"Parses and loads the data into instance attributes.\n The indexes of the data and meta dataframes should be the same.\n\n :param path: The path to the dataset on the filesystem\n :param regs: the regions that are to be analyzed\n :param meta: the meta-data that are to be analyzed\n :param values: the values that are to be selected\n :param full_load: Specifies the method of parsing the data. If False then parser omits the parsing of zero(0)\n values in order to speed up and save memory. However, while creating the matrix, those zero values are going to be put into the matrix.\n (unless a row contains \"all zero columns\". This parsing is strongly recommended for sparse datasets.\n If the full_load parameter is True then all the zero(0) data are going to be read.\n\n \"\"\"\n if not full_load:\n warnings.warn(\"\\n\\nYou are using the optimized loading technique. \"\n \"All-zero rows are not going to be loaded into memory. \"\n \"To load all the data please set the full_load parameter equal to True.\")\n p = Parser(_path)\n self.meta = p.parse_meta(meta)\n self.data = p.parse_data(regs, values, full_load=full_load, extension=file_extension)\n self._path = _path"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set_meta(self, selected_meta):\n meta_names = list(selected_meta)\n meta_names.append('sample')\n meta_index = []\n # To set the index for existing samples in the region dataframe.\n # The index size of the region dataframe does not necessarily be equal to that of metadata df.\n warnings.warn(\"\\n\\nThis method assumes that the last level of the index is the sample_id.\\n\"\n \"In case of single index, the index itself should be the sample_id\")\n for x in meta_names:\n meta_index.append(self.meta.ix[self.data.index.get_level_values(-1)][x].values)\n meta_index = np.asarray(meta_index)\n multi_meta_index = pd.MultiIndex.from_arrays(meta_index, names=meta_names)\n self.data.index = multi_meta_index", "response": "Sets one axis of the 2D multi - indexed dataframe\n index to the selected meta data."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate a 2D multi-indexed matrix representation of the data. This representation allows the data to be sent to the machine learning algorithms. Args: :param values: The value or values that are going to fill the matrix. :param selected_regions: The index to one axis of the matrix. :param default_value: The default fill value of the matrix", "response": "def to_matrix(self, values, selected_regions, default_value=0):\n \"\"\"Creates a 2D multi-indexed matrix representation of the data.\n This representation allows the data to be sent to the machine learning algorithms.\n\n Args:\n :param values: The value or values that are going to fill the matrix.\n :param selected_regions: The index to one axis of the matrix.\n :param default_value: The default fill value of the matrix\n\n \"\"\"\n if isinstance(values, list):\n for v in values:\n try:\n self.data[v] = self.data[v].map(float)\n except:\n print(self.data[v])\n else:\n self.data[values] = self.data[values].map(float)\n print(\"started pivoting\")\n self.data = pd.pivot_table(self.data,\n values=values, columns=selected_regions, index=['sample'],\n fill_value=default_value)\n print(\"end of pivoting\")"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_values(self, set, selected_meta):\n warnings.warn(\"\\n\\nThis method assumes that the last level of the index is the sample_id.\\n\"\n \"In case of single index, the index itself should be the sample_id\")\n sample_ids = set.index.get_level_values(-1)\n corresponding_meta = self.meta.loc[sample_ids]\n values = corresponding_meta[selected_meta]\n\n try:\n values = values.astype(float)\n except ValueError:\n print(\"the values should be numeric\")\n return values", "response": "Returns the values of the selected metadata values of the given set"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef group_statistics(self, group, selected_meta, stat_code='mean'):\n values = self.get_values(group, selected_meta)\n if stat_code == 'mean':\n res = statistics.mean(values)\n elif stat_code == 'variance':\n res = statistics.variance(values)\n elif stat_code == 'std':\n res = statistics.stdev(values)\n return res", "response": "Provides statistics of a group based on the meta data selected."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef to_bag_of_genomes(self, clustering_object):\n\n meta_files = Parser._get_files('meta', self._path)\n meta_dict = {}\n for f in meta_files:\n meta_dict[Parser.get_sample_id(f)] = f\n\n clusters = []\n if isinstance(clustering_object, Clustering):\n if Clustering.is_pyclustering_instance(clustering_object.model):\n no_clusters = len(clustering_object.model.get_clusters())\n else:\n no_clusters = clustering_object.model.n_clusters\n for c in range(0, no_clusters):\n clusters.append(clustering_object.retrieve_cluster(self.data, c).index.get_level_values(-1).values)\n\n elif isinstance(clustering_object, Biclustering):\n no_clusters = clustering_object.model.n_clusters[0] # 0 for the rows\n no_col_clusters = clustering_object.model.n_clusters[1] # 1 for the columns\n for c in range(0, no_clusters):\n clusters.append(clustering_object.retrieve_bicluster(self.data, c*no_col_clusters, 0).index.get_level_values(-1).values) # sample names\n\n # to create the bag of genomes files\n print(\"creating the bag_of_genomes...\")\n for c in tqdm(range(0, no_clusters)):\n document = open('./bag_of_genomes/document' + str(c) + '.bag_of_genome', 'w')\n for sample in clusters[c]:\n f = open(meta_dict[sample], 'r')\n for line in f:\n line = line.replace(' ', '_')\n splitted = line.split('\\t')\n document.write(splitted[0] + '=' + splitted[1])\n f.close()\n document.close()", "response": "Creates a bag of genomes representation for data mining purposes"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a term - document matrix that describes the frequency of terms that occur in a collection of genomes.", "response": "def to_term_document_matrix(path_to_bag_of_genomes, max_df=0.99, min_df=1, use_idf=False):\n \"\"\"\n Creates a term-document matrix which is a mathematical matrix that describes the frequency of terms\n that occur in a collection of documents (in our case a collection of genomes).\n\n :param path_to_bag_of_genomes: Path to the documents (genomes)\n :param max_df: To prune the terms that are existing in the given portion of documents (if set to 1 then it does not prune)\n :return: returns the term-document dataframe\n \"\"\"\n token_dict = {}\n\n def BoG_tokenizer(_text):\n return _text.split('\\n')\n\n print(\"creating the term-document matrix...\")\n for file in tqdm(Parser._get_files('.bag_of_genome', path_to_bag_of_genomes)):\n f = open(file, 'r')\n text = f.read()\n token_dict[file] = text\n\n tfidf = TfidfVectorizer(tokenizer=BoG_tokenizer, use_idf=use_idf, smooth_idf=False,\n max_df=max_df, min_df=min_df) # max df is less than 1.0 to ignore the tokens existing in all of the documents\n\n tfs = tfidf.fit_transform(token_dict.values())\n data = tfs.toarray()\n columns = tfidf.get_feature_names()\n df = pd.DataFrame(data, columns=columns)\n term_document_df = df.T\n return term_document_df"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef tf(cluster):\n counts = dict()\n words = cluster.split(' ')\n for word in words:\n counts[word] = counts.get(word, 0) + 1\n return counts", "response": "Computes the term frequency and stores it as a dictionary"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncomputes the dictionary of metadata for each cluster.", "response": "def best_descriptive_meta_dict(path_to_bag_of_genomes, cluster_no):\n \"\"\"\n Computes the importance of each metadata by using tf * coverage (the percentage of the term occuring in a cluster)\n\n :param path_to_bag_of_genomes: The directory path\n :param cluster_no: cluster number\n :param preprocess: to remove the redundant information from the metadata\n :return: returns the computed dictionary\n \"\"\"\n from nltk.corpus import stopwords\n clusters = []\n for file in Parser._get_files('.bag_of_genome', path_to_bag_of_genomes):\n f = open(file, 'r')\n text = f.read()\n # process the text\n word_list = []\n for line in text.split('\\n'):\n try:\n # take the right hand side of the = character\n rhs = line.split('=')[1]\n # omit the numbers\n\n if not (len(rhs) < 3 or rhs[0].isdigit() or any([x in rhs for x in ['.','//','tcga']]) or GenometricSpace.validate_uuid(rhs)):\n word_list.append(rhs)\n except Exception as e:\n # logging.exception(e)\n pass\n\n english_stopwords = stopwords.words('english')\n genomic_stopwords = ['tcga']\n extra_stopwords = ['yes','no', 'normal', 'low', 'high']\n all_stopwords = english_stopwords + genomic_stopwords + extra_stopwords\n filtered_words = [word for word in word_list if word not in all_stopwords]\n\n new_text = \"\"\n for word in filtered_words:\n new_text += word\n new_text += ' '\n clusters.append(new_text)\n\n all_clusters = \"\"\n for c in clusters:\n all_clusters += c\n\n all_clusters_tf = GenometricSpace.tf(all_clusters)\n\n cluster_dict = GenometricSpace.tf(clusters[cluster_no])\n for key, value in cluster_dict.items():\n new_val = cluster_dict[key] * (cluster_dict[key] / all_clusters_tf[key])\n cluster_dict[key] = new_val\n return cluster_dict"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nrendering the cloud of words representation for a given dictionary of frequencies.", "response": "def visualize_cloud_of_words(dictionary, image_path=None):\n \"\"\"\n Renders the cloud of words representation for a given dictionary of frequencies\n\n :param dictionary: the dictionary object that contains key-frequency pairs\n :param image_path: the path to the image mask, None if no masking is needed\n\n \"\"\"\n from PIL import Image\n\n if image_path is not None:\n mask = np.array(Image.open(image_path))\n wc = WordCloud(mask=mask, background_color='white', width=1600, height=1200, prefer_horizontal=0.8)\n wc = wc.generate_from_frequencies(dictionary)\n else:\n # Generate a word cloud image\n wc = WordCloud(background_color='white', width=1600, height=1200, prefer_horizontal=0.8)\n wc = wc.generate_from_frequencies(dictionary)\n\n # Display the generated image:\n # the matplotlib way:\n import matplotlib.pyplot as plt\n plt.rcParams['figure.figsize'] = (15, 15)\n plt.imshow(wc, interpolation='bilinear')\n plt.axis(\"off\")\n plt.show()"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndrawing the cloud of words representation", "response": "def cloud_of_words(path_to_bog, cluster_no, image_path=None):\n \"\"\"\n Draws the cloud of words representation\n\n :param path_to_bog: path to bag of words\n :param cluster_no: the number of document to be visualized\n :param image_path: path to the image file for the masking, None if no masking is needed\n\n \"\"\"\n dictionary = GenometricSpace.best_descriptive_meta_dict(path_to_bog, cluster_no)\n GenometricSpace.visualize_cloud_of_words(dictionary, image_path)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a sorted list of all of the files having the same extension under the same directory", "response": "def _get_files(extension, path):\n \"\"\"\n Returns a sorted list of all of the files having the same extension under the same directory\n :param extension: the extension of the data files such as 'gdm'\n :param path: path to the folder containing the files\n :return: sorted list of files\n \"\"\"\n # retrieves the files sharing the same extension\n files = []\n for file in os.listdir(path):\n if file.endswith(extension):\n files.append(os.path.join(path, file))\n return sorted(files)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_schema_file(extension, path):\n for file in os.listdir(path):\n if file.endswith(extension):\n return os.path.join(path, file)", "response": "Returns the schema file containing the specified extension"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef parse_schema(schema_file):\n e = xml.etree.ElementTree.parse(schema_file)\n root = e.getroot()\n cols = []\n for elem in root.findall(\".//{http://genomic.elet.polimi.it/entities}field\"): # XPATH\n cols.append(elem.text)\n return cols", "response": "Parses the schema file and returns the columns that are later going to represent the columns of the genometric space dataframe\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nparsing a single meta data file into a pandas dataframe", "response": "def parse_single_meta(self, fname, selected_meta_data):\n \"\"\"\n Parses a single meta data file\n :param fname: name of the file\n :param selected_meta_data: If not none then only the specified columns of metadata are parsed\n :return: the resulting pandas series\n \"\"\"\n # reads a meta data file into a dataframe\n columns = []\n data = []\n with open(fname) as f:\n for line in f:\n splitted = line.split('\\t')\n columns.append(splitted[0])\n data.append(splitted[1].split('\\n')[0]) # to remove the \\n values\n df = pd.DataFrame(data=data, index=columns)\n df = df.T\n sample = self._get_sample_name(fname)\n if selected_meta_data: # if null then keep all the columns\n try:\n df = df[selected_meta_data]\n except:\n pass\n df['sample'] = sample\n return df"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef parse_meta(self, selected_meta_data):\n # reads all meta data files\n files = self._get_files(\"meta\", self.path)\n df = pd.DataFrame()\n print(\"Parsing the metadata files...\")\n for f in tqdm(files):\n data = self.parse_single_meta(f, selected_meta_data)\n if data is not None:\n df = pd.concat([df, data], axis=0)\n df.index = df['sample']\n #\n # df = df.drop('sample', 1) # 1 for the columns\n return df", "response": "Parses all of the metadata files and returns a dataframe with the metadata columns that are contained in the selected_meta_data."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nparse a single region data file and returns a dataframe containing the region data.", "response": "def parse_single_data(self, path, cols, selected_region_data, selected_values, full_load):\n \"\"\"\n Parses a single region data file\n :param path: path to the file\n :param cols: the column names coming from the schema file\n :param selected_region_data: the selected of the region data to be parsed\n In most cases the analyst only needs a small subset of the region data\n :param selected_values: the selected values to be put in the matrix cells\n :param full_load: Specifies the method of parsing the data. If False then parser omits the parsing of zero(0)\n values in order to speed up and save memory. However, while creating the matrix, those zero values are going to be put into the matrix.\n (unless a row contains \"all zero columns\". This parsing is strongly recommended for sparse datasets.\n If the full_load parameter is True then all the zero(0) data are going to be read.\n :return: the dataframe containing the region data\n \"\"\"\n # reads a sample file\n df = pd.read_table(path, engine='c', sep=\"\\t\", lineterminator=\"\\n\", header=None)\n df.columns = cols # column names from schema\n df = df[selected_region_data]\n\n if not full_load:\n if type(selected_values) is list:\n df_2 = pd.DataFrame(dtype=float)\n for value in selected_values:\n df_3 = df.loc[df[value] != 0]\n df_2 = pd.concat([df_2, df_3], axis=0)\n df = df_2\n else:\n df = df.loc[df[selected_values] != 0]\n\n sample = self._get_sample_name(path)\n df['sample'] = sample\n return df"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef parse_data(self, selected_region_data, selected_values, full_load=False, extension=\"gdm\"):\n regions = list(selected_region_data)\n if type(selected_values) is list:\n regions.extend(selected_values)\n else:\n regions.append(selected_values)\n\n files = self._get_files(extension, self.path)\n df = pd.DataFrame(dtype=float)\n\n cols = self.parse_schema(self.schema)\n print(\"Parsing the data files...\")\n list_of_data = []\n for f in tqdm(files):\n data = self.parse_single_data(f, cols, regions, selected_values, full_load)\n list_of_data.append(data)\n print(\"pre-concat\")\n df = pd.concat(list_of_data)\n return df", "response": "Parses the data files and returns a dataframe with the data."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef from_pandas(regs, meta=None, chr_name=None, start_name=None, stop_name=None,\n strand_name=None, sample_name=None):\n \"\"\" Creates a GDataframe from a pandas dataframe of region and a pandas dataframe of metadata\n\n :param regs: a pandas Dataframe of regions that is coherent with the GMQL data model\n :param meta: (optional) a pandas Dataframe of metadata that is coherent with the regions\n :param chr_name: (optional) which column of :attr:`~.regs` is the chromosome\n :param start_name: (optional) which column of :attr:`~.regs` is the start\n :param stop_name: (optional) which column of :attr:`~.regs` is the stop\n :param strand_name: (optional) which column of :attr:`~.regs` is the strand\n :param sample_name: (optional) which column of :attr:`~.regs` represents the sample name\n of that region. If nothing is provided, all the region will be put in a single sample.\n :return: a GDataframe\n \"\"\"\n regs = check_regs(regs, chr_name, start_name, stop_name, strand_name, sample_name)\n regs = to_gmql_regions(regs)\n if meta is not None:\n if not check_meta(meta, regs):\n raise ValueError(\"Error. Meta dataframe is not GMQL standard\")\n else:\n meta = empty_meta(regs)\n return GDataframe(regs, meta)", "response": "Creates a GDataframe from a pandas dataframe of region and metadata."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nchecking that a dataframe of regions that are coherent with the GMQL data model.", "response": "def check_regs(region_df, chr_name=None, start_name=None, stop_name=None,\n strand_name=None, sample_name=None):\n \"\"\" Modifies a region dataframe to be coherent with the GMQL data model\n\n :param region_df: a pandas Dataframe of regions that is coherent with the GMQL data model\n :param chr_name: (optional) which column of :attr:`~.region_df` is the chromosome\n :param start_name: (optional) which column of :attr:`~.region_df` is the start\n :param stop_name: (optional) which column of :attr:`~.region_df` is the stop\n :param strand_name: (optional) which column of :attr:`~.region_df` is the strand\n :return: a modified pandas Dataframe\n \"\"\"\n\n if sample_name is None:\n region_df.index = np.repeat(default_id_sample, len(region_df))\n else:\n region_df = search_column(region_df, id_sample_aliases,\n id_sample_types, 'id_sample', sample_name)\n region_df = region_df.set_index(\"id_sample\", drop=True)\n region_df = region_df.sort_index()\n\n region_df = search_column(region_df, chr_aliases, chr_types, 'chr', chr_name)\n region_df = search_column(region_df, start_aliases, start_types, 'start', start_name)\n region_df = search_column(region_df, stop_aliases, stop_types, 'stop', stop_name)\n region_df = search_column(region_df, strand_aliases, strand_types, 'strand', strand_name)\n return region_df"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsaves the GDataframe to a local or remote location", "response": "def to_dataset_files(self, local_path=None, remote_path=None):\n \"\"\" Save the GDataframe to a local or remote location\n\n :param local_path: a local path to the folder in which the data must be saved\n :param remote_path: a remote dataset name that wants to be used for these data\n :return: None\n \"\"\"\n return FrameToGMQL.to_dataset_files(self, path_local=local_path, path_remote=remote_path)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef to_GMQLDataset(self, local_path=None, remote_path=None):\n local = None\n remote = None\n if (local_path is None) and (remote_path is None):\n # get a temporary path\n local = TempFileManager.get_new_dataset_tmp_folder()\n if local_path is not None:\n local = local_path\n if remote_path is not None:\n remote = remote_path\n self.to_dataset_files(local, remote)\n\n if local is not None:\n return Loader.load_from_path(local_path=local)\n elif remote is not None:\n raise NotImplementedError(\"The remote loading is not implemented yet!\")", "response": "Converts the GDataframe into a GMQLDataset for later local or remote computation."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef project_meta(self, attributes):\n if not isinstance(attributes, list):\n raise TypeError('attributes must be a list')\n meta_to_project = self.meta[attributes].applymap(lambda l: \", \".join(l))\n new_regs = self.regs.merge(meta_to_project, left_index=True, right_index=True)\n return GDataframe(regs=new_regs, meta=self.meta)", "response": "Projects the specified metadata attributes to a new GDataframe with additional region fields\n "} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ntransforms the GDataframe to a pivot matrix having as index and columns the ones specified.", "response": "def to_matrix(self, index_regs=None, index_meta=None,\n columns_regs=None, columns_meta=None,\n values_regs=None, values_meta=None, **kwargs):\n \"\"\" Transforms the GDataframe to a pivot matrix having as index and columns the\n ones specified. This function is a wrapper around the pivot_table function of Pandas.\n\n :param index_regs: list of region fields to use as index\n :param index_meta: list of metadata attributes to use as index\n :param columns_regs: list of region fields to use as columns\n :param columns_meta: list of metadata attributes to use as columns\n :param values_regs: list of region fields to use as values\n :param values_meta: list of metadata attributes to use as values\n :param kwargs: other parameters to pass to the pivot_table function\n :return: a Pandas dataframe having as index the union of index_regs and index_meta, as\n columns the union of columns_regs and columns_meta and as values ths union\n of values_regs and values_meta\n \"\"\"\n\n index_regs = index_regs if index_regs is not None else []\n index_meta = index_meta if index_meta is not None else []\n columns_regs = columns_regs if columns_regs is not None else []\n columns_meta = columns_meta if columns_meta is not None else []\n values_regs = values_regs if values_regs is not None else []\n values_meta = values_meta if values_meta is not None else []\n\n index_meta_s = set(index_meta)\n columns_meta_s = set(columns_meta)\n values_meta_s = set(values_meta)\n\n meta_to_project = list(index_meta_s.union(columns_meta_s)\\\n .union(values_meta_s)\\\n .difference(set(self.regs.columns)))\n res = self.project_meta(meta_to_project)\n pivot_columns = columns_meta + columns_regs\n pivot_index = index_meta + index_regs\n pivot_values = values_regs + values_meta\n\n return res.regs.pivot_table(index=pivot_index, columns=pivot_columns, values=pivot_values, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets Charts from Pollster.", "response": "def charts_get(self, **kwargs):\n \"\"\"\n Charts\n Returns a list of Charts, ordered by creation date (newest first). A Chart is chosen by Pollster editors. One example is \\\"Obama job approval - Democrats\\\". It is always based upon a single Question. Users should strongly consider basing their analysis on Questions instead. Charts are derived data; Pollster editors publish them and change them as editorial priorities change. \n\n This method makes a synchronous HTTP request by default. To make an\n asynchronous HTTP request, please define a `callback` function\n to be invoked when receiving the response.\n >>> def callback_function(response):\n >>> pprint(response)\n >>>\n >>> thread = api.charts_get(callback=callback_function)\n\n :param callback function: The callback function\n for asynchronous request. (optional)\n :param str cursor: Special string to index into the Array\n :param str tags: Comma-separated list of tag slugs. Only Charts with one or more of these tags and Charts based on Questions with one or more of these tags will be returned.\n :param date election_date: Date of an election, in YYYY-MM-DD format. Only Charts based on Questions pertaining to an election on this date will be returned.\n :return: InlineResponse200\n If the method is called asynchronously,\n returns the request thread.\n \"\"\"\n kwargs['_return_http_data_only'] = True\n if kwargs.get('callback'):\n return self.charts_get_with_http_info(**kwargs)\n else:\n (data) = self.charts_get_with_http_info(**kwargs)\n return data"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef charts_slug_get(self, slug, **kwargs):\n kwargs['_return_http_data_only'] = True\n if kwargs.get('callback'):\n return self.charts_slug_get_with_http_info(slug, **kwargs)\n else:\n (data) = self.charts_slug_get_with_http_info(slug, **kwargs)\n return data", "response": "Get a Chart by its unique identifier."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef charts_slug_pollster_chart_poll_questions_tsv_get(self, slug, **kwargs):\n kwargs['_return_http_data_only'] = True\n if kwargs.get('callback'):\n return self.charts_slug_pollster_chart_poll_questions_tsv_get_with_http_info(slug, **kwargs)\n else:\n (data) = self.charts_slug_pollster_chart_poll_questions_tsv_get_with_http_info(slug, **kwargs)\n return data", "response": "This method returns a TSV for a particular poll question in a Pollster Chart."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nestimating of what the polls suggest about trends Derived data presented on a Pollster Chart. The trendlines on a Pollster chart don't add up to 100: we calculate each label's trendline separately. Use the `charts/{slug}` response's `chart.pollster_estimates[0].algorithm` to find the algorithm Pollster used to generate these estimates. Pollster recalculates trendlines every time a new poll is entered. It also recalculates trendlines daily if they use the `bayesian-kallman` algorithm, because that algorithm's output changes depending on the end date. This method makes a synchronous HTTP request by default. To make an asynchronous HTTP request, please define a `callback` function to be invoked when receiving the response. >>> def callback_function(response): >>> pprint(response) >>> >>> thread = api.charts_slug_pollster_trendlines_tsv_get(slug, callback=callback_function) :param callback function: The callback function for asynchronous request. (optional) :param str slug: Unique Chart identifier. For example: `obama-job-approval` (required) :return: InlineResponse2002 If the method is called asynchronously, returns the request thread.", "response": "def charts_slug_pollster_trendlines_tsv_get(self, slug, **kwargs):\n \"\"\"\n Estimates of what the polls suggest about trends\n Derived data presented on a Pollster Chart. The trendlines on a Pollster chart don't add up to 100: we calculate each label's trendline separately. Use the `charts/{slug}` response's `chart.pollster_estimates[0].algorithm` to find the algorithm Pollster used to generate these estimates. Pollster recalculates trendlines every time a new poll is entered. It also recalculates trendlines daily if they use the `bayesian-kallman` algorithm, because that algorithm's output changes depending on the end date. \n\n This method makes a synchronous HTTP request by default. To make an\n asynchronous HTTP request, please define a `callback` function\n to be invoked when receiving the response.\n >>> def callback_function(response):\n >>> pprint(response)\n >>>\n >>> thread = api.charts_slug_pollster_trendlines_tsv_get(slug, callback=callback_function)\n\n :param callback function: The callback function\n for asynchronous request. (optional)\n :param str slug: Unique Chart identifier. For example: `obama-job-approval` (required)\n :return: InlineResponse2002\n If the method is called asynchronously,\n returns the request thread.\n \"\"\"\n kwargs['_return_http_data_only'] = True\n if kwargs.get('callback'):\n return self.charts_slug_pollster_trendlines_tsv_get_with_http_info(slug, **kwargs)\n else:\n (data) = self.charts_slug_pollster_trendlines_tsv_get_with_http_info(slug, **kwargs)\n return data"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef polls_get(self, **kwargs):\n kwargs['_return_http_data_only'] = True\n if kwargs.get('callback'):\n return self.polls_get_with_http_info(**kwargs)\n else:\n (data) = self.polls_get_with_http_info(**kwargs)\n return data", "response": "Get a list of polls on a Pollster."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets a Poll from the Pollster with the given slug.", "response": "def polls_slug_get(self, slug, **kwargs):\n \"\"\"\n Poll\n A Poll on Pollster is a collection of questions and responses published by a reputable survey house. This endpoint provides raw data from the survey house, plus Pollster-provided metadata about each question. Pollster editors don't include every question when they enter Polls, and they don't necessarily enter every subpopulation for the responses they _do_ enter. They make editorial decisions about which questions belong in the database. \n\n This method makes a synchronous HTTP request by default. To make an\n asynchronous HTTP request, please define a `callback` function\n to be invoked when receiving the response.\n >>> def callback_function(response):\n >>> pprint(response)\n >>>\n >>> thread = api.polls_slug_get(slug, callback=callback_function)\n\n :param callback function: The callback function\n for asynchronous request. (optional)\n :param str slug: Unique Poll identifier. For example: `gallup-26892`. (required)\n :return: Poll\n If the method is called asynchronously,\n returns the request thread.\n \"\"\"\n kwargs['_return_http_data_only'] = True\n if kwargs.get('callback'):\n return self.polls_slug_get_with_http_info(slug, **kwargs)\n else:\n (data) = self.polls_slug_get_with_http_info(slug, **kwargs)\n return data"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef questions_get(self, **kwargs):\n kwargs['_return_http_data_only'] = True\n if kwargs.get('callback'):\n return self.questions_get_with_http_info(**kwargs)\n else:\n (data) = self.questions_get_with_http_info(**kwargs)\n return data", "response": "Get Questions from Pollster."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets a single Question by its unique identifier.", "response": "def questions_slug_get(self, slug, **kwargs):\n \"\"\"\n Question\n A Question is chosen by Pollster editors. One example is \\\"Obama job approval\\\". Different survey houses may publish varying phrasings (\\\"Do you approve or disapprove\\\" vs \\\"What do you think of the job\\\") and prompt readers with varying responses (one poll might have \\\"Approve\\\" and \\\"Disapprove\\\"; another poll might have \\\"Strongly approve\\\" and \\\"Somewhat approve\\\"). Those variations do not appear in this API endpoint. \n\n This method makes a synchronous HTTP request by default. To make an\n asynchronous HTTP request, please define a `callback` function\n to be invoked when receiving the response.\n >>> def callback_function(response):\n >>> pprint(response)\n >>>\n >>> thread = api.questions_slug_get(slug, callback=callback_function)\n\n :param callback function: The callback function\n for asynchronous request. (optional)\n :param str slug: Unique Question identifier. For example: `00c -Pres (44) Obama - Job Approval - National`. (Remember to URL-encode this parameter when querying.) (required)\n :return: Question\n If the method is called asynchronously,\n returns the request thread.\n \"\"\"\n kwargs['_return_http_data_only'] = True\n if kwargs.get('callback'):\n return self.questions_slug_get_with_http_info(slug, **kwargs)\n else:\n (data) = self.questions_slug_get_with_http_info(slug, **kwargs)\n return data"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef questions_slug_poll_responses_clean_tsv_get(self, slug, **kwargs):\n kwargs['_return_http_data_only'] = True\n if kwargs.get('callback'):\n return self.questions_slug_poll_responses_clean_tsv_get_with_http_info(slug, **kwargs)\n else:\n (data) = self.questions_slug_poll_responses_clean_tsv_get_with_http_info(slug, **kwargs)\n return data", "response": "This method is used to get the TSV of a Poll from the TSV table."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets a single entry in the raw data from which we derived a question.", "response": "def questions_slug_poll_responses_raw_tsv_get(self, slug, **kwargs):\n \"\"\"\n One row per PollQuestion+Subpopulation+Response concerning the given Question (Large)\n Raw data from which we derived `poll-responses-clean.tsv`. Each row represents a single PollQuestion+Subpopulation+Response. See the Poll API for a description of these terms. Group results by `(poll_slug, subpopulation, question_text)`: that's how the survey houses group them. This response can be several megabytes large. We encourage you to consider `poll-responses-clean.tsv` instead. \n\n This method makes a synchronous HTTP request by default. To make an\n asynchronous HTTP request, please define a `callback` function\n to be invoked when receiving the response.\n >>> def callback_function(response):\n >>> pprint(response)\n >>>\n >>> thread = api.questions_slug_poll_responses_raw_tsv_get(slug, callback=callback_function)\n\n :param callback function: The callback function\n for asynchronous request. (optional)\n :param str slug: Unique Question identifier. For example: `00c -Pres (44) Obama - Job Approval - National`. (Remember to URL-encode this parameter when querying.) (required)\n :return: InlineResponse2005\n If the method is called asynchronously,\n returns the request thread.\n \"\"\"\n kwargs['_return_http_data_only'] = True\n if kwargs.get('callback'):\n return self.questions_slug_poll_responses_raw_tsv_get_with_http_info(slug, **kwargs)\n else:\n (data) = self.questions_slug_poll_responses_raw_tsv_get_with_http_info(slug, **kwargs)\n return data"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting a list of Tags.", "response": "def tags_get(self, **kwargs):\n \"\"\"\n Tags\n Returns the list of Tags. A Tag can apply to any number of Charts and Questions; Charts and Questions, in turn, can have any number of Tags. Tags all look `like-this`: lowercase letters, numbers and hyphens. \n\n This method makes a synchronous HTTP request by default. To make an\n asynchronous HTTP request, please define a `callback` function\n to be invoked when receiving the response.\n >>> def callback_function(response):\n >>> pprint(response)\n >>>\n >>> thread = api.tags_get(callback=callback_function)\n\n :param callback function: The callback function\n for asynchronous request. (optional)\n :return: list[Tag]\n If the method is called asynchronously,\n returns the request thread.\n \"\"\"\n kwargs['_return_http_data_only'] = True\n if kwargs.get('callback'):\n return self.tags_get_with_http_info(**kwargs)\n else:\n (data) = self.tags_get_with_http_info(**kwargs)\n return data"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef sanitize_for_serialization(self, obj):\n types = (str, float, bool, bytes) + tuple(integer_types) + (text_type,)\n if isinstance(obj, type(None)):\n return None\n elif isinstance(obj, types):\n return obj\n elif isinstance(obj, list):\n return [self.sanitize_for_serialization(sub_obj)\n for sub_obj in obj]\n elif isinstance(obj, tuple):\n return tuple(self.sanitize_for_serialization(sub_obj)\n for sub_obj in obj)\n elif isinstance(obj, (datetime, date)):\n return obj.isoformat()\n else:\n if isinstance(obj, dict):\n obj_dict = obj\n else:\n # Convert model obj to dict except\n # attributes `swagger_types`, `attribute_map`\n # and attributes which value is not None.\n # Convert attribute name to json key in\n # model definition for request.\n obj_dict = {obj.attribute_map[attr]: getattr(obj, attr)\n for attr, _ in iteritems(obj.swagger_types)\n if getattr(obj, attr) is not None}\n\n return {key: self.sanitize_for_serialization(val)\n for key, val in iteritems(obj_dict)}", "response": "Sanitize the data for serialization."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nselecting the content - type based on an array of content - types provided.", "response": "def select_header_content_type(self, content_types):\n \"\"\"\n Returns `Content-Type` based on an array of content_types provided.\n\n :param content_types: List of content-types.\n :return: Content-Type (e.g. application/json).\n \"\"\"\n if not content_types:\n return 'application/json'\n\n content_types = list(map(lambda x: x.lower(), content_types))\n\n if 'application/json' in content_types or '*/*' in content_types:\n return 'application/json'\n else:\n return content_types[0]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef __deserialize_model(self, data, klass):\n instance = klass()\n\n if not instance.swagger_types:\n return data\n\n for attr, attr_type in iteritems(instance.swagger_types):\n if data is not None \\\n and instance.attribute_map[attr] in data\\\n and isinstance(data, (list, dict)):\n value = data[instance.attribute_map[attr]]\n setattr(instance, '_' + attr, self.__deserialize(value, attr_type))\n\n return instance", "response": "Deserializes list or dict to model object."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef render_flatpage(request, f):\n # If registration is required for accessing this page, and the user isn't\n # logged in, redirect to the login page.\n if f.registration_required and not request.user.is_authenticated():\n from django.contrib.auth.views import redirect_to_login\n return redirect_to_login(request.path)\n if f.template_name:\n t = loader.select_template((f.template_name, DEFAULT_TEMPLATE))\n else:\n t = loader.get_template(DEFAULT_TEMPLATE)\n\n # To avoid having to always use the \"|safe\" filter in flatpage templates,\n # mark the title and content as already safe (since they are raw HTML\n # content in the first place).\n f.title = mark_safe(f.title)\n f.content = mark_safe(f.content)\n\n response = HttpResponse(t.render({\n 'flatpage': f\n }, request))\n\n try:\n from django.core.xheaders import populate_xheaders\n populate_xheaders(request, response, FlatPage_i18n, f.id)\n except ImportError:\n pass\n return response", "response": "Render a flat page."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef pidfile(self):\n return os.path.abspath(\n os.path.expandvars(\n os.path.expanduser(\n self._pidfile,\n ),\n ),\n )", "response": "Get the absolute path of the pidfile."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef SQRT(argument):\n if isinstance(argument, MetaField):\n return argument._unary_expression(\"SQRT\")\n elif isinstance(argument, RegField):\n return argument._unary_expression(\"SQRT\")\n else:\n raise TypeError(\"You have to give as input a RegField (dataset.field)\"\n \"or a MetaField (dataset['field']\")", "response": "Computes the square matrix of the argument"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nenables the user to login to the remote GMQL service.", "response": "def login():\n \"\"\" Enables the user to login to the remote GMQL service.\n If both username and password are None, the user will be connected as guest.\n \"\"\"\n from .RemoteConnection.RemoteManager import RemoteManager\n global __remote_manager, __session_manager\n\n logger = logging.getLogger()\n remote_address = get_remote_address()\n res = __session_manager.get_session(remote_address)\n if res is None:\n # there is no session for this address, let's login as guest\n warnings.warn(\"There is no active session for address {}. Logging as Guest user\".format(remote_address))\n rm = RemoteManager(address=remote_address)\n rm.login()\n session_type = \"guest\"\n else:\n # there is a previous session for this address, let's do an auto login\n # using that access token\n logger.info(\"Logging using stored authentication token\")\n rm = RemoteManager(address=remote_address, auth_token=res[1])\n # if the access token is not valid anymore (therefore we are in guest mode)\n # the auto_login function will perform a guest login from scratch\n session_type = rm.auto_login(how=res[2])\n # store the new session\n __remote_manager = rm\n access_time = int(time.time())\n auth_token = rm.auth_token\n __session_manager.add_session(remote_address, auth_token, access_time, session_type)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef main(idle):\n while True:\n\n LOG.debug(\"Sleeping for {0} seconds.\".format(idle))\n time.sleep(idle)", "response": "This is the main function which runs a loop."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_metadata(self):\n if self.path_or_name is None:\n raise ValueError(\"You cannot explore the metadata of an intermediate query.\"\n \"You can get metadata only after a load_from_local or load_from_remote\")\n if self.location == 'local':\n return self.__get_metadata_local()\n elif self.location == 'remote':\n return self.__get_metadata_remote()", "response": "Returns the metadata related to the current GMQLDataset."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef MetaField(self, name, t=None):\n return MetaField(name=name, index=self.__index, t=t)", "response": "Creates an instance of a MetaField object that is used in creating expressions for projection or selection."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef select(self, meta_predicate=None, region_predicate=None,\n semiJoinDataset=None, semiJoinMeta=None):\n \"\"\" *Wrapper of* ``SELECT``\n\n Selection operation. Enables to filter datasets on the basis of region features or metadata attributes. In\n addition it is possibile to perform a selection based on the existence of certain metadata\n :attr:`~.semiJoinMeta` attributes and the matching of their values with those associated with at\n least one sample in an external dataset :attr:`~.semiJoinDataset`.\n\n Therefore, the selection can be based on:\n\n * *Metadata predicates*: selection based on the existence and values of certain\n metadata attributes in each sample. In predicates, attribute-value conditions\n can be composed using logical predicates & (and), | (or) and ~ (not)\n * *Region predicates*: selection based on region attributes. Conditions\n can be composed using logical predicates & (and), | (or) and ~ (not)\n * *SemiJoin clauses*: selection based on the existence of certain metadata :attr:`~.semiJoinMeta`\n attributes and the matching of their values with those associated with at\n least one sample in an external dataset :attr:`~.semiJoinDataset`\n\n\n In the following example we select all the samples from Example_Dataset_1 regarding antibody CTCF.\n From these samples we select only the regions on chromosome 6. Finally we select only the samples which have\n a matching antibody_targetClass in Example_Dataset_2::\n\n import gmql as gl\n d1 = gl.get_example_dataset(\"Example_Dataset_1\")\n d2 = gl.get_example_dataset(\"Example_Dataset_2\")\n\n d_select = d.select(meta_predicate = d['antibody'] == \"CTCF\",\n region_predicate = d.chr == \"chr6\",\n semiJoinDataset=d2, semiJoinMeta=[\"antibody_targetClass\"])\n\n :param meta_predicate: logical predicate on the metadata pairs\n :param region_predicate: logical predicate on the region feature values\n :param semiJoinDataset: an other GMQLDataset\n :param semiJoinMeta: a list of metadata attributes (strings)\n :return: a new GMQLDataset\n \"\"\"\n\n semiJoinDataset_exists = False\n if isinstance(meta_predicate, MetaField):\n meta_condition = Some(meta_predicate.getMetaCondition())\n elif meta_predicate is None:\n meta_condition = none()\n else:\n raise TypeError(\"meta_predicate must be a MetaField or None.\"\n \" {} was provided\".format(type(meta_predicate)))\n\n if isinstance(region_predicate, RegField):\n region_condition = Some(region_predicate.getRegionCondition())\n elif region_predicate is None:\n region_condition = none()\n else:\n raise TypeError(\"region_predicate must be a RegField or None.\"\n \" {} was provided\".format(type(region_predicate)))\n\n new_location = self.location\n new_local_sources, new_remote_sources = self._local_sources, self._remote_sources\n\n if isinstance(semiJoinDataset, GMQLDataset):\n other_dataset = Some(semiJoinDataset.__index)\n semiJoinDataset_exists = True\n elif semiJoinDataset is None:\n other_dataset = none()\n else:\n raise TypeError(\"semiJoinDataset must be a GMQLDataset or None.\"\n \" {} was provided\".format(type(semiJoinDataset)))\n\n if isinstance(semiJoinMeta, list) and \\\n all([isinstance(x, str) for x in semiJoinMeta]):\n if semiJoinDataset_exists:\n semi_join = Some(self.opmng.getMetaJoinCondition(semiJoinMeta))\n\n new_local_sources, new_remote_sources = self.__combine_sources(self, semiJoinDataset)\n new_location = self.__combine_locations(self, semiJoinDataset)\n else:\n raise ValueError(\"semiJoinDataset and semiJoinMeta must be present at the \"\n \"same time or totally absent\")\n elif semiJoinMeta is None:\n semi_join = none()\n else:\n raise TypeError(\"semiJoinMeta must be a list of strings or None.\"\n \" {} was provided\".format(type(semiJoinMeta)))\n\n new_index = self.opmng.select(self.__index, other_dataset,\n semi_join, meta_condition, region_condition)\n\n return GMQLDataset(index=new_index, location=new_location, local_sources=new_local_sources,\n remote_sources=new_remote_sources, meta_profile=self.meta_profile)", "response": "Select metadata and region features from the external dataset."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nwrapping of ~. select function filtering samples only based on metadata.", "response": "def meta_select(self, predicate=None, semiJoinDataset=None, semiJoinMeta=None):\n \"\"\"\n *Wrapper of* ``SELECT``\n\n Wrapper of the :meth:`~.select` function filtering samples only based on metadata.\n\n :param predicate: logical predicate on the values of the rows\n :param semiJoinDataset: an other GMQLDataset \n :param semiJoinMeta: a list of metadata\n :return: a new GMQLDataset\n\n \n Example 1::\n \n output_dataset = dataset.meta_select(dataset['patient_age'] < 70)\n # This statement can be written also as\n output_dataset = dataset[ dataset['patient_age'] < 70 ]\n\n Example 2::\n\n output_dataset = dataset.meta_select( (dataset['tissue_status'] == 'tumoral') &\n (tumor_tag != 'gbm') | (tumor_tag == 'brca'))\n # This statement can be written also as\n output_dataset = dataset[ (dataset['tissue_status'] == 'tumoral') &\n (tumor_tag != 'gbm') | (tumor_tag == 'brca') ]\n\n Example 3::\n\n JUN_POLR2A_TF = HG19_ENCODE_NARROW.meta_select( JUN_POLR2A_TF['antibody_target'] == 'JUN',\n semiJoinDataset=POLR2A_TF, semiJoinMeta=['cell'])\n\n The meta selection predicate can use all the classical equalities and disequalities\n {>, <, >=, <=, ==, !=} and predicates can be connected by the classical logical symbols\n {& (AND), | (OR), ~ (NOT)} plus the *isin* function.\n\n \"\"\"\n\n return self.select(meta_predicate=predicate, semiJoinDataset=semiJoinDataset, semiJoinMeta=semiJoinMeta)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nwrap of ~. select function that returns a new GMQLDataset containing only the regions whose field score is greater than the sample s size.", "response": "def reg_select(self, predicate=None, semiJoinDataset=None, semiJoinMeta=None):\n \"\"\"\n *Wrapper of* ``SELECT``\n\n Wrapper of the :meth:`~.select` function filtering regions only based on region attributes.\n\n :param predicate: logical predicate on the values of the regions\n :param semiJoinDataset: an other GMQLDataset \n :param semiJoinMeta: a list of metadata\n :return: a new GMQLDataset\n\n An example of usage::\n\n new_dataset = dataset.reg_select((dataset.chr == 'chr1') | (dataset.pValue < 0.9))\n\n You can also use Metadata attributes in selection::\n\n new_dataset = dataset.reg_select(dataset.score > dataset['size'])\n\n This statement selects all the regions whose field score is strictly higher than the sample\n metadata attribute size.\n\n The region selection predicate can use all the classical equalities and disequalities\n {>, <, >=, <=, ==, !=} and predicates can be connected by the classical logical symbols\n {& (AND), | (OR), ~ (NOT)} plus the *isin* function.\n\n In order to be sure about the correctness of the expression, please use parenthesis to delimit\n the various predicates.\n \"\"\"\n return self.select(region_predicate=predicate, semiJoinMeta=semiJoinMeta, semiJoinDataset=semiJoinDataset)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a new GMQLDataset from the input dataset and returns the new dataset with all the samples in the input one.", "response": "def project(self, projected_meta=None, new_attr_dict=None, all_but_meta=None,\n projected_regs=None, new_field_dict=None, all_but_regs=None):\n \"\"\"\n *Wrapper of* ``PROJECT``\n\n The PROJECT operator creates, from an existing dataset, a new dataset with all the samples\n (with their regions and region values) in the input one, but keeping for each sample in the input\n dataset only those metadata and/or region attributes expressed in the operator parameter list.\n Region coordinates and values of the remaining metadata and region attributes remain equal\n to those in the input dataset. Differently from the SELECT operator, PROJECT allows to:\n * Remove existing metadata and/or region attributes from a dataset;\n * Create new metadata and/or region attributes to be added to the result.\n\n :param projected_meta: list of metadata attributes to project on\n :param new_attr_dict: an optional dictionary of the form {'new_meta_1': function1,\n 'new_meta_2': function2, ...} in which every function computes\n the new metadata attribute based on the values of the others\n :param all_but_meta: list of metadata attributes that must be excluded from the projection\n :param projected_regs: list of the region fields to select\n :param new_field_dict: an optional dictionary of the form {'new_field_1':\n function1, 'new_field_2': function2, ...} in which every function\n computes the new region field based on the values of the others\n :param all_but_regs: list of region fields that must be excluded from the projection\n :return: a new GMQLDataset\n \"\"\"\n projected_meta_exists = False\n if isinstance(projected_meta, list) and \\\n all([isinstance(x, str) for x in projected_meta]):\n projected_meta = Some(projected_meta)\n projected_meta_exists = True\n elif projected_meta is None:\n projected_meta = none()\n else:\n raise TypeError(\"projected_meta must be a list of strings or None.\"\n \" {} was provided\".format(type(projected_meta)))\n\n if isinstance(new_attr_dict, dict):\n meta_ext = []\n expBuild = self.pmg.getNewExpressionBuilder(self.__index)\n for k in new_attr_dict.keys():\n item = new_attr_dict[k]\n if isinstance(k, str):\n if isinstance(item, MetaField):\n me = expBuild.createMetaExtension(k, item.getMetaExpression())\n elif isinstance(item, int):\n me = expBuild.createMetaExtension(k, expBuild.getMEType(\"int\", str(item)))\n elif isinstance(item, str):\n me = expBuild.createMetaExtension(k, expBuild.getMEType(\"string\", item))\n elif isinstance(item, float):\n me = expBuild.createMetaExtension(k, expBuild.getMEType(\"float\", str(item)))\n else:\n raise TypeError(\"Type {} of item of new_attr_dict is not valid\".format(type(item)))\n meta_ext.append(me)\n else:\n raise TypeError(\"The key of new_attr_dict must be a string. \"\n \"{} was provided\".format(type(k)))\n meta_ext = Some(meta_ext)\n elif new_attr_dict is None:\n meta_ext = none()\n else:\n raise TypeError(\"new_attr_dict must be a dictionary.\"\n \" {} was provided\".format(type(new_attr_dict)))\n\n if isinstance(all_but_meta, list) and \\\n all([isinstance(x, str) for x in all_but_meta]):\n if not projected_meta_exists:\n all_but_meta = Some(all_but_meta)\n all_but_value = True\n else:\n raise ValueError(\"all_but_meta and projected_meta are mutually exclusive\")\n elif all_but_meta is None:\n all_but_meta = none()\n all_but_value = False\n else:\n raise TypeError(\"all_but_meta must be a list of strings.\"\n \" {} was provided\".format(type(all_but_meta)))\n projected_meta = all_but_meta if all_but_value else projected_meta\n\n projected_regs_exists = False\n if isinstance(projected_regs, list) and \\\n all([isinstance(x, str) for x in projected_regs]):\n projected_regs = Some(projected_regs)\n projected_regs_exists = True\n elif projected_regs is None:\n projected_regs = none()\n else:\n raise TypeError(\"projected_regs must be a list of strings or None.\"\n \" {} was provided\".format(type(projected_regs)))\n\n if isinstance(new_field_dict, dict):\n regs_ext = []\n expBuild = self.pmg.getNewExpressionBuilder(self.__index)\n for k in new_field_dict.keys():\n item = new_field_dict[k]\n if isinstance(k, str):\n if isinstance(item, RegField):\n re = expBuild.createRegionExtension(k, item.getRegionExpression())\n elif isinstance(item, MetaField):\n re = expBuild.createRegionExtension(k, item.reMetaNode)\n elif isinstance(item, int):\n re = expBuild.createRegionExtension(k, expBuild.getREType(\"float\", str(item)))\n elif isinstance(item, str):\n re = expBuild.createRegionExtension(k, expBuild.getREType(\"string\", item))\n elif isinstance(item, float):\n re = expBuild.createRegionExtension(k, expBuild.getREType(\"float\", str(item)))\n else:\n raise TypeError(\"Type {} of item of new_field_dict is not valid\".format(type(item)))\n regs_ext.append(re)\n else:\n raise TypeError(\"The key of new_field_dict must be a string. \"\n \"{} was provided\".format(type(k)))\n regs_ext = Some(regs_ext)\n elif new_field_dict is None:\n regs_ext = none()\n else:\n raise TypeError(\"new_field_dict must be a dictionary.\"\n \" {} was provided\".format(type(new_field_dict)))\n\n if isinstance(all_but_regs, list) and \\\n all([isinstance(x, str) for x in all_but_regs]):\n if not projected_regs_exists:\n all_but_regs = Some(all_but_regs)\n else:\n raise ValueError(\"all_but_meta and projected_meta are mutually exclusive\")\n elif all_but_regs is None:\n all_but_regs = none()\n else:\n raise TypeError(\"all_but_regs must be a list of strings.\"\n \" {} was provided\".format(type(all_but_regs)))\n\n new_index = self.opmng.project(self.__index, projected_meta, meta_ext, all_but_value,\n projected_regs, all_but_regs, regs_ext)\n return GMQLDataset(index=new_index, location=self.location,\n local_sources=self._local_sources, remote_sources=self._remote_sources,\n meta_profile=self.meta_profile)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nprojects the metadata based on a list of attribute names and return a new GMQLDataset object", "response": "def meta_project(self, attr_list=None, all_but=None, new_attr_dict=None):\n \"\"\"\n *Wrapper of* ``PROJECT``\n\n Project the metadata based on a list of attribute names\n \n :param attr_list: list of the metadata fields to select\n :param all_but: list of metadata that must be excluded from the projection.\n :param new_attr_dict: an optional dictionary of the form {'new_field_1': function1,\n 'new_field_2': function2, ...} in which every function computes\n the new field based on the values of the others\n :return: a new GMQLDataset\n\n Notice that if attr_list is specified, all_but cannot be specified and viceversa.\n\n Examples::\n\n new_dataset = dataset.meta_project(attr_list=['antibody', 'ID'],\n new_attr_dict={'new_meta': dataset['ID'] + 100})\n\n \"\"\"\n return self.project(projected_meta=attr_list, new_attr_dict=new_attr_dict, all_but_meta=all_but)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef extend(self, new_attr_dict):\n if isinstance(new_attr_dict, dict):\n expBuild = self.pmg.getNewExpressionBuilder(self.__index)\n aggregates = []\n for k in new_attr_dict.keys():\n if isinstance(k, str):\n item = new_attr_dict[k]\n if isinstance(item, Aggregate):\n op_name = item.get_aggregate_name()\n op_argument = Some(item.get_argument()) if item.is_unary() else none()\n regsToMeta = expBuild.getRegionsToMeta(op_name, k, op_argument)\n aggregates.append(regsToMeta)\n else:\n raise TypeError(\"The items in new_reg_fields must be Aggregates.\"\n \" {} was provided\".format(type(item)))\n else:\n raise TypeError(\"The key of new_attr_dict must be a string. \"\n \"{} was provided\".format(type(k)))\n else:\n raise TypeError(\"new_attr_dict must be a dictionary. \"\n \"{} was provided\".format(type(new_attr_dict)))\n\n new_index = self.opmng.extend(self.__index, aggregates)\n return GMQLDataset(index=new_index, location=self.location,\n local_sources=self._local_sources,\n remote_sources=self._remote_sources, meta_profile=self.meta_profile)", "response": "This method extends the current GMQLDataset with the values from the input dictionary."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef normal_cover(self, minAcc, maxAcc, groupBy=None, new_reg_fields=None):\n return self.cover(minAcc, maxAcc, groupBy, new_reg_fields, cover_type=\"normal\")", "response": "A wrapper around the COVER method that returns a new set of normal entries for the given time range."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef flat_cover(self, minAcc, maxAcc, groupBy=None, new_reg_fields=None):\n return self.cover(minAcc, maxAcc, groupBy, new_reg_fields, cover_type=\"flat\")", "response": "Returns a flat version of the cover function that returns all the regions in the COVER that are in the given range."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a COVER that returns only those that are within the given minimum and maximum accumulation index.", "response": "def summit_cover(self, minAcc, maxAcc, groupBy=None, new_reg_fields=None):\n \"\"\"\n *Wrapper of* ``COVER``\n\n Variant of the function :meth:`~.cover` that returns only those portions of the COVER\n result where the maximum number of regions overlap (this is done by returning only\n regions that start from a position after which the number of overlaps does not \n increase, and stop at a position where either the number of overlapping regions decreases \n or violates the maximum accumulation index).\n \n Equivalent to calling::\n \n cover(\"summit\", ...)\n \"\"\"\n return self.cover(minAcc, maxAcc, groupBy, new_reg_fields, cover_type=\"summit\")"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a histogram of all regions contributing to a particular accumulation value.", "response": "def histogram_cover(self, minAcc, maxAcc, groupBy=None, new_reg_fields=None):\n \"\"\"\n *Wrapper of* ``COVER``\n\n Variant of the function :meth:`~.cover` that returns all regions contributing to \n the COVER divided in different (contiguous) parts according to their accumulation \n index value (one part for each different accumulation value), which is assigned to \n the AccIndex region attribute.\n \n Equivalent to calling::\n \n cover(\"histogram\", ...)\n \"\"\"\n return self.cover(minAcc, maxAcc, groupBy, new_reg_fields, cover_type=\"histogram\")"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef join(self, experiment, genometric_predicate, output=\"LEFT\", joinBy=None,\n refName=\"REF\", expName=\"EXP\", left_on=None, right_on=None):\n \"\"\"\n *Wrapper of* ``JOIN``\n\n The JOIN operator takes in input two datasets, respectively known as anchor\n (the first/left one) and experiment (the second/right one) and returns a\n dataset of samples consisting of regions extracted from the operands \n according to the specified condition (known as genometric predicate). \n The number of generated output samples is the Cartesian product of the number \n of samples in the anchor and in the experiment dataset (if no joinby close \n if specified). The attributes (and their values) of the regions in the output \n dataset are the union of the region attributes (with their values) in the input \n datasets; homonymous attributes are disambiguated by prefixing their name with \n their dataset name. The output metadata are the union of the input metadata, \n with their attribute names prefixed with their input dataset name.\n \n :param experiment: an other GMQLDataset\n :param genometric_predicate: a list of Genometric atomic conditions. For an explanation of each of them\n go to the respective page.\n :param output: one of four different values that declare which region is given in output \n for each input pair of anchor and experiment regions satisfying the genometric predicate:\n \n * 'LEFT': outputs the anchor regions from the anchor dataset that satisfy the genometric predicate\n * 'RIGHT': outputs the anchor regions from the experiment dataset that satisfy the genometric predicate\n * 'INT': outputs the overlapping part (intersection) of the anchor and experiment regions that satisfy\n the genometric predicate; if the intersection is empty, no output is produced\n * 'CONTIG': outputs the concatenation between the anchor and experiment regions that satisfy the \n genometric predicate, i.e. the output region is defined as having left (right) coordinates \n equal to the minimum (maximum) of the corresponding coordinate values in the anchor and \n experiment regions satisfying the genometric predicate\n\n\n :param joinBy: list of metadata attributes\n :param refName: name that you want to assign to the reference dataset\n :param expName: name that you want to assign to the experiment dataset\n :param left_on: list of region fields of the reference on which the join must be performed\n :param right_on: list of region fields of the experiment on which the join must be performed\n :return: a new GMQLDataset\n \n An example of usage, in which we perform the join operation between Example_Dataset_1 and Example_Dataset_2\n specifying than we want to join the regions of the former with the first regions at a minimim distance of 120Kb\n of the latter and finally we want to output the regions of Example_Dataset_2 matching the criteria::\n \n import gmql as gl\n\n d1 = gl.get_example_dataset(\"Example_Dataset_1\")\n d2 = gl.get_example_dataset(\"Example_Dataset_2\")\n\n result_dataset = d1.join(experiment=d2,\n genometric_predicate=[gl.MD(1), gl.DGE(120000)],\n output=\"right\")\n \"\"\"\n\n if isinstance(experiment, GMQLDataset):\n other_idx = experiment.__index\n else:\n raise TypeError(\"experiment must be a GMQLDataset. \"\n \"{} was provided\".format(type(experiment)))\n if isinstance(genometric_predicate, list) and \\\n all([isinstance(x, GenometricCondition) for x in genometric_predicate]):\n regionJoinCondition = self.opmng.getRegionJoinCondition(list(map(lambda x: x.get_gen_condition(),\n genometric_predicate)))\n else:\n raise TypeError(\"genometric_predicate must be a list og GenometricCondition. \"\n \"{} was found\".format(type(genometric_predicate)))\n\n if isinstance(output, str):\n regionBuilder = self.opmng.getRegionBuilderJoin(output)\n else:\n raise TypeError(\"output must be a string. \"\n \"{} was provided\".format(type(output)))\n\n if not isinstance(expName, str):\n raise TypeError(\"expName must be a string. {} was provided\".format(type(expName)))\n\n if not isinstance(refName, str):\n raise TypeError(\"refName must be a string. {} was provided\".format(type(expName)))\n\n if isinstance(joinBy, list) and \\\n all([isinstance(x, str) for x in joinBy]):\n metaJoinCondition = Some(self.opmng.getMetaJoinCondition(joinBy))\n elif joinBy is None:\n metaJoinCondition = none()\n else:\n raise TypeError(\"joinBy must be a list of strings. \"\n \"{} was found\".format(type(joinBy)))\n\n left_on_exists = False\n left_on_len = 0\n if isinstance(left_on, list) and \\\n all([isinstance(x, str) for x in left_on]):\n left_on_len = len(left_on)\n left_on = Some(left_on)\n left_on_exists = True\n elif left_on is None:\n left_on = none()\n else:\n raise TypeError(\"left_on must be a list of strings. \"\n \"{} was provided\".format(type(left_on)))\n\n if isinstance(right_on, list) and \\\n all([isinstance(x, str)] for x in right_on) and \\\n left_on_exists and len(right_on) == left_on_len:\n right_on = Some(right_on)\n elif right_on is None and not left_on_exists:\n right_on = none()\n else:\n raise TypeError(\"right_on must be a list of strings. \"\n \"{} was provided\".format(type(right_on)))\n\n new_index = self.opmng.join(self.__index, other_idx,\n metaJoinCondition, regionJoinCondition, regionBuilder,\n refName, expName, left_on, right_on)\n new_local_sources, new_remote_sources = self.__combine_sources(self, experiment)\n new_location = self.__combine_locations(self, experiment)\n return GMQLDataset(index=new_index, location=new_location,\n local_sources=new_local_sources,\n remote_sources=new_remote_sources, meta_profile=self.meta_profile)", "response": "This function joins two GMQLDataset objects and returns a GMQLDataset object that is the union of the input two datasets with the same number of samples and the number of samples in the output."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef difference(self, other, joinBy=None, exact=False):\n\n if isinstance(other, GMQLDataset):\n other_idx = other.__index\n else:\n raise TypeError(\"other must be a GMQLDataset. \"\n \"{} was provided\".format(type(other)))\n\n if isinstance(joinBy, list) and \\\n all([isinstance(x, str) for x in joinBy]):\n metaJoinCondition = Some(self.opmng.getMetaJoinCondition(joinBy))\n elif joinBy is None:\n metaJoinCondition = none()\n else:\n raise TypeError(\"joinBy must be a list of strings. \"\n \"{} was provided\".format(type(joinBy)))\n\n if not isinstance(exact, bool):\n raise TypeError(\"exact must be a boolean. \"\n \"{} was provided\".format(type(exact)))\n\n new_index = self.opmng.difference(self.__index, other_idx, metaJoinCondition, exact)\n\n new_local_sources, new_remote_sources = self.__combine_sources(self, other)\n new_location = self.__combine_locations(self, other)\n return GMQLDataset(index=new_index, location=new_location,\n local_sources=new_local_sources,\n remote_sources=new_remote_sources,\n meta_profile=self.meta_profile)", "response": "This method returns a GMQLDataset that contains the difference between the current and the given other GMQLDataset."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef union(self, other, left_name=\"LEFT\", right_name=\"RIGHT\"):\n\n if not isinstance(left_name, str) or \\\n not isinstance(right_name, str):\n raise TypeError(\"left_name and right_name must be strings. \"\n \"{} - {} was provided\".format(type(left_name), type(right_name)))\n\n if isinstance(other, GMQLDataset):\n other_idx = other.__index\n else:\n raise TypeError(\"other must be a GMQLDataset. \"\n \"{} was provided\".format(type(other)))\n\n if len(left_name) == 0 or len(right_name) == 0:\n raise ValueError(\"left_name and right_name must not be empty\")\n new_index = self.opmng.union(self.__index, other_idx, left_name, right_name)\n new_local_sources, new_remote_sources = self.__combine_sources(self, other)\n new_location = self.__combine_locations(self, other)\n return GMQLDataset(index=new_index, location=new_location,\n local_sources=new_local_sources,\n remote_sources=new_remote_sources,\n meta_profile=self.meta_profile)", "response": "This function performs a union operation on the current GMQLDataset and returns the result."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef merge(self, groupBy=None):\n\n if isinstance(groupBy, list) and \\\n all([isinstance(x, str) for x in groupBy]):\n groupBy = Some(groupBy)\n elif groupBy is None:\n groupBy = none()\n else:\n raise TypeError(\"groupBy must be a list of strings. \"\n \"{} was provided\".format(type(groupBy)))\n\n new_index = self.opmng.merge(self.__index, groupBy)\n return GMQLDataset(index=new_index, location=self.location,\n local_sources=self._local_sources,\n remote_sources=self._remote_sources,\n meta_profile=self.meta_profile)", "response": "This method creates a new GMQLDataset that contains all the samples in the input dataset and the union of all the metadata attribute - values of the input samples."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngrouping the samples by meta and meta_aggregates.", "response": "def group(self, meta=None, meta_aggregates=None, regs=None,\n regs_aggregates=None, meta_group_name=\"_group\"):\n \"\"\"\n *Wrapper of* ``GROUP``\n\n The GROUP operator is used for grouping both regions and/or metadata of input\n dataset samples according to distinct values of certain attributes (known as grouping\n attributes); new grouping attributes are added to samples in the output dataset,\n storing the results of aggregate function evaluations over metadata and/or regions\n in each group of samples.\n Samples having missing values for any of the grouping attributes are discarded.\n\n\n :param meta: (optional) a list of metadata attributes\n :param meta_aggregates: (optional) {'new_attr': fun}\n :param regs: (optional) a list of region fields\n :param regs_aggregates: {'new_attr': fun}\n :param meta_group_name: (optional) the name to give to the group attribute in the\n metadata\n :return: a new GMQLDataset\n\n Example of usage. We group samples by `antibody` and we aggregate the region pvalues taking the maximum value\n calling the new region field `maxPvalue`::\n\n import gmql as gl\n\n d1 = gl.get_example_dataset(\"Example_Dataset_1\")\n result = d1.group(meta=['antibody'], regs_aggregates={'maxPvalue': gl.MAX(\"pvalue\")})\n \"\"\"\n\n if isinstance(meta, list) and \\\n all([isinstance(x, str) for x in meta]):\n meta = Some(meta)\n elif meta is None:\n meta = none()\n else:\n raise TypeError(\"meta must be a list of strings. \"\n \"{} was provided\".format(type(meta)))\n expBuild = self.pmg.getNewExpressionBuilder(self.__index)\n if isinstance(meta_aggregates, dict):\n metaAggregates = []\n for k in meta_aggregates:\n if isinstance(k, str):\n item = meta_aggregates[k]\n if isinstance(item, (SUM, MIN, MAX, AVG, BAG,\n BAGD, STD, MEDIAN, COUNTSAMP)):\n functionName = item.get_aggregate_name()\n argument = item.get_argument()\n if argument is None:\n argument = none()\n else:\n argument = Some(argument)\n metaAggregates.append(expBuild.createMetaAggregateFunction(functionName,\n k, argument))\n else:\n raise TypeError(\"the item of the dictionary must be an Aggregate of the following: \"\n \"SUM, MIN, MAX, AVG, BAG, BAGD, STD, COUNTSAMP. \"\n \"{} was provided\".format(type(item)))\n else:\n raise TypeError(\"keys of meta_aggregates must be string. \"\n \"{} was provided\".format(type(k)))\n metaAggregates = Some(metaAggregates)\n elif meta_aggregates is None:\n metaAggregates = none()\n else:\n raise TypeError(\"meta_aggregates must be a dictionary of Aggregate functions. \"\n \"{} was provided\".format(type(meta_aggregates)))\n if isinstance(regs, list) and \\\n all([isinstance(x, str) for x in regs]):\n regs = Some(regs)\n elif regs is None:\n regs = none()\n else:\n raise TypeError(\"regs must be a list of strings. \"\n \"{} was provided\".format(type(regs)))\n\n if isinstance(regs_aggregates, dict):\n regionAggregates = []\n for k in regs_aggregates.keys():\n if isinstance(k, str):\n item = regs_aggregates[k]\n if isinstance(item, (SUM, MIN, MAX, AVG, BAG, BAGD,\n MEDIAN, COUNT)):\n op_name = item.get_aggregate_name()\n op_argument = item.get_argument()\n if op_argument is None:\n op_argument = none()\n else:\n op_argument = Some(op_argument)\n regsToReg = expBuild.getRegionsToRegion(op_name, k, op_argument)\n regionAggregates.append(regsToReg)\n else:\n raise TypeError(\"the item of the dictionary must be an Aggregate of the following: \"\n \"SUM, MIN, MAX, AVG, BAG, BAGD, MEDIAN, COUNT. \"\n \"{} was provided\".format(type(item)))\n else:\n raise TypeError(\"The key of new_reg_fields must be a string. \"\n \"{} was provided\".format(type(k)))\n regionAggregates = Some(regionAggregates)\n elif regs_aggregates is None:\n regionAggregates = none()\n else:\n raise TypeError(\"new_reg_fields must be a list of dictionary. \"\n \"{} was provided\".format(type(regs_aggregates)))\n\n if isinstance(meta_group_name, str):\n pass\n else:\n raise TypeError(\"meta_group_name must be a string. \"\n \"{} was provided\".format(type(meta_group_name)))\n\n new_index = self.opmng.group(self.__index, meta, metaAggregates, meta_group_name, regs, regionAggregates)\n\n return GMQLDataset(index=new_index, location=self.location,\n local_sources=self._local_sources,\n remote_sources=self._remote_sources,\n meta_profile=self.meta_profile)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef meta_group(self, meta, meta_aggregates=None):\n return self.group(meta=meta, meta_aggregates=meta_aggregates)", "response": "Group only for metadata."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef regs_group(self, regs, regs_aggregates=None):\n return self.group(regs=regs, regs_aggregates=regs_aggregates)", "response": "*Wrapper of* ``GROUP``\n\n Group operation only for region data. For further information check :meth:`~.group`"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef materialize(self, output_path=None, output_name=None, all_load=True):\n current_mode = get_mode()\n new_index = self.__modify_dag(current_mode)\n if current_mode == 'local':\n return Materializations.materialize_local(new_index, output_path, all_load)\n elif current_mode == 'remote':\n return Materializations.materialize_remote(new_index, output_name, output_path, all_load)\n else:\n raise ValueError(\"Current mode is not defined. {} given\".format(current_mode))", "response": "Materialize the GMQL dataset."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef step(self):\n message = self.get_message()\n if message is None:\n\n self.sleep(self.idle_time)\n return None\n\n self.dispatch(message)\n # In non-greenthread environments this does nothing. In green-thread\n # environments this yields the context so messages can be acted upon\n # before exhausting the threadpool.\n self.sleep(0)", "response": "Grab a new message and dispatch it to the handler."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef command(self, command, token, team_id=None, methods=['GET'], **kwargs):\n if team_id is None:\n team_id = self.team_id\n if team_id is None:\n raise RuntimeError('TEAM_ID is not found in your configuration!')\n\n def deco(func):\n self._commands[(team_id, command)] = (func, token, methods, kwargs)\n return func\n return deco", "response": "A decorator to register a command."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef dispatch(self):\n from flask import request\n\n method = request.method\n\n data = request.args\n if method == 'POST':\n data = request.form\n\n token = data.get('token')\n team_id = data.get('team_id')\n command = data.get('command') or data.get('trigger_word')\n\n if isinstance(command, string_types):\n command = command.strip().lstrip('/')\n\n try:\n self.validate(command, token, team_id, method)\n except SlackError as e:\n return self.response(e.msg)\n\n func, _, _, kwargs = self._commands[(team_id, command)]\n kwargs.update(data.to_dict())\n\n return func(**kwargs)", "response": "Dispatch http request to registerd commands."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef validate(self, command, token, team_id, method):\n if (team_id, command) not in self._commands:\n raise SlackError('Command {0} is not found in team {1}'.format(\n command, team_id))\n\n func, _token, methods, kwargs = self._commands[(team_id, command)]\n\n if method not in methods:\n raise SlackError('{} request is not allowed'.format(method))\n\n if token != _token:\n raise SlackError('Your token {} is invalid'.format(token))", "response": "Validate request queries with registerd commands"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef response(self, text, response_type='ephemeral', attachments=None):\n from flask import jsonify\n if attachments is None:\n attachments = []\n\n data = {\n 'response_type': response_type,\n 'text': text,\n 'attachments': attachments,\n }\n return jsonify(**data)", "response": "Return a response with json format"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef pid(self):\n try:\n\n with open(self.pidfile, 'r') as pidfile:\n\n try:\n\n pid = int(pidfile.read().strip())\n\n except ValueError:\n\n return None\n\n try:\n\n os.kill(pid, 0)\n\n except OSError as e:\n\n if e.errno == errno.EPERM:\n\n return pid\n\n elif e.errno == errno.ESRCH:\n\n return None\n\n LOG.exception(\n \"os.kill returned unhandled error \"\n \"{0}\".format(e.strerror)\n )\n sys.exit(exit.PIDFILE_ERROR)\n\n return pid\n\n except IOError:\n\n if not os.path.isfile(self.pidfile):\n\n return None\n\n LOG.exception(\"Failed to read pidfile {0}.\".format(self.pidfile))\n sys.exit(exit.PIDFILE_INACCESSIBLE)", "response": "Get the pid which represents a daemonized process."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsets the pid for a running process.", "response": "def pid(self, pidnum):\n \"\"\"Set the pid for a running process.\"\"\"\n try:\n\n with open(self.pidfile, \"w+\") as pidfile:\n\n pidfile.write(\"{0}\\n\".format(pidnum))\n\n except IOError:\n\n LOG.exception(\"Failed to write pidfile {0}).\".format(self.pidfile))\n sys.exit(exit.PIDFILE_INACCESSIBLE)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pid(self):\n try:\n\n os.remove(self.pidfile)\n\n except IOError:\n\n if not os.path.isfile(self.pidfile):\n\n return None\n\n LOG.exception(\"Failed to clear pidfile {0}).\".format(self.pidfile))\n sys.exit(exit.PIDFILE_INACCESSIBLE)", "response": "Stop managing the current pid."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget an gevent pool used to dispatch requests.", "response": "def pool(self):\n \"\"\"Get an gevent pool used to dispatch requests.\"\"\"\n self._pool = self._pool or gevent.pool.Pool(size=self.pool_size)\n return self._pool"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set_essid(interface, essid):\n interface = _get_bytes(interface)\n essid = _get_bytes(essid)\n\n wrq = ffi.new('struct iwreq*')\n\n with iwlib_socket() as sock:\n if essid.lower() in (b'off', b'any'):\n wrq.u.essid.flags = 0\n essid = b''\n elif essid.lower() == b'on':\n buf = ffi.new('char []', iwlib.IW_ESSID_MAX_SIZE+1)\n wrq.u.essid.pointer = buf\n wrq.u.essid.length = iwlib.IW_ESSID_MAX_SIZE + 1\n wrq.u.essid.flags = 0\n if iwlib.iw_get_ext(sock, interface, iwlib.SIOCGIWESSID, wrq) < 0:\n raise ValueError(\"Error retrieving previous ESSID: %s\" % (os.strerror(ffi.errno)))\n wrq.u.essid.flags = 1\n elif len(essid) > iwlib.IW_ESSID_MAX_SIZE:\n raise ValueError(\"ESSID '%s' is longer than the maximum %d\" % (essid, iwlib.IW_ESSID_MAX_SIZE))\n else:\n wrq.u.essid.pointer = ffi.new_handle(essid)\n wrq.u.essid.length = len(essid)\n wrq.u.essid.flags = 1\n\n if iwlib.iw_get_kernel_we_version() < 21:\n wrq.u.essid.length += 1\n\n if iwlib.iw_set_ext(sock, interface, iwlib.SIOCSIWESSID, wrq) < 0:\n errno = ffi.errno\n strerror = \"Couldn't set essid on device '%s': %s\" % (interface.decode('utf8'), os.strerror(errno))\n raise OSError(errno, strerror)", "response": "Sets the ESSID of a given interface."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef read_adjacency_matrix(file_path, separator):\n # Open file\n file_row_generator = get_file_row_generator(file_path, separator)\n\n # Initialize lists for row and column sparse matrix arguments\n row = list()\n col = list()\n append_row = row.append\n append_col = col.append\n\n # Read all file rows\n for file_row in file_row_generator:\n source_node = np.int64(file_row[0])\n target_node = np.int64(file_row[1])\n\n # Add edge\n append_row(source_node)\n append_col(target_node)\n\n # Since this is an undirected network also add the reciprocal edge\n append_row(target_node)\n append_col(source_node)\n\n row = np.array(row, dtype=np.int64)\n col = np.array(col, dtype=np.int64)\n data = np.ones_like(row, dtype=np.float64)\n\n number_of_nodes = np.max(row) # I assume that there are no missing nodes at the end.\n\n # Array count should start from 0.\n row -= 1\n col -= 1\n\n # Form sparse adjacency matrix\n adjacency_matrix = sparse.coo_matrix((data, (row, col)), shape=(number_of_nodes, number_of_nodes))\n\n return adjacency_matrix", "response": "Reads an edge list in csv format and returns the adjacency matrix in SciPy Sparse COOrdinate format."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef read_node_label_matrix(file_path, separator, number_of_nodes):\n # Open file\n file_row_generator = get_file_row_generator(file_path, separator)\n\n # Initialize lists for row and column sparse matrix arguments\n row = list()\n col = list()\n append_row = row.append\n append_col = col.append\n\n # Populate the arrays\n for file_row in file_row_generator:\n node = np.int64(file_row[0])\n label = np.int64(file_row[1])\n\n # Add label\n append_row(node)\n append_col(label)\n\n number_of_categories = len(set(col)) # I assume that there are no missing labels. There may be missing nodes.\n labelled_node_indices = np.array(list(set(row)))\n\n row = np.array(row, dtype=np.int64)\n col = np.array(col, dtype=np.int64)\n data = np.ones_like(row, dtype=np.float64)\n\n # Array count should start from 0.\n row -= 1\n col -= 1\n labelled_node_indices -= 1\n\n # Form sparse adjacency matrix\n node_label_matrix = sparse.coo_matrix((data, (row, col)), shape=(number_of_nodes, number_of_categories))\n node_label_matrix = node_label_matrix.tocsr()\n\n return node_label_matrix, number_of_categories, labelled_node_indices", "response": "Reads node - label pairs in csv format and returns a list of tuples and a sparse adjacency matrix."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nensuring that the Garuda directory and files are created.", "response": "def ensure_data():\n '''\n Ensure that the Garuda directory and files\n '''\n if not os.path.exists(GARUDA_DIR):\n os.makedirs(GARUDA_DIR)\n Path(f'{GARUDA_DIR}/__init__.py').touch()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef protoc_arguments():\n '''\n Construct protobuf compiler arguments\n '''\n proto_include = resource_filename('grpc_tools', '_proto')\n return [\n protoc.__file__, '-I', GARUDA_DIR, f'--python_out={GARUDA_DIR}',\n f'--grpc_python_out={GARUDA_DIR}', GARUDA_PROTO_PATH,\n f'-I{proto_include}']", "response": "Construct protobuf compiler arguments"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef write_average_score_row(fp, score_name, scores):\n row = \"--\" + score_name + \"--\"\n fp.write(row)\n for vector in scores:\n row = list(vector)\n row = [str(score) for score in row]\n row = \"\\n\" + \"\\t\".join(row)\n fp.write(row)", "response": "Simple utility function that writes an average score row in a file designated by a file pointer."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncatches specific errors in admin actions and shows a friendly error.", "response": "def catch_errors(f):\n \"\"\"\n Catches specific errors in admin actions and shows a friendly error.\n \"\"\"\n\n @functools.wraps(f)\n def wrapper(self, request, *args, **kwargs):\n try:\n return f(self, request, *args, **kwargs)\n except exceptions.CertificateExpired:\n self.message_user(\n request,\n _('The AFIP Taxpayer certificate has expired.'),\n messages.ERROR,\n )\n except exceptions.UntrustedCertificate:\n self.message_user(\n request,\n _('The AFIP Taxpayer certificate is untrusted.'),\n messages.ERROR,\n )\n except exceptions.CorruptCertificate:\n self.message_user(\n request,\n _('The AFIP Taxpayer certificate is corrupt.'),\n messages.ERROR,\n )\n except exceptions.AuthenticationError as e:\n logger.exception('AFIP auth failed')\n self.message_user(\n request,\n _('An unknown authentication error has ocurred: %s') % e,\n messages.ERROR,\n )\n\n return wrapper"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef augmented_tf_idf(attribute_matrix):\n number_of_documents = attribute_matrix.shape[0]\n\n max_term_frequencies = np.ones(number_of_documents, dtype=np.float64)\n idf_array = np.ones(attribute_matrix.shape[1], dtype=np.float64)\n\n # Calculate inverse document frequency\n attribute_matrix = attribute_matrix.tocsc()\n for j in range(attribute_matrix.shape[1]):\n document_frequency = attribute_matrix.getcol(j).data.size\n if document_frequency > 1:\n idf_array[j] = np.log(number_of_documents/document_frequency)\n\n # Calculate maximum term frequencies for a user\n attribute_matrix = attribute_matrix.tocsr()\n for i in range(attribute_matrix.shape[0]):\n max_term_frequency = attribute_matrix.getrow(i).data\n if max_term_frequency.size > 0:\n max_term_frequency = max_term_frequency.max()\n if max_term_frequency > 0.0:\n max_term_frequencies[i] = max_term_frequency\n\n # Do augmented tf-idf normalization\n attribute_matrix = attribute_matrix.tocoo()\n attribute_matrix.data = 0.5 + np.divide(0.5*attribute_matrix.data, np.multiply((max_term_frequencies[attribute_matrix.row]), (idf_array[attribute_matrix.col])))\n attribute_matrix = attribute_matrix.tocsr()\n\n return attribute_matrix", "response": "Augmented TF - IDF normalization on a bag - of - words vector representation of data."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_file_row_generator(file_path, separator, encoding=None):\n with open(file_path, encoding=encoding) as file_object:\n for line in file_object:\n words = line.strip().split(separator)\n yield words", "response": "Reads an separated value file row by row."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nstores a pickle object to a given path.", "response": "def store_pickle(file_path, data):\n \"\"\"\n Pickle some data to a given path.\n\n Inputs: - file_path: Target file path.\n - data: The python object to be serialized via pickle.\n \"\"\"\n pkl_file = open(file_path, 'wb')\n pickle.dump(data, pkl_file)\n pkl_file.close()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef load_pickle(file_path):\n pkl_file = open(file_path, 'rb')\n data = pickle.load(pkl_file)\n pkl_file.close()\n return data", "response": "Load a pickle file from disk."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef makeServoIDPacket(curr_id, new_id):\n\tpkt = Packet.makeWritePacket(curr_id, xl320.XL320_ID, [new_id])\n\treturn pkt", "response": "Returns a packet to set the servo to a new ID"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef pagerank_limit_push(s, r, w_i, a_i, push_node, rho):\n # Calculate the A and B quantities to infinity\n A_inf = rho*r[push_node]\n B_inf = (1-rho)*r[push_node]\n\n # Update approximate Pagerank and residual vectors\n s[push_node] += A_inf\n r[push_node] = 0.0\n\n # Update residual vector at push node's adjacent nodes\n r[a_i] += B_inf * w_i", "response": "Calculates the approximate Pagerank and residual vectors at a given push node."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef pagerank_lazy_push(s, r, w_i, a_i, push_node, rho, lazy):\n # Calculate the A, B and C quantities\n A = rho*r[push_node]\n B = (1-rho)*(1 - lazy)*r[push_node]\n C = (1-rho)*lazy*(r[push_node])\n\n # Update approximate Pagerank and residual vectors\n s[push_node] += A\n r[push_node] = C\n\n # Update residual vector at push node's adjacent nodes\n r[a_i] += B * w_i", "response": "Calculates the approximate Pagerank and residual vectors at a given push node."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef cumulative_pagerank_difference_limit_push(s, r, w_i, a_i, push_node, rho):\n # Calculate the commute quantity\n commute = (1-rho)*r[push_node]\n\n # Update approximate regularized commute and residual vectors\n r[push_node] = 0.0\n\n # Update residual vector at push node's adjacent nodes\n commute_probabilities = commute * w_i\n s[a_i] += commute_probabilities\n r[a_i] += commute_probabilities", "response": "This function performs a random step without a self - loop."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nincluding pyramid_multiauth into a pyramid configurator.", "response": "def includeme(config):\n \"\"\"Include pyramid_multiauth into a pyramid configurator.\n\n This function provides a hook for pyramid to include the default settings\n for auth via pyramid_multiauth. Activate it like so:\n\n config.include(\"pyramid_multiauth\")\n\n This will pull the list of registered authn policies from the deployment\n settings, and configure and install each policy in order. The policies to\n use can be specified in one of two ways:\n\n * as the name of a module to be included.\n * as the name of a callable along with a set of parameters.\n\n Here's an example suite of settings:\n\n multiauth.policies = ipauth1 ipauth2 pyramid_browserid\n\n multiauth.policy.ipauth1.use = pyramid_ipauth.IPAuthentictionPolicy\n multiauth.policy.ipauth1.ipaddrs = 123.123.0.0/16\n multiauth.policy.ipauth1.userid = local1\n\n multiauth.policy.ipauth2.use = pyramid_ipauth.IPAuthentictionPolicy\n multiauth.policy.ipauth2.ipaddrs = 124.124.0.0/16\n multiauth.policy.ipauth2.userid = local2\n\n This will configure a MultiAuthenticationPolicy with three policy objects.\n The first two will be IPAuthenticationPolicy objects created by passing\n in the specified keyword arguments. The third will be a BrowserID\n authentication policy just like you would get from executing:\n\n config.include(\"pyramid_browserid\")\n\n As a side-effect, the configuration will also get the additional views\n that pyramid_browserid sets up by default.\n\n The *group finder function* and the *authorization policy* are also read\n from configuration if specified:\n\n multiauth.authorization_policy = mypyramidapp.acl.Custom\n multiauth.groupfinder = mypyramidapp.acl.groupfinder\n \"\"\"\n # Grab the pyramid-wide settings, to look for any auth config.\n settings = config.get_settings()\n # Hook up a default AuthorizationPolicy.\n # Get the authorization policy from config if present.\n # Default ACLAuthorizationPolicy is usually what you want.\n authz_class = settings.get(\"multiauth.authorization_policy\",\n \"pyramid.authorization.ACLAuthorizationPolicy\")\n authz_policy = config.maybe_dotted(authz_class)()\n # If the app configures one explicitly then this will get overridden.\n # In autocommit mode this needs to be done before setting the authn policy.\n config.set_authorization_policy(authz_policy)\n # Get the groupfinder from config if present.\n groupfinder = settings.get(\"multiauth.groupfinder\", None)\n groupfinder = config.maybe_dotted(groupfinder)\n # Look for callable policy definitions.\n # Suck them all out at once and store them in a dict for later use.\n policy_definitions = get_policy_definitions(settings)\n # Read and process the list of policies to load.\n # We build up a list of callables which can be executed at config commit\n # time to obtain the final list of policies.\n # Yeah, it's complicated. But we want to be able to inherit any default\n # views or other config added by the sub-policies when they're included.\n # Process policies in reverse order so that things at the front of the\n # list can override things at the back of the list.\n policy_factories = []\n policy_names = settings.get(\"multiauth.policies\", \"\").split()\n for policy_name in reversed(policy_names):\n if policy_name in policy_definitions:\n # It's a policy defined using a callable.\n # Just append it straight to the list.\n definition = policy_definitions[policy_name]\n factory = config.maybe_dotted(definition.pop(\"use\"))\n policy_factories.append((factory, policy_name, definition))\n else:\n # It's a module to be directly included.\n try:\n factory = policy_factory_from_module(config, policy_name)\n except ImportError:\n err = \"pyramid_multiauth: policy %r has no settings \"\\\n \"and is not importable\" % (policy_name,)\n raise ValueError(err)\n policy_factories.append((factory, policy_name, {}))\n # OK. We now have a list of callbacks which need to be called at\n # commit time, and will return the policies in reverse order.\n # Register a special action to pull them into our list of policies.\n policies = []\n\n def grab_policies():\n for factory, name, kwds in policy_factories:\n policy = factory(**kwds)\n if policy:\n policy._pyramid_multiauth_name = name\n if not policies or policy is not policies[0]:\n # Remember, they're being processed in reverse order.\n # So each new policy needs to go at the front.\n policies.insert(0, policy)\n\n config.action(None, grab_policies, order=PHASE2_CONFIG)\n authn_policy = MultiAuthenticationPolicy(policies, groupfinder)\n config.set_authentication_policy(authn_policy)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef policy_factory_from_module(config, module):\n # Remember the policy that's active before including the module, if any.\n orig_policy = config.registry.queryUtility(IAuthenticationPolicy)\n # Include the module, so we get any default views etc.\n config.include(module)\n # That might have registered and commited a new policy object.\n policy = config.registry.queryUtility(IAuthenticationPolicy)\n if policy is not None and policy is not orig_policy:\n return lambda: policy\n # Or it might have set up a pending action to register one later.\n # Find the most recent IAuthenticationPolicy action, and grab\n # out the registering function so we can call it ourselves.\n for action in reversed(config.action_state.actions):\n # Extract the discriminator and callable. This is complicated by\n # Pyramid 1.3 changing action from a tuple to a dict.\n try:\n discriminator = action[\"discriminator\"]\n callable = action[\"callable\"]\n except TypeError: # pragma: nocover\n discriminator = action[0] # pragma: nocover\n callable = action[1] # pragma: nocover\n # If it's not setting the authn policy, keep looking.\n if discriminator is not IAuthenticationPolicy:\n continue\n\n # Otherwise, wrap it up so we can extract the registered object.\n def grab_policy(register=callable):\n old_policy = config.registry.queryUtility(IAuthenticationPolicy)\n register()\n new_policy = config.registry.queryUtility(IAuthenticationPolicy)\n config.registry.registerUtility(old_policy, IAuthenticationPolicy)\n return new_policy\n\n return grab_policy\n # Or it might not have done *anything*.\n # So return a null policy factory.\n return lambda: None", "response": "Create a function that returns a function that will be called when the module is loaded."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_policy_definitions(settings):\n policy_definitions = {}\n for name in settings:\n if not name.startswith(\"multiauth.policy.\"):\n continue\n value = settings[name]\n name = name[len(\"multiauth.policy.\"):]\n policy_name, setting_name = name.split(\".\", 1)\n if policy_name not in policy_definitions:\n policy_definitions[policy_name] = {}\n policy_definitions[policy_name][setting_name] = value\n return policy_definitions", "response": "Find all multiauth policy definitions from the settings dict."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef load_file(filename):\n if filename in ('stdin', '-', None):\n instances = [i for i in gc.get_objects() if isinstance(i, pycodestyle.Checker) or isinstance(i, pep8.Checker)]\n if len(instances) != 1:\n raise ValueError('Expected only 1 instance of pycodestyle.Checker, got {0} instead.'.format(len(instances)))\n return 'stdin', ''.join(instances[0].lines)\n with codecs.open(filename, encoding='utf-8') as handle:\n return filename, handle.read()", "response": "Read file to memory."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns True if code should be ignored False otherwise.", "response": "def ignore(code):\n \"\"\"Should this code be ignored.\n\n :param str code: Error code (e.g. D201).\n\n :return: True if code should be ignored, False otherwise.\n :rtype: bool\n \"\"\"\n if code in Main.options['ignore']:\n return True\n if any(c in code for c in Main.options['ignore']):\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nadd options to flake8.", "response": "def add_options(cls, parser):\n \"\"\"Add options to flake8.\n\n :param parser: optparse.OptionParser from pycodestyle.\n \"\"\"\n parser.add_option('--show-pydocstyle', action='store_true', help='show explanation of each PEP 257 error')\n parser.config_options.append('show-pydocstyle')"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef parse_options(cls, options):\n # Handle flake8 options.\n cls.options['explain'] = bool(options.show_pydocstyle)\n cls.options['ignore'] = options.ignore\n\n # Handle pydocstyle options.\n config = pydocstyle.RawConfigParser()\n for file_name in pydocstyle.ConfigurationParser.PROJECT_CONFIG_FILES:\n if config.read(os.path.join(os.path.abspath('.'), file_name)):\n break\n if not config.has_section('pydocstyle'):\n return\n native_options = dict()\n for option in config.options('pydocstyle'):\n if option == 'ignore':\n native_options['ignore'] = config.get('pydocstyle', option)\n if option in ('explain', 'source'):\n native_options[option] = config.getboolean('pydocstyle', option)\n native_options['show-source'] = native_options.pop('source', None)\n if native_options.get('ignore'):\n native_options['ignore'] = native_options['ignore'].split(',')\n cls.options.update(dict((k, v) for k, v in native_options.items() if v))", "response": "Read parsed options from flake8."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nrun analysis on a single file.", "response": "def run(self):\n \"\"\"Run analysis on a single file.\"\"\"\n pydocstyle.Error.explain = self.options['explain']\n filename, source = load_file(self.filename)\n for error in pydocstyle.PEP257Checker().check_source(source, filename):\n if not hasattr(error, 'code') or ignore(error.code):\n continue\n lineno = error.line\n offset = 0 # Column number starting from 0.\n explanation = error.explanation if pydocstyle.Error.explain else ''\n text = '{0} {1}{2}'.format(error.code, error.message.split(': ', 1)[1], explanation)\n yield lineno, offset, text, Main"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the barcode s number without the verification digit.", "response": "def numbers(self):\n \"\"\"\"\n Returns the barcode's number without the verification digit.\n\n :return: list(int)\n \"\"\"\n numstring = '{:011d}{:02d}{:04d}{}{}'.format(\n self._receipt.point_of_sales.owner.cuit, # 11 digits\n int(self._receipt.receipt_type.code), # 2 digits\n self._receipt.point_of_sales.number, # point of sales\n self._receipt.validation.cae, # 14 digits\n self._receipt.validation.cae_expiration.strftime('%Y%m%d'), # 8\n )\n return [int(num) for num in numstring]"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the verification digit for a given list of numbers.", "response": "def verification_digit(numbers):\n \"\"\"\n Returns the verification digit for a given numbre.\n\n The verification digit is calculated as follows:\n\n * A = sum of all even-positioned numbers\n * B = A * 3\n * C = sum of all odd-positioned numbers\n * D = B + C\n * The results is the smallset number N, such that (D + N) % 10 == 0\n\n NOTE: Afip's documentation seems to have odd an even mixed up in the\n explanation, but all examples follow the above algorithm.\n\n :param list(int) numbers): The numbers for which the digits is to be\n calculated.\n :return: int\n \"\"\"\n a = sum(numbers[::2])\n b = a * 3\n c = sum(numbers[1::2])\n d = b + c\n e = d % 10\n if e == 0:\n return e\n return 10 - e"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef full_number(self):\n return '{}{}'.format(\n ''.join(str(n) for n in self.numbers),\n ReceiptBarcodeGenerator.verification_digit(self.numbers),\n )", "response": "Returns the full number including the verification digit."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreading an adjacency matrix in csv format and returns the adjacency matrix in SciPy Sparse COOrdinate format.", "response": "def read_adjacency_matrix(file_path, separator, numbering=\"matlab\"):\n \"\"\"\n Reads an edge list in csv format and returns the adjacency matrix in SciPy Sparse COOrdinate format.\n\n Inputs: - file_path: The path where the adjacency matrix is stored.\n - separator: The delimiter among values (e.g. \",\", \"\\t\", \" \")\n - numbering: Array numbering style: * \"matlab\"\n * \"c\"\n\n Outputs: - adjacency_matrix: The adjacency matrix in SciPy Sparse COOrdinate format.\n \"\"\"\n # Open file\n file_row_generator = get_file_row_generator(file_path, separator)\n\n file_row = next(file_row_generator)\n number_of_rows = file_row[1]\n number_of_columns = file_row[3]\n directed = file_row[7]\n if directed == \"True\":\n directed = True\n elif directed == \"False\":\n directed = False\n else:\n print(\"Invalid metadata.\")\n raise RuntimeError\n\n # Initialize lists for row and column sparse matrix arguments\n row = list()\n col = list()\n data = list()\n append_row = row.append\n append_col = col.append\n append_data = data.append\n\n # Read all file rows\n for file_row in file_row_generator:\n source_node = np.int64(file_row[0])\n target_node = np.int64(file_row[1])\n edge_weight = np.float64(file_row[2])\n\n # Add edge\n append_row(source_node)\n append_col(target_node)\n append_data(edge_weight)\n\n # Since this is an undirected network also add the reciprocal edge\n if not directed:\n if source_node != target_node:\n append_row(target_node)\n append_col(source_node)\n append_data(edge_weight)\n\n row = np.array(row, dtype=np.int64)\n col = np.array(col, dtype=np.int64)\n data = np.array(data, dtype=np.float64)\n\n if numbering == \"matlab\":\n row -= 1\n col -= 1\n elif numbering == \"c\":\n pass\n else:\n print(\"Invalid numbering style.\")\n raise RuntimeError\n\n # Form sparse adjacency matrix\n adjacency_matrix = spsp.coo_matrix((data, (row, col)), shape=(number_of_rows, number_of_columns))\n\n return adjacency_matrix"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nread node - label pairs in csv format and returns a list of tuples and a matrix of node - label pairs.", "response": "def read_node_label_matrix(file_path, separator, numbering=\"matlab\"):\n \"\"\"\n Reads node-label pairs in csv format and returns a list of tuples and a node-label matrix.\n\n Inputs: - file_path: The path where the node-label matrix is stored.\n - separator: The delimiter among values (e.g. \",\", \"\\t\", \" \")\n - number_of_nodes: The number of nodes of the full graph. It is possible that not all nodes are labelled.\n - numbering: Array numbering style: * \"matlab\"\n * \"c\"\n\n Outputs: - node_label_matrix: The node-label associations in a NumPy array of tuples format.\n - number_of_categories: The number of categories/classes the nodes may belong to.\n - labelled_node_indices: A NumPy array containing the labelled node indices.\n \"\"\"\n # Open file\n file_row_generator = get_file_row_generator(file_path, separator)\n\n file_row = next(file_row_generator)\n number_of_rows = file_row[1]\n number_of_categories = int(file_row[3])\n\n # Initialize lists for row and column sparse matrix arguments\n row = list()\n col = list()\n append_row = row.append\n append_col = col.append\n\n # Populate the arrays\n for file_row in file_row_generator:\n node = np.int64(file_row[0])\n label = np.int64(file_row[1])\n\n # Add label\n append_row(node)\n append_col(label)\n\n labelled_node_indices = np.array(list(set(row)))\n\n row = np.array(row, dtype=np.int64)\n col = np.array(col, dtype=np.int64)\n data = np.ones_like(row, dtype=np.float64)\n\n if numbering == \"matlab\":\n row -= 1\n col -= 1\n labelled_node_indices -= 1\n elif numbering == \"c\":\n pass\n else:\n print(\"Invalid numbering style.\")\n raise RuntimeError\n\n # Form sparse adjacency matrix\n node_label_matrix = spsp.coo_matrix((data, (row, col)), shape=(number_of_rows, number_of_categories))\n node_label_matrix = node_label_matrix.tocsr()\n\n return node_label_matrix, number_of_categories, labelled_node_indices"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef write_screen_name_to_topics(filepath, user_label_matrix, node_to_id, id_to_name, label_to_lemma, lemma_to_keyword, separator=\",\"):\n user_label_matrix = spsp.coo_matrix(user_label_matrix)\n\n shape = user_label_matrix.shape\n nnz = user_label_matrix.getnnz()\n\n row = user_label_matrix.row\n col = user_label_matrix.col\n data = user_label_matrix.data\n\n name_to_topic_set = defaultdict(set)\n\n for edge in range(row.size):\n node = row[edge]\n user_twitter_id = node_to_id[node]\n name = id_to_name[user_twitter_id]\n\n label = col[edge]\n lemma = label_to_lemma[label]\n # topic = lemma_to_keyword[lemma]\n\n name_to_topic_set[name].add(lemma)\n\n with open(filepath, \"w\") as f:\n # Write metadata.\n file_row = \"n_rows:\" + separator + str(shape[0]) + separator +\\\n \"nnz:\" + separator + str(nnz) + separator +\\\n \"\\n\"\n f.write(file_row)\n\n for name, topic_set in name_to_topic_set.items():\n file_row = list()\n file_row.append(name)\n file_row.extend(topic_set)\n file_row = separator.join(file_row) + \"\\n\"\n f.write(file_row)", "response": "Writes a user name and associated topic names per row."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef packetToDict(pkt):\n\n\td = {\n\t\t'id': pkt[4],\n\t\t# 'instruction': xl320.InstrToStr[pkt[7]],\n\t\t# 'length': (pkt[6] << 8) + pkt[5],\n\t\t# 'params': pkt[8:-2],\n\t\t'Model Number': (pkt[10] << 8) + pkt[9],\n\t\t'Firmware Ver': pkt[11],\n\t\t'Error': ErrorStatusMsg[pkt[8]],\n\t\t# 'crc': pkt[-2:]\n\t}\n\n\treturn d", "response": "This function turns a packet into a dictionary."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef sweep(port, rate, ID, retry=3):\n\tif port == 'dummy':\n\t\ts = ServoSerial(port, rate, fake=True)\n\telse:\n\t\ts = ServoSerial(port, rate)\n\n\tif ID < 0:\n\t\tID = xl320.XL320_BROADCAST_ADDR\n\n\ttry:\n\t\ts.open()\n\texcept SerialException as e:\n\t\t# print('Error opening serial port:')\n\t\tprint('-'*40)\n\t\tprint(sys.argv[0], ':')\n\t\tprint(e)\n\t\texit(1)\n\n\tpkt = makePingPacket(ID)\n\t# print('ping', pkt)\n\ts.write(pkt)\n\n\t# as more servos add up, I might need to increase the cnt number???\n\tfor cnt in range(retry):\n\t\tans = s.read()\n\n\t\tif ans:\n\t\t\tfor pkt in ans:\n\t\t\t\tservo = packetToDict(pkt)\n\t\t\t\tutils.prettyPrintPacket(servo)\n\t\t\t\tprint('raw pkt: {}'.format(pkt))\n\t\telse:\n\t\t\tprint('Try {}: no servos found'.format(cnt))\n\n\t\ttime.sleep(0.1)\n\n\ts.close()", "response": "Send a ping packet to the specified ID and returns any returned values."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef calculate_epsilon_effective(rho, epsilon, seed_degree, neighbor_degrees, mean_degree):\n # Calculate a weighted neighborhood degree average.\n # neighborhood_degree = rho*seed_degree + (1-rho)*neighbor_degrees.mean()\n neighborhood_degree = neighbor_degrees.mean()\n\n # Calculate the seed neighborhood normalized effective epsilon.\n epsilon_effective = (epsilon*np.log(1 + seed_degree))/np.log(1 + neighborhood_degree)\n\n # Calculate the maximum epsilon for at least one push on a neighboring node.\n # Also the minimum epsilon for a push on all the neighboring nodes.\n epsilon_effective_maximum = np.max(1/(seed_degree*neighbor_degrees))\n epsilon_effective_minimum = np.min(1/(seed_degree*neighbor_degrees))\n\n # print(epsilon_effective, epsilon_effective_maximum, epsilon_effective_minimum)\n\n # The maximum epsilon is absolute, whereas we regularize for the minimum.\n if epsilon_effective > epsilon_effective_maximum:\n epsilon_effective = epsilon_effective_maximum\n elif epsilon_effective < epsilon_effective_minimum:\n epsilon_effective = (epsilon_effective_minimum + epsilon_effective)/2\n\n return epsilon_effective", "response": "Calculates the effective epsilon threshold for a single node."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef arcte_with_lazy_pagerank(adjacency_matrix, rho, epsilon, number_of_threads=None):\n adjacency_matrix = sparse.csr_matrix(adjacency_matrix)\n number_of_nodes = adjacency_matrix.shape[0]\n\n if number_of_threads is None:\n number_of_threads = get_threads_number()\n if number_of_threads == 1:\n # Calculate natural random walk transition probability matrix.\n rw_transition, out_degree, in_degree = get_natural_random_walk_matrix(adjacency_matrix, make_shared=False)\n\n a = adjacency_matrix.copy()\n a.data = np.ones_like(a.data)\n edge_count_vector = np.squeeze(np.asarray(a.sum(axis=0), dtype=np.int64))\n\n iterate_nodes = np.where(edge_count_vector != 0)[0]\n argsort_indices = np.argsort(edge_count_vector[iterate_nodes])\n iterate_nodes = iterate_nodes[argsort_indices][::-1]\n iterate_nodes = iterate_nodes[np.where(edge_count_vector[iterate_nodes] > 1.0)[0]]\n\n # iterate_nodes = np.where(out_degree != 0)[0]\n # argsort_indices = np.argsort(out_degree[iterate_nodes])\n # iterate_nodes = iterate_nodes[argsort_indices][::-1]\n # iterate_nodes = iterate_nodes[np.where(out_degree[iterate_nodes] > 1.0)[0]]\n\n local_features = arcte_with_lazy_pagerank_worker(iterate_nodes,\n rw_transition.indices,\n rw_transition.indptr,\n rw_transition.data,\n out_degree,\n in_degree,\n rho,\n epsilon)\n else:\n # Calculate natural random walk transition probability matrix.\n rw_transition, out_degree, in_degree = get_natural_random_walk_matrix(adjacency_matrix, make_shared=True)\n\n a = adjacency_matrix.copy()\n a.data = np.ones_like(a.data)\n edge_count_vector = np.squeeze(np.asarray(a.sum(axis=0), dtype=np.int64))\n\n iterate_nodes = np.where(edge_count_vector != 0)[0]\n argsort_indices = np.argsort(edge_count_vector[iterate_nodes])\n iterate_nodes = iterate_nodes[argsort_indices][::-1]\n iterate_nodes = iterate_nodes[np.where(edge_count_vector[iterate_nodes] > 1.0)[0]]\n\n # iterate_nodes = np.where(out_degree != 0)[0]\n # argsort_indices = np.argsort(out_degree[iterate_nodes])\n # iterate_nodes = iterate_nodes[argsort_indices][::-1]\n # iterate_nodes = iterate_nodes[np.where(out_degree[iterate_nodes] > 1.0)[0]]\n\n pool = mp.Pool(number_of_threads)\n node_chunks = list(parallel_chunks(iterate_nodes, number_of_threads))\n node_count = 0\n for chunk in node_chunks:\n node_count += len(list(chunk))\n results = list()\n for chunk_no in range(len(pool._pool)):\n pool.apply_async(arcte_with_lazy_pagerank_worker,\n args=(node_chunks[chunk_no],\n rw_transition.indices,\n rw_transition.indptr,\n rw_transition.data,\n out_degree,\n in_degree,\n rho,\n epsilon),\n callback=results.append)\n pool.close()\n pool.join()\n # local_features = sparse.hstack(results)\n local_features = results[0]\n for additive_features in results[1:]:\n local_features += additive_features\n local_features = sparse.csr_matrix(local_features)\n\n # Form base community feature matrix.\n identity_matrix = sparse.csr_matrix(sparse.eye(number_of_nodes, number_of_nodes, dtype=np.float64))\n adjacency_matrix_ones = adjacency_matrix\n adjacency_matrix_ones.data = np.ones_like(adjacency_matrix.data)\n base_community_features = identity_matrix + adjacency_matrix_ones\n\n # Stack horizontally matrices to form feature matrix.\n try:\n features = sparse.hstack([base_community_features, local_features]).tocsr()\n except ValueError as e:\n print(\"Failure with horizontal feature stacking.\")\n features = base_community_features\n\n return features", "response": "This function extracts local community features for all graph nodes based on the partitioning of node - centric similarity vectors."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_folds_generator(node_label_matrix,\n labelled_node_indices,\n number_of_categories,\n dataset_memory_folder,\n percentage,\n number_of_folds=10):\n \"\"\"\n Read or form and store the seed nodes for training and testing.\n\n Inputs: - node_label_matrix: The node-label ground truth in a SciPy sparse matrix format.\n - labelled_node_indices: A NumPy array containing the labelled node indices.\n - number_of_categories: The number of categories/classes in the learning.\n - memory_path: The folder where the results are stored.\n - percentage: The percentage of labelled samples that will be used for training.\n\n Output: - folds: A generator containing train/test set folds.\n \"\"\"\n number_of_labeled_nodes = labelled_node_indices.size\n training_set_size = int(np.ceil(percentage*number_of_labeled_nodes/100))\n\n ####################################################################################################################\n # Read or generate folds\n ####################################################################################################################\n fold_file_path = dataset_memory_folder + \"/folds/\" + str(percentage) + \"_folds.txt\"\n train_list = list()\n test_list = list()\n if not os.path.exists(fold_file_path):\n with open(fold_file_path, \"w\") as fp:\n for trial in np.arange(number_of_folds):\n train, test = valid_train_test(node_label_matrix[labelled_node_indices, :],\n training_set_size,\n number_of_categories,\n trial)\n train = labelled_node_indices[train]\n test = labelled_node_indices[test]\n\n # Write test nodes\n row = [str(node) for node in test]\n row = \"\\t\".join(row) + \"\\n\"\n fp.write(row)\n\n # Write train nodes\n row = [str(node) for node in train]\n row = \"\\t\".join(row) + \"\\n\"\n fp.write(row)\n\n train_list.append(train)\n test_list.append(test)\n else:\n file_row_gen = get_file_row_generator(fold_file_path, \"\\t\")\n\n for trial in np.arange(number_of_folds):\n # Read test nodes\n test = next(file_row_gen)\n test = [int(node) for node in test]\n test = np.array(test)\n\n # Read train nodes\n train = next(file_row_gen)\n train = [int(node) for node in train]\n train = np.array(train)\n\n train_list.append(train)\n test_list.append(test)\n\n folds = ((train, test) for train, test in zip(train_list, test_list))\n return folds", "response": "Returns a generator that generates train and test folds for training and testing."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef generate_folds(node_label_matrix, labelled_node_indices, number_of_categories, percentage, number_of_folds=10):\n number_of_labeled_nodes = labelled_node_indices.size\n training_set_size = int(np.ceil(percentage*number_of_labeled_nodes/100))\n\n ####################################################################################################################\n # Generate folds\n ####################################################################################################################\n train_list = list()\n test_list = list()\n for trial in np.arange(number_of_folds):\n train, test = valid_train_test(node_label_matrix[labelled_node_indices, :],\n training_set_size,\n number_of_categories,\n trial)\n train = labelled_node_indices[train]\n test = labelled_node_indices[test]\n\n train_list.append(train)\n test_list.append(test)\n\n folds = ((train, test) for train, test in zip(train_list, test_list))\n return folds", "response": "Generates the train and test set folds for training and testing."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef iterative_stratification(node_label_matrix, training_set_size, number_of_categories, random_seed=0):\n number_of_labelled_nodes = node_label_matrix.shape[0]\n testing_set_size = number_of_labelled_nodes - training_set_size\n training_set_proportion = training_set_size/number_of_labelled_nodes\n testing_set_proportion = testing_set_size/number_of_labelled_nodes\n\n # Calculate the desired number of examples of each label at each subset.\n desired_label_number = np.zeros((2, number_of_categories), dtype=np.int64)\n node_label_matrix = node_label_matrix.tocsc()\n for j in range(number_of_categories):\n category_label_number = node_label_matrix.getcol(j).indices.size\n desired_label_number[0, j] = math.ceil(category_label_number*training_set_proportion)\n desired_label_number[1, j] = category_label_number - desired_label_number[0, j]\n\n train_ids = list()\n test_ids = list()\n\n append_train_id = train_ids.append\n append_test_id = test_ids.append\n\n # Randomize process\n np.random.seed(random_seed)\n while True:\n if len(train_ids) + len(test_ids) >= number_of_labelled_nodes:\n break\n # Find the label with the fewest (but at least one) remaining examples, breaking the ties randomly\n remaining_label_distribution = desired_label_number.sum(axis=0)\n min_label = np.min(remaining_label_distribution[np.where(remaining_label_distribution > 0)[0]])\n label_indices = np.where(remaining_label_distribution == min_label)[0]\n chosen_label = int(np.random.choice(label_indices, 1)[0])\n\n # Find the subset with the largest number of desired examples for this label,\n # breaking ties by considering the largest number of desired examples, breaking further ties randomly.\n fold_max_remaining_labels = np.max(desired_label_number[:, chosen_label])\n fold_indices = np.where(desired_label_number[:, chosen_label] == fold_max_remaining_labels)[0]\n chosen_fold = int(np.random.choice(fold_indices, 1)[0])\n\n # Choose a random example for the selected label.\n relevant_nodes = node_label_matrix.getcol(chosen_label).indices\n chosen_node = int(np.random.choice(np.setdiff1d(relevant_nodes,\n np.union1d(np.array(train_ids),\n np.array(test_ids))),\n 1)[0])\n if chosen_fold == 0:\n append_train_id(chosen_node)\n desired_label_number[0, node_label_matrix.getrow(chosen_node).indices] -= 1\n elif chosen_fold == 1:\n append_test_id(chosen_node)\n desired_label_number[1, node_label_matrix.getrow(chosen_node).indices] -= 1\n else:\n raise RuntimeError\n\n return np.array(train_ids), np.array(test_ids)", "response": "This function generates an iterative stratification of the data folds."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngiving the discriminator distances y_pred and y_test this function forms the node - label prediction matrix.", "response": "def form_node_label_prediction_matrix(y_pred, y_test):\n \"\"\"\n Given the discriminator distances, this function forms the node-label prediction matrix.\n\n It is assumed that the number of true labels is known.\n\n Inputs: - y_pred: A NumPy array that contains the distance from the discriminator for each label for each user.\n - y_test: The node-label ground truth for the test set in a SciPy sparse CSR matrix format.\n\n Outputs: - y_pred: The node-label prediction for the test set in a SciPy sparse CSR matrix format.\n \"\"\"\n number_of_test_nodes = y_pred.shape[0]\n\n # We calculate the number of true labels for each node.\n true_number_of_labels = np.squeeze(y_test.sum(axis=1))\n\n # We sort the prediction array for each node.\n index = np.argsort(y_pred, axis=1)\n\n row = np.empty(y_test.getnnz(), dtype=np.int64)\n col = np.empty(y_test.getnnz(), dtype=np.int64)\n start = 0\n for n in np.arange(number_of_test_nodes):\n end = start + true_number_of_labels[0, n]\n row[start:end] = n\n col[start:end] = index[n, -1:-true_number_of_labels[0, n]-1:-1]\n start = end\n data = np.ones_like(row, dtype=np.int8)\n\n y_pred = sparse.coo_matrix((data, (row, col)), shape=y_test.shape)\n\n return y_pred"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef calculate_measures(y_pred, y_test):\n y_pred = y_pred.toarray()\n y_test = y_test.toarray()\n\n macro_precision, macro_recall, macro_F1, macro_support = precision_recall_fscore_support(y_test,\n y_pred,\n beta=1.0,\n average=\"macro\")\n\n micro_precision, micro_recall, micro_F1, micro_support = precision_recall_fscore_support(y_test,\n y_pred,\n beta=1.0,\n average=\"micro\")\n\n F1 = f1_score(y_test,\n y_pred,\n average=None)\n\n measures = [macro_recall, micro_recall, macro_precision, micro_precision, macro_F1, micro_F1, F1]\n return measures", "response": "Calculates the F - scores and F - score averages given a classification result and ground truth."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef decode(buff):\n\t\tpp = list(map(ord, buff))\n\t\tif 0 == len(pp) == 1:\n\t\t\tpp = []\n\t\treturn pp", "response": "Transforms the raw buffer data read in into a list of bytes\n\t"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef read(self, how_much=128): # FIXME: 128 might be too much ... what is largest?\n\t\t# ret = self.readPkts(how_much)\n\t\t# return ret\n\t\tret = []\n\t\tself.setRTS(self.DD_READ)\n\n\t\t# this in_waiting is wrong ... i can read more than is there\n\t\t# if self.serial.in_waiting < 12:\n\t\t# \ttime.sleep(0.001)\n\t\t# \tprint('waiting:', self.serial.in_waiting)\n\t\t# else:\n\t\t# \tprint('waiting:', self.serial.in_waiting)\n\n\t\tdata = self.serial.read(how_much)\n\t\t# print('read() data', data, 'len(data)', len(data))\n\t\tif data:\n\t\t\tdata = self.decode(data)\n\t\t\t# print('decode', data)\n\t\t\tret = Packet.findPkt(data)\n\t\t\t# print('ret', ret)\n\t\treturn ret", "response": "This function reads in data from the RTS pin and returns a list of valid packets."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef write(self, pkt):\n\t\tself.setRTS(self.DD_WRITE)\n\t\tself.flushInput()\n\t\t# prep data array for transmition\n\t\tpkt = bytearray(pkt)\n\t\tpkt = bytes(pkt)\n\n\t\tnum = self.serial.write(pkt)\n\t\t# self.serial.flush()\n\t\t# print('wrote {} of len(pkt) = {}'.format(num, len(pkt)))\n\t\treturn num", "response": "This is a simple serial write command. It writes a packet to the serial port and returns the number of bytes written."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef sendPkt(self, pkt, retry=5, sleep_time=0.01):\n\t\tfor cnt in range(retry):\n\t\t\tself.serial.flushInput()\n\t\t\tself.write(pkt) # send packet to servo\n\t\t\tans = self.read() # get return status packet\n\n\t\t\tif ans:\n\t\t\t\t# check for error and resend\n\t\t\t\treturn ans\n\n\t\t\telse:\n\t\t\t\t# print('>> retry {} <<'.format(cnt))\n\t\t\t\ttime.sleep(sleep_time)\n\n\t\treturn None", "response": "sends a packet and waits for a return."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef scipy_sparse_to_csv(filepath, matrix, separator=\",\", directed=False, numbering=\"matlab\"):\n matrix = spsp.coo_matrix(matrix)\n\n shape = matrix.shape\n nnz = matrix.getnnz()\n\n if numbering == \"matlab\":\n row = matrix.row + 1\n col = matrix.col + 1\n data = matrix.data\n elif numbering == \"c\":\n row = matrix.row\n col = matrix.col\n data = matrix.data\n else:\n print(\"Invalid numbering style.\")\n raise RuntimeError\n\n with open(filepath, \"w\") as f:\n # Write metadata.\n file_row = \"n_rows:\" + separator + str(shape[0]) + separator +\\\n \"n_cols:\" + separator + str(shape[1]) + separator +\\\n \"nnz:\" + separator + str(nnz) + separator +\\\n \"directed:\" + separator + str(directed) +\\\n \"\\n\"\n f.write(file_row)\n\n for edge in range(row.size):\n if directed is False:\n if col[edge] < row[edge]:\n continue\n file_row = str(row[edge]) + separator + str(col[edge]) + separator + str(data[edge]) + \"\\n\"\n f.write(file_row)", "response": "Writes sparse matrix in separated value format."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef read_adjacency_matrix(file_path, separator, undirected):\n # Open file\n file_row_generator = get_file_row_generator(file_path, separator)\n\n # Initialize lists for row and column sparse matrix arguments.\n row = list()\n col = list()\n data = list()\n append_row = row.append\n append_col = col.append\n append_data = data.append\n\n # Initialize node anonymizer.\n id_to_node = dict()\n\n # Read all file rows.\n for file_row in file_row_generator:\n if file_row[0][0] == \"#\":\n continue\n\n source_node_id = int(file_row[0])\n target_node_id = int(file_row[1])\n\n number_of_nodes = len(id_to_node)\n source_node = id_to_node.setdefault(source_node_id,\n number_of_nodes)\n number_of_nodes = len(id_to_node)\n target_node = id_to_node.setdefault(target_node_id,\n number_of_nodes)\n\n edge_weight = float(file_row[2])\n\n # Add edge.\n append_row(source_node)\n append_col(target_node)\n append_data(edge_weight)\n\n # Since this is an undirected network also add the reciprocal edge.\n if undirected:\n if source_node != target_node:\n append_row(target_node)\n append_col(source_node)\n append_data(edge_weight)\n\n number_of_nodes = len(id_to_node)\n node_to_id = dict(zip(list(id_to_node.values()),\n list(id_to_node.keys())))\n\n row = np.array(row, dtype=np.int64)\n col = np.array(col, dtype=np.int64)\n data = np.array(data, dtype=np.float64)\n\n # Form sparse adjacency matrix.\n adjacency_matrix = spsp.coo_matrix((data, (row, col)), shape=(number_of_nodes,\n number_of_nodes))\n\n return adjacency_matrix, node_to_id", "response": "Reads an adjacency matrix in csv format and returns the adjacency matrix."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef publish(build):\n build.packages.install(\"wheel\")\n build.packages.install(\"twine\")\n build.executables.run([\n \"python\", \"setup.py\",\n \"sdist\", \"bdist_wheel\", \"--universal\", \"--release\"\n ])\n build.executables.run([\n \"twine\", \"upload\", \"dist/*\"\n ])", "response": "publish the package itself"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncalculating the approximate personalized PageRank starting from a seed node without self - loops.", "response": "def fast_approximate_personalized_pagerank(s,\n r,\n w_i,\n a_i,\n out_degree,\n in_degree,\n seed_node,\n rho=0.2,\n epsilon=0.00001):\n \"\"\"\n Calculates the approximate personalized PageRank starting from a seed node without self-loops.\n \"\"\"\n # Initialize approximate PageRank and residual distributions\n # s = np.zeros(number_of_nodes, dtype=np.float64)\n # r = np.zeros(number_of_nodes, dtype=np.float64)\n r[seed_node] = 1.0\n\n # Initialize queue of nodes to be pushed\n pushable = deque()\n pushable.append(seed_node)\n\n # Do one push anyway\n push_node = pushable.popleft()\n\n pagerank_limit_push(s,\n r,\n w_i[push_node],\n a_i[push_node],\n push_node,\n rho)\n number_of_push_operations = 1\n\n i = np.where(np.divide(r[a_i[push_node]], in_degree[a_i[push_node]]) >= epsilon)[0]\n if i.size > 0:\n pushable.extend(a_i[push_node][i])\n\n while len(pushable) > 0:\n # While there are nodes with large residual probabilities, push\n push_node = pushable.popleft()\n if r[push_node]/in_degree[push_node] >= epsilon:\n pagerank_limit_push(s,\n r,\n w_i[push_node],\n a_i[push_node],\n push_node,\n rho)\n number_of_push_operations += 1\n\n i = np.where(np.divide(r[a_i[push_node]], in_degree[a_i[push_node]]) >= epsilon)[0]\n if i.size > 0:\n pushable.extend(a_i[push_node][i])\n\n return number_of_push_operations"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating the approximate personalized PageRank starting from a seed node with self - loops.", "response": "def lazy_approximate_personalized_pagerank(s,\n r,\n w_i,\n a_i,\n out_degree,\n in_degree,\n seed_node,\n rho=0.2,\n epsilon=0.00001,\n laziness_factor=0.5):\n \"\"\"\n Calculates the approximate personalized PageRank starting from a seed node with self-loops.\n\n Introduced in: Andersen, R., Chung, F., & Lang, K. (2006, October).\n Local graph partitioning using pagerank vectors.\n In Foundations of Computer Science, 2006. FOCS'06. 47th Annual IEEE Symposium on (pp. 475-486). IEEE.\n \"\"\"\n # Initialize approximate PageRank and residual distributions\n # s = np.zeros(number_of_nodes, dtype=np.float64)\n # r = np.zeros(number_of_nodes, dtype=np.float64)\n r[seed_node] = 1.0\n\n # Initialize queue of nodes to be pushed\n pushable = deque()\n pushable.append(seed_node)\n\n # Do one push anyway\n push_node = pushable.popleft()\n\n pagerank_lazy_push(s,\n r,\n w_i[push_node],\n a_i[push_node],\n push_node,\n rho,\n laziness_factor)\n number_of_push_operations = 1\n\n i = np.where(np.divide(r[a_i[push_node]], in_degree[a_i[push_node]]) >= epsilon)[0]\n if i.size > 0:\n pushable.extend(a_i[push_node][i])\n\n while r[push_node]/in_degree[push_node] >= epsilon:\n pagerank_lazy_push(s,\n r,\n w_i[push_node],\n a_i[push_node],\n push_node,\n rho,\n laziness_factor)\n number_of_push_operations += 1\n\n # While there are nodes with large residual probabilities, push\n while len(pushable) > 0:\n push_node = pushable.popleft()\n\n if r[push_node]/in_degree[push_node] >= epsilon:\n pagerank_lazy_push(s,\n r,\n w_i[push_node],\n a_i[push_node],\n push_node,\n rho,\n laziness_factor)\n number_of_push_operations += 1\n\n i = np.where(np.divide(r[a_i[push_node]], in_degree[a_i[push_node]]) >= epsilon)[0]\n if i.size > 0:\n pushable.extend(a_i[push_node][i])\n\n while r[push_node]/in_degree[push_node] >= epsilon:\n pagerank_lazy_push(s,\n r,\n w_i[push_node],\n a_i[push_node],\n push_node,\n rho,\n laziness_factor)\n number_of_push_operations += 1\n\n return number_of_push_operations"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef read_matlab_features(array_paths, number_of_nodes, dimensionality):\n # Read the data array\n file_row_gen = get_file_row_generator(array_paths[0], \"\\t\")\n data = list()\n append_data = data.append\n for file_row in file_row_gen:\n append_data(float(file_row[0]))\n\n # Read the row array\n file_row_gen = get_file_row_generator(array_paths[1], \"\\t\")\n row = list()\n append_row = row.append\n for file_row in file_row_gen:\n append_row(int(float(file_row[0])))\n\n # Read the data array\n file_row_gen = get_file_row_generator(array_paths[2], \"\\t\")\n col = list()\n append_col = col.append\n for file_row in file_row_gen:\n append_col(int(float(file_row[0])))\n\n data = np.array(data).astype(np.float64)\n row = np.array(row).astype(np.int64) - 1 # Due to Matlab numbering\n col = np.array(col).astype(np.int64) - 1 # Due to Matlab numbering\n\n print(np.max(row), np.min(row))\n print(np.max(col), np.min(col))\n\n # centroids_new = sparse.coo_matrix((data, (row, col)), shape=(number_of_nodes + 1, k))\n features = spsp.coo_matrix((data, (row, col)), shape=(number_of_nodes, dimensionality))\n\n return features", "response": "Reads the Matlab feature matrix from the given array of files."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef mroc(adjacency_matrix, alpha):\n # Find number of nodes\n number_of_nodes = adjacency_matrix.shape[0]\n\n ####################################################################################################################\n # Base community calculation\n ####################################################################################################################\n # Initialize empty lists\n base_list = list()\n base_row = list()\n base_col = list()\n\n # Save function handles for speed\n append_base_list = base_list.append\n append_base_row = base_row.append\n append_base_col = base_col.append\n\n # Find base communities\n adjacency_matrix = adjacency_matrix.tocsc()\n number_of_base_communities = 0\n for i in range(number_of_nodes):\n # Calculate base community\n base_community = set(adjacency_matrix.getcol(i).indices)\n base_community.add(i)\n flag = True\n for c in base_list:\n if c == base_community:\n flag = False\n break\n if flag:\n append_base_list(base_community)\n\n for n in base_community:\n append_base_row(n)\n append_base_col(number_of_base_communities)\n\n number_of_base_communities += 1\n\n # Form sparse matrices\n base_row = np.array(base_row)\n base_col = np.array(base_col)\n base_data = np.ones(base_row.size, dtype=np.float64)\n features = sparse.coo_matrix((base_data, (base_row, base_col)),\n shape=(number_of_nodes, number_of_base_communities))\n\n features = features.tocsr()\n base_community_number = features.shape[1]\n\n print('Base communities calculated.')\n\n reverse_index_csr = copy.copy(features)\n reverse_index_csc = reverse_index_csr.tocsc()\n reverse_index_csr = reverse_index_csr.tocsr()\n\n reverse_index_rows = np.ndarray(number_of_nodes, dtype=np.ndarray)\n reverse_index_cols = np.ndarray(number_of_nodes, dtype=np.ndarray)\n for n in range(number_of_nodes):\n reverse_index_row = reverse_index_csr.getrow(n)\n reverse_index_rows[n] = reverse_index_row.indices\n\n if n < base_community_number:\n reverse_index_col = reverse_index_csc.getcol(n)\n reverse_index_cols[n] = reverse_index_col.indices\n\n flag = True\n\n print('Start merge iterations.')\n\n iteration = 0\n\n while flag:\n level_row = list()\n level_col = list()\n\n append_level_row = level_row.append\n append_level_col = level_col.append\n\n unavailable_communities = -1*np.ones(reverse_index_csc.shape[1])\n unavailable_communities_counter = 0\n next_level_communities = list()\n append_next_level_community = next_level_communities.append\n number_of_communities = 0\n for j in range(reverse_index_csr.shape[1]):\n if j in unavailable_communities:\n continue\n must_break = reverse_index_csr.shape[1] - unavailable_communities_counter\n print(must_break)\n if must_break < 1:\n break\n unavailable_communities[unavailable_communities_counter] = j\n unavailable_communities_counter += 1\n c_j = reverse_index_cols[j]\n\n indices = community_neighbors(c_j, reverse_index_rows, unavailable_communities, unavailable_communities_counter)\n\n max_similarity = -1\n community_index = 0\n for jj in indices:\n c_jj = reverse_index_cols[jj]\n similarity = jaccard(c_j, c_jj)\n if similarity > max_similarity:\n max_similarity = similarity\n community_index = jj\n\n jj = community_index\n if max_similarity > 0:\n # Merge two communities\n c_jj = reverse_index_cols[jj]\n c_new = np.union1d(c_j, c_jj)\n\n flag_1 = np.setdiff1d(c_new, c_j)\n flag_2 = np.setdiff1d(c_new, c_jj)\n if (flag_1.size != 0) and (flag_2.size != 0):\n for n in c_new:\n append_level_row(n)\n append_level_col(number_of_communities)\n\n if c_new.size < alpha:\n append_next_level_community(number_of_communities)\n\n number_of_communities += 1\n unavailable_communities[unavailable_communities_counter] = jj\n unavailable_communities_counter += 1\n\n level_row = np.array(level_row)\n level_col = np.array(level_col)\n level_data = np.ones(level_row.size, dtype=np.float64)\n communities = sparse.coo_matrix((level_data, (level_row, level_col)),\n shape=(number_of_nodes, number_of_communities))\n\n if communities.getnnz() == 0:\n break\n\n features = sparse.hstack([features, communities])\n\n reverse_index_csc = copy.copy(communities)\n reverse_index_csc = reverse_index_csc.tocsc()\n reverse_index_csc = reverse_index_csc[:, np.array(next_level_communities)]\n reverse_index_csr = reverse_index_csc.tocsr()\n\n reverse_index_rows = np.ndarray(number_of_nodes, dtype=np.ndarray)\n reverse_index_cols = np.ndarray(len(next_level_communities), dtype=np.ndarray)\n for n in range(number_of_nodes):\n reverse_index_row = reverse_index_csr.getrow(n)\n reverse_index_rows[n] = reverse_index_row.indices\n\n if n < len(next_level_communities):\n reverse_index_col = reverse_index_csc.getcol(n)\n reverse_index_cols[n] = reverse_index_col.indices\n\n if len(next_level_communities) > 1:\n flag = True\n\n iteration += 1\n print('Iteration: ', iteration)\n print('List length', len(next_level_communities))\n\n return features", "response": "This function extracts hierarchical community features using the MROC method."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfind communities with shared nodes to a seed community.", "response": "def community_neighbors(c_j, reverse_index_rows, unavailable_communities, unavailable_communities_counter):\n \"\"\"\n Finds communities with shared nodes to a seed community. Called by mroc.\n\n Inputs: - c_j: The seed community for which we want to find which communities overlap.\n - reverse_index_rows: A node to community indicator matrix.\n - unavailable_communities: A set of communities that have already either been merged or failed to merge.\n - unavailable_communities_counter: The number of such communities.\n\n Outputs: - indices: An array containing the communities that exhibit overlap with the seed community.\n \"\"\"\n indices = list()\n extend = indices.extend\n for node in c_j:\n extend(reverse_index_rows[node])\n\n indices = np.array(indices)\n indices = np.setdiff1d(indices, unavailable_communities[:unavailable_communities_counter+1])\n\n return indices"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncalculating the Jaccard similarity between two communities.", "response": "def jaccard(c_1, c_2):\n \"\"\"\n Calculates the Jaccard similarity between two sets of nodes. Called by mroc.\n\n Inputs: - c_1: Community (set of nodes) 1.\n - c_2: Community (set of nodes) 2.\n\n Outputs: - jaccard_similarity: The Jaccard similarity of these two communities.\n \"\"\"\n nom = np.intersect1d(c_1, c_2).size\n denom = np.union1d(c_1, c_2).size\n return nom/denom"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nperform community embedding using the LOUVAIN method. Introduced in: Blondel, V. D., Guillaume, J. L., Lambiotte, R., & Lefebvre, E. (2008). Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment, 2008(10), P10008. Inputs: - A in R^(nxn): Adjacency matrix of an undirected network represented as a SciPy Sparse COOrdinate matrix. Outputs: - X in R^(nxC_n): The latent space embedding represented as a SciPy Sparse COOrdinate matrix.", "response": "def louvain(adjacency_matrix):\n \"\"\"\n Performs community embedding using the LOUVAIN method.\n\n Introduced in: Blondel, V. D., Guillaume, J. L., Lambiotte, R., & Lefebvre, E. (2008).\n Fast unfolding of communities in large networks.\n Journal of Statistical Mechanics: Theory and Experiment, 2008(10), P10008.\n\n Inputs: - A in R^(nxn): Adjacency matrix of an undirected network represented as a SciPy Sparse COOrdinate matrix.\n\n Outputs: - X in R^(nxC_n): The latent space embedding represented as a SciPy Sparse COOrdinate matrix.\n \"\"\"\n # Convert to networkx undirected graph.\n adjacency_matrix = nx.from_scipy_sparse_matrix(adjacency_matrix, create_using=nx.Graph())\n\n # Call LOUVAIN algorithm to calculate a hierarchy of communities.\n tree = community.generate_dendogram(adjacency_matrix, part_init=None)\n\n # Embed communities\n row = list()\n col = list()\n append_row = row.append\n append_col = col.append\n\n community_counter = 0\n for i in range(len(tree)):\n partition = community.partition_at_level(tree, i)\n for n, c in partition.items():\n append_row(n)\n append_col(community_counter + c)\n\n community_counter += max(partition.values()) + 1\n\n row = np.array(row)\n col = np.array(col)\n data = np.ones(row.size, dtype=np.float64)\n\n louvain_features = sparse.coo_matrix((data, (row, col)), shape=(len(partition.keys()), community_counter),\n dtype=np.float64)\n\n return louvain_features"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef laplacian_eigenmaps(adjacency_matrix, k):\n # Calculate sparse graph Laplacian.\n laplacian = get_normalized_laplacian(adjacency_matrix)\n\n # Calculate bottom k+1 eigenvalues and eigenvectors of normalized Laplacian.\n try:\n eigenvalues, eigenvectors = spla.eigsh(laplacian,\n k=k,\n which='SM',\n return_eigenvectors=True)\n except spla.ArpackNoConvergence as e:\n print(\"ARPACK has not converged.\")\n eigenvalue = e.eigenvalues\n eigenvectors = e.eigenvectors\n\n # Discard the eigenvector corresponding to the zero-valued eigenvalue.\n eigenvectors = eigenvectors[:, 1:]\n\n return eigenvectors", "response": "This function computes the eigenmaps of the graph symmetrically normalized Laplacian matrix."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nperforming spectral graph embedding on the centrality reweighted adjacency matrix Inputs: - A in R^(nxn): Adjacency matrix of an undirected network represented as a scipy.sparse.coo_matrix - k: The number of social dimensions/eigenvectors to extract - max_iter: The maximum number of iterations for the iterative eigensolution method Outputs: - S in R^(nxk): The social dimensions represented as a numpy.array matrix", "response": "def replicator_eigenmaps(adjacency_matrix, k):\n \"\"\"\n Performs spectral graph embedding on the centrality reweighted adjacency matrix\n\n Inputs: - A in R^(nxn): Adjacency matrix of an undirected network represented as a scipy.sparse.coo_matrix\n - k: The number of social dimensions/eigenvectors to extract\n - max_iter: The maximum number of iterations for the iterative eigensolution method\n\n Outputs: - S in R^(nxk): The social dimensions represented as a numpy.array matrix\n \"\"\"\n number_of_nodes = adjacency_matrix.shape[0]\n\n max_eigenvalue = spla.eigsh(adjacency_matrix,\n k=1,\n which='LM',\n return_eigenvectors=False)\n\n # Calculate Replicator matrix\n eye_matrix = sparse.eye(number_of_nodes, number_of_nodes, dtype=np.float64)\n eye_matrix = eye_matrix.tocsr()\n eye_matrix.data = eye_matrix.data*max_eigenvalue\n replicator = eye_matrix - adjacency_matrix\n\n # Calculate bottom k+1 eigenvalues and eigenvectors of normalised Laplacian\n try:\n eigenvalues, eigenvectors = spla.eigsh(replicator,\n k=k+1,\n which='SM',\n return_eigenvectors=True)\n except spla.ArpackNoConvergence as e:\n print(\"ARPACK has not converged.\")\n eigenvalue = e.eigenvalues\n eigenvectors = e.eigenvectors\n\n eigenvectors = eigenvectors[:, 1:]\n\n return eigenvectors"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef base_communities(adjacency_matrix):\n number_of_nodes = adjacency_matrix.shape[0]\n\n # X = A + I\n adjacency_matrix = adjacency_matrix.tocsr()\n adjacency_matrix = adjacency_matrix.transpose()\n features = sparse.csr_matrix(sparse.eye(number_of_nodes, number_of_nodes)) + adjacency_matrix.tocsr()\n features = features.tocsr()\n features.data = np.ones_like(features.data)\n\n return features", "response": "Returns the community indicator normalized feature matrix for any graph."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create_embeded_pkcs7_signature(data, cert, key):\n\n assert isinstance(data, bytes)\n assert isinstance(cert, str)\n\n try:\n pkey = crypto.load_privatekey(crypto.FILETYPE_PEM, key)\n signcert = crypto.load_certificate(crypto.FILETYPE_PEM, cert)\n except crypto.Error as e:\n raise exceptions.CorruptCertificate from e\n\n bio_in = crypto._new_mem_buf(data)\n pkcs7 = crypto._lib.PKCS7_sign(\n signcert._x509, pkey._pkey, crypto._ffi.NULL, bio_in, PKCS7_NOSIGS\n )\n bio_out = crypto._new_mem_buf()\n crypto._lib.i2d_PKCS7_bio(bio_out, pkcs7)\n signed_data = crypto._bio_to_string(bio_out)\n\n return signed_data", "response": "Create an embeded PKCS7 signature."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create_key(file_):\n pkey = crypto.PKey()\n pkey.generate_key(crypto.TYPE_RSA, 2048)\n\n file_.write(crypto.dump_privatekey(crypto.FILETYPE_PEM, pkey))\n file_.flush()", "response": "Create a key and save it into file_."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create_csr(key_file, organization_name, common_name, serial_number, file_):\n key = crypto.load_privatekey(crypto.FILETYPE_PEM, key_file.read())\n\n req = crypto.X509Req()\n subj = req.get_subject()\n\n subj.O = organization_name # noqa: E741 (we can't do anything about this)\n subj.CN = common_name\n subj.serialNumber = serial_number\n\n req.set_pubkey(key)\n req.sign(key, 'md5')\n\n file_.write(crypto.dump_certificate_request(crypto.FILETYPE_PEM, req))", "response": "Create a CSR for a key and save it into file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfitting a Linear Support Vector Classifier to the labelled graph - based features using the LIBLINEAR library.", "response": "def model_fit(X_train, y_train, svm_hardness, fit_intercept, number_of_threads, classifier_type=\"LinearSVC\"):\n \"\"\"\n Fits a Linear Support Vector Classifier to the labelled graph-based features using the LIBLINEAR library.\n\n One-vs-All: http://scikit-learn.org/stable/modules/generated/sklearn.multiclass.OneVsRestClassifier.html\n LinearSVC: http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html\n\n Inputs: - feature_matrix: The graph based-features in either NumPy or SciPy sparse array format.\n - node_label_matrix: The node-label ground truth in a SciPy sparse matrix format.\n - svm_hardness: Penalty of the error term.\n - fit_intercept: Data centering as per scikit-learn.\n - number_of_threads: The number of threads to use for training the multi-label scheme.\n - classifier_type: A string to be chosen among: * LinearSVC (LibLinear)\n * LogisticRegression (LibLinear)\n * RandomForest\n\n Output: - model: A trained scikit-learn One-vs-All multi-label scheme of linear SVC models.\n \"\"\"\n if classifier_type == \"LinearSVC\":\n if X_train.shape[0] > X_train.shape[1]:\n dual = False\n else:\n dual = True\n\n model = OneVsRestClassifier(LinearSVC(C=svm_hardness, random_state=0, dual=dual,\n fit_intercept=fit_intercept),\n n_jobs=number_of_threads)\n model.fit(X_train, y_train)\n elif classifier_type == \"LogisticRegression\":\n if X_train.shape[0] > X_train.shape[1]:\n dual = False\n else:\n dual = True\n\n model = OneVsRestClassifier(LogisticRegression(C=svm_hardness, random_state=0, dual=dual,\n fit_intercept=fit_intercept),\n n_jobs=number_of_threads)\n model.fit(X_train, y_train)\n elif classifier_type == \"RandomForest\":\n model = OneVsRestClassifier(RandomForestClassifier(n_estimators=1000, criterion=\"gini\",\n n_jobs=number_of_threads, random_state=0))\n if issparse(X_train):\n model.fit(X_train.tocsc(), y_train.toarray())\n else:\n model.fit(X_train, y_train.toarray())\n else:\n print(\"Invalid classifier type.\")\n raise RuntimeError\n\n return model"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfit a meta - labeler model to the data.", "response": "def meta_model_fit(X_train, y_train, svm_hardness, fit_intercept, number_of_threads, regressor_type=\"LinearSVR\"):\n \"\"\"\n Trains meta-labeler for predicting number of labels for each user.\n\n Based on: Tang, L., Rajan, S., & Narayanan, V. K. (2009, April).\n Large scale multi-label classification via metalabeler.\n In Proceedings of the 18th international conference on World wide web (pp. 211-220). ACM.\n \"\"\"\n if regressor_type == \"LinearSVR\":\n if X_train.shape[0] > X_train.shape[1]:\n dual = False\n else:\n dual = True\n\n model = LinearSVR(C=svm_hardness, random_state=0, dual=dual,\n fit_intercept=fit_intercept)\n y_train_meta = y_train.sum(axis=1)\n model.fit(X_train, y_train_meta)\n else:\n print(\"Invalid regressor type.\")\n raise RuntimeError\n\n return model"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nusing a trained model and the unlabelled features to produce a user-to-label distance matrix. Inputs: - feature_matrix: The graph based-features in either NumPy or SciPy sparse array format. - model: A trained scikit-learn One-vs-All multi-label scheme of linear SVC models. - classifier_type: A string to be chosen among: * LinearSVC (LibLinear) * LogisticRegression (LibLinear) * RandomForest Output: - decision_weights: A NumPy array containing the distance of each user from each label discriminator.", "response": "def weigh_users(X_test, model, classifier_type=\"LinearSVC\"):\n \"\"\"\n Uses a trained model and the unlabelled features to produce a user-to-label distance matrix.\n\n Inputs: - feature_matrix: The graph based-features in either NumPy or SciPy sparse array format.\n - model: A trained scikit-learn One-vs-All multi-label scheme of linear SVC models.\n - classifier_type: A string to be chosen among: * LinearSVC (LibLinear)\n * LogisticRegression (LibLinear)\n * RandomForest\n\n Output: - decision_weights: A NumPy array containing the distance of each user from each label discriminator.\n \"\"\"\n if classifier_type == \"LinearSVC\":\n decision_weights = model.decision_function(X_test)\n elif classifier_type == \"LogisticRegression\":\n decision_weights = model.predict_proba(X_test)\n elif classifier_type == \"RandomForest\":\n # if issparse(X_test):\n # decision_weights = np.hstack(a[:, 1].reshape(X_test.shape[0], 1) for a in model.predict_proba(X_test.tocsr()))\n # else:\n # decision_weights = np.hstack(a[:, 1].reshape(X_test.shape[0], 1) for a in model.predict_proba(X_test))\n if issparse(X_test):\n decision_weights = model.predict_proba(X_test.tocsr())\n else:\n decision_weights = model.predict_proba(X_test)\n else:\n print(\"Invalid classifier type.\")\n raise RuntimeError\n\n return decision_weights"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nclassifying the users in a single - class graph based on the unlabelled features.", "response": "def classify_users(X_test, model, classifier_type, meta_model, upper_cutoff):\n \"\"\"\n Uses a trained model and the unlabelled features to associate users with labels.\n\n The decision is done as per scikit-learn:\n http://scikit-learn.org/stable/modules/generated/sklearn.multiclass.OneVsRestClassifier.html\n http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC.predict\n\n Inputs: - feature_matrix: The graph based-features in either NumPy or SciPy sparse array format.\n - model: A trained scikit-learn One-vs-All multi-label scheme of linear SVC models.\n\n Output: - decision_weights: A NumPy array containing the distance of each user from each label discriminator.\n \"\"\"\n if classifier_type == \"LinearSVC\":\n prediction = model.decision_function(X_test)\n # prediction = penalize_large_classes(prediction)\n\n meta_prediction = meta_model.predict(X_test)\n meta_prediction = np.rint(meta_prediction) + 1\n meta_prediction[meta_prediction > upper_cutoff] = upper_cutoff\n meta_prediction[meta_prediction < 2] = 2\n\n prediction_indices = np.argsort(prediction, axis=1)\n\n prediction_row = np.empty(int(np.sum(meta_prediction)), dtype=np.int32)\n prediction_col = np.empty(int(np.sum(meta_prediction)), dtype=np.int32)\n prediction_data = np.empty(int(np.sum(meta_prediction)), dtype=np.float64)\n\n nnz_counter = 0\n for i in range(X_test.shape[0]):\n jj = prediction_indices[i, -int(meta_prediction[i]):]\n dd = prediction[i, jj]\n\n prediction_row[nnz_counter:nnz_counter+int(meta_prediction[i])] = i\n prediction_col[nnz_counter:nnz_counter+int(meta_prediction[i])] = jj\n prediction_data[nnz_counter:nnz_counter+int(meta_prediction[i])] = dd\n\n nnz_counter += int(meta_prediction[i])\n\n prediction = spsp.coo_matrix((prediction_data,\n (prediction_row,\n prediction_col)),\n shape=prediction.shape)\n\n prediction = normalize(prediction, norm=\"l2\", axis=0)\n elif classifier_type == \"LogisticRegression\":\n prediction = model.predict_proba(X_test)\n # prediction = penalize_large_classes(prediction)\n\n meta_prediction = meta_model.predict(X_test)\n meta_prediction = np.rint(meta_prediction) + 1\n meta_prediction[meta_prediction > upper_cutoff] = upper_cutoff\n meta_prediction[meta_prediction < 2] = 2\n\n prediction_indices = np.argsort(prediction, axis=1)\n\n prediction_row = np.empty(int(np.sum(meta_prediction)), dtype=np.int32)\n prediction_col = np.empty(int(np.sum(meta_prediction)), dtype=np.int32)\n prediction_data = np.empty(int(np.sum(meta_prediction)), dtype=np.float64)\n\n nnz_counter = 0\n for i in range(X_test.shape[0]):\n jj = prediction_indices[i, -int(meta_prediction[i]):]\n dd = prediction[i, jj]\n\n prediction_row[nnz_counter:nnz_counter+int(meta_prediction[i])] = i\n prediction_col[nnz_counter:nnz_counter+int(meta_prediction[i])] = jj\n prediction_data[nnz_counter:nnz_counter+int(meta_prediction[i])] = dd\n\n nnz_counter += int(meta_prediction[i])\n\n prediction = spsp.coo_matrix((prediction_data,\n (prediction_row,\n prediction_col)),\n shape=prediction.shape)\n elif classifier_type == \"RandomForest\":\n if issparse(X_test):\n prediction = model.predict_proba(X_test.tocsr())\n else:\n prediction = model.predict_proba(X_test)\n # prediction = penalize_large_classes(prediction)\n\n meta_prediction = meta_model.predict(X_test)\n meta_prediction = np.rint(meta_prediction) + 1\n meta_prediction[meta_prediction > upper_cutoff] = upper_cutoff\n meta_prediction[meta_prediction < 2] = 2\n\n prediction_indices = np.argsort(prediction, axis=1)\n\n prediction_row = np.empty(int(np.sum(meta_prediction)), dtype=np.int32)\n prediction_col = np.empty(int(np.sum(meta_prediction)), dtype=np.int32)\n prediction_data = np.empty(int(np.sum(meta_prediction)), dtype=np.float64)\n\n nnz_counter = 0\n for i in range(X_test.shape[0]):\n jj = prediction_indices[i, -int(meta_prediction[i]):]\n dd = prediction[i, jj]\n\n prediction_row[nnz_counter:nnz_counter+int(meta_prediction[i])] = i\n prediction_col[nnz_counter:nnz_counter+int(meta_prediction[i])] = jj\n prediction_data[nnz_counter:nnz_counter+int(meta_prediction[i])] = dd\n\n nnz_counter += int(meta_prediction[i])\n\n prediction = spsp.coo_matrix((prediction_data,\n (prediction_row,\n prediction_col)),\n shape=prediction.shape)\n else:\n print(\"Invalid classifier type.\")\n raise RuntimeError\n\n return prediction"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_client(service_name, sandbox=False):\n key = (service_name.lower(), sandbox,)\n\n try:\n if key not in cached_clients:\n cached_clients[key] = Client(wsdls[key], transport=transport)\n\n return cached_clients[key]\n except KeyError:\n raise ValueError('Unknown service name, {}'.format(service_name))", "response": "Returns a zeep client for a given service."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert a sequence to primitive equivalents.", "response": "def _extract_seq(extractor, seq):\n \"\"\"Convert a sequence to primitive equivalents.\"\"\"\n subtype = Any\n if seq.__args__ and seq.__args__[0] is not Any:\n subtype = seq.__args__[0]\n return _array_type(extractor.extract(subtype))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn True if the type in question is a Sequence object from the typing module.", "response": "def _is_sequence(typ):\n \"\"\"\n returns True if the type in question\n is a Sequence[] object from the typing module.\n \"\"\"\n # PEP_560 deprecates issubclass for\n # List types, for the time being\n # we'll support a specific escape hatch.\n if PEP_560: # pragma: no cover\n return isinstance(typ, _GenericAlias) and typ.__origin__ is list\n else: # pragma: no cover\n return issubclass(typ, Sequence)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntakes an attrs based class and convert it to jsonschema.", "response": "def extract(cls, extractor, typ):\n \"\"\"\n take an attrs based class, and convert it\n to jsonschema.\n \"\"\"\n schema = {\n \"title\": typ.__name__,\n \"type\": \"object\",\n \"properties\": {},\n \"required\": []\n }\n for attribute in attr.fields(typ):\n details = cls._extract_attribute(extractor, attribute)\n if details.is_required:\n schema[\"required\"].append(details.name)\n schema[\"properties\"][details.name] = details.schema\n return schema"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ntries to read README. rst or return empty string if failed.", "response": "def readme():\n \"\"\"Try to read README.rst or return empty string if failed.\n\n :return: File contents.\n :rtype: str\n \"\"\"\n path = os.path.realpath(os.path.join(os.path.dirname(__file__), 'README.rst'))\n handle = None\n url_prefix = 'https://raw.githubusercontent.com/Robpol86/{name}/v{version}/'.format(name=NAME, version=VERSION)\n try:\n handle = codecs.open(path, encoding='utf-8')\n return handle.read(131072).replace('.. image:: docs', '.. image:: {0}docs'.format(url_prefix))\n except IOError:\n return ''\n finally:\n getattr(handle, 'close', lambda: None)()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nlisting available serial ports", "response": "def listSerialPorts():\n\t\"\"\"\n\thttp://pyserial.readthedocs.io/en/latest/shortintro.html\n\n\tThis calls the command line tool from pyserial to list the available\n\tserial ports.\n\t\"\"\"\n\tcmd = 'python -m serial.tools.list_ports'\n\terr, ret = commands.getstatusoutput(cmd)\n\tif not err:\n\t\tr = ret.split('\\n')\n\t\tret = []\n\t\tfor line in r:\n\t\t\tif line.find('/dev/') >= 0:\n\t\t\t\tline = line.replace(' ', '')\n\t\t\t\tret.append(line)\n\treturn err, ret"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef prettyPrintPacket(ctrl_table):\n\tprint('---------------------------------------')\n\tprint(\"{:.<29} {}\".format('id', ctrl_table['id']))\n\tctrl_table.pop('id')\n\tfor key, value in ctrl_table.items():\n\t\tprint(\"{:.<29} {}\".format(key, value))", "response": "This will pretty print out a packet s fields."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreading a Json file and returns a dictionary", "response": "def read(fname):\n\t\t\"\"\"\n\t\tReads a Json file\n\t\tin: file name\n\t\tout: length of file, dictionary\n\t\t\"\"\"\n\t\ttry:\n\t\t\twith open(fname, 'r') as f:\n\t\t\t\tdata = json.load(f)\n\t\t\treturn data\n\n\t\texcept IOError:\n\t\t\traise Exception('Could not open {0!s} for reading'.format((fname)))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef write(fname, data):\n\t\ttry:\n\t\t\twith open(fname, 'w') as f:\n\t\t\t\tjson.dump(data, f)\n\n\t\texcept IOError:\n\t\t\traise Exception('Could not open {0!s} for writing'.format((fname)))", "response": "Writes a Json file with the given data to the given file."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nmaking a write packet that writes a value to the servo ID at the given register.", "response": "def makeWritePacket(ID, reg, values=None):\n\t\"\"\"\n\tCreates a packet that writes a value(s) to servo ID at location reg. Make\n\tsure the values are in little endian (use Packet.le() if necessary) for 16 b\n\t(word size) values.\n\t\"\"\"\n\tpkt = makePacket(ID, xl320.XL320_WRITE, reg, values)\n\treturn pkt"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nmake a read packet that reads the register of servo ID at location reg.", "response": "def makeReadPacket(ID, reg, values=None):\n\t\"\"\"\n\tCreates a packet that reads the register(s) of servo ID at location reg. Make\n\tsure the values are in little endian (use Packet.le() if necessary) for 16 b\n\t(word size) values.\n\t\"\"\"\n\tpkt = makePacket(ID, xl320.XL320_READ, reg, values)\n\treturn pkt"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef makeResetPacket(ID, param):\n\tif param not in [0x01, 0x02, 0xff]:\n\t\traise Exception('Packet.makeResetPacket invalide parameter {}'.format(param))\n\t# pkt = makePacket(ID, xl320.XL320_RESET, None, [param])\n\tpkt = makePacket(ID, xl320.XL320_RESET, None, [1])\n\treturn pkt", "response": "This function creates a packet that is used to reset a servo to one of 3 reset states."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncommands the servo to an angle (in degrees)", "response": "def makeServoPacket(ID, angle):\n\t\"\"\"\n\tCommands the servo to an angle (in degrees)\n\t\"\"\"\n\tif not (0.0 <= angle <= 300.0):\n\t\traise Exception('makeServoPacket(), angle out of bounds: {}'.format(angle))\n\tval = int(angle/300*1023)\n\t# lo, hi = le(val)\n\t# print('servo cmd {} deg : {} : L {} H {}'.format(angle, val, lo, hi))\n\tpkt = makeWritePacket(ID, xl320.XL320_GOAL_POSITION, le(val))\n\treturn pkt"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nmake a servo max limit packet.", "response": "def makeServoMaxLimitPacket(ID, angle):\n\t\"\"\"\n\tSets the maximum servo angle (in the CCW direction)\n\t\"\"\"\n\tangle = int(angle/300.0*1023)\n\tpkt = makeWritePacket(ID, xl320.XL320_CCW_ANGLE_LIMIT, le(angle))\n\treturn pkt"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef makeServoMinLimitPacket(ID, angle):\n\tangle = int(angle/300.0*1023)\n\tpkt = makeWritePacket(ID, xl320.XL320_CW_ANGLE_LIMIT, le(angle))\n\treturn pkt", "response": "Make a servo minimum limit packet."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef makeServoSpeedPacket(ID, maxSpeed):\n\tif 0.0 > maxSpeed > 1.0:\n\t\traise Exception('makeServoSpeed: max speed is a percentage (0.0-1.0)')\n\tspeed = int(maxSpeed*1023)\n\tpkt = makeWritePacket(ID, xl320.XL320_GOAL_VELOCITY, le(speed))\n\treturn pkt", "response": "Make a servo speed packet."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nturn on or off the servo LED.", "response": "def makeLEDPacket(ID, color):\n\t\"\"\"\n\tTurn on/off the servo LED and also sets the color.\n\t\"\"\"\n\t# pkt = [255, 255, 253, 0, ID, 11, 0, 3, 25, 0, 2 crc_l, crc_h]\n\tpkt = makeWritePacket(ID, xl320.XL320_LED, [color])\n\treturn pkt"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef makeDelayPacket(ID, delay):\n\tpkt = makeWritePacket(ID, xl320.XL320_DELAY_TIME, delay)\n\treturn pkt", "response": "Make a delay packet."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef makeControlModePacket(ID, mode):\n\tpkt = makeWritePacket(ID, xl320.XL320_CONTROL_MODE, le(mode))\n\treturn pkt", "response": "Make a xl - 320 control mode packet."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef makeBaudRatePacket(ID, rate):\n\tif rate not in [0, 1, 2, 3]:\n\t\traise Exception('Packet.makeBaudRatePacket: wrong rate {}'.format(rate))\n\tpkt = makeWritePacket(ID, xl320.XL320_BAUD_RATE, [rate])\n\treturn pkt", "response": "makeBaudRatePacket - make baud rate packet"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nmakes sync angle packet", "response": "def makeSyncAnglePacket(info):\n\t\"\"\"\n\tWrite sync angle information to servos.\n\n\tinfo = [[ID, angle], [ID, angle], ...]\n\t\"\"\"\n\taddr = le(xl320.XL320_GOAL_POSITION)\n\tdata = []\n\n\t# since all servo angles have the same register addr (XL320_GOAL_POSITION)\n\t# and data size (2), a sinc packet is smart choice\n\t# compare bulk vs sync for the same commands:\n\t# bulk = 94 bytes\n\t# sync = 50 bytes\n\tdata.append(addr[0]) # LSB\n\tdata.append(addr[1]) # MSB\n\tdata.append(2) # data size LSM\n\tdata.append(0) # data size MSB\n\tfor pkt in info:\n\t\tdata.append(pkt[0]) # ID\n\t\tangle = le(int(pkt[1]/300*1023))\n\t\tdata.append(angle[0]) # LSB\n\t\tdata.append(angle[1]) # MSB\n\n\tID = xl320.XL320_BROADCAST_ADDR\n\tinstr = xl320.XL320_SYNC_WRITE\n\tpkt = makePacket(ID, instr, None, data) # create packet\n\n\t# print(pkt)\n\n\treturn pkt"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nmaking a bulk angle packet.", "response": "def makeBulkAnglePacket(info):\n\t\"\"\"\n\tWrite bulk angle information to servos.\n\n\tinfo = [[ID, angle], [ID, angle], ...]\n\t\"\"\"\n\taddr = le(xl320.XL320_GOAL_POSITION)\n\tdata = []\n\tfor pkt in info:\n\t\tdata.append(pkt[0]) # ID\n\t\tdata.append(addr[0]) # LSB\n\t\tdata.append(addr[1]) # MSB\n\t\tdata.append(2) # 2 bytes\n\t\tdata.append(0)\n\t\tangle = le(int(pkt[1]/300*1023))\n\t\tdata.append(angle[0]) # LSB\n\t\tdata.append(angle[1]) # MSB\n\n\tID = xl320.XL320_BROADCAST_ADDR\n\tinstr = xl320.XL320_BULK_WRITE\n\tpkt = makePacket(ID, instr, None, data) # create packet\n\n\treturn pkt"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef findPkt(pkt):\n\t# print('findpkt', pkt)\n\t# print('-----------------------')\n\tret = []\n\twhile len(pkt)-10 >= 0:\n\t\tif pkt[0:4] != [0xFF, 0xFF, 0xFD, 0x00]:\n\t\t\tpkt.pop(0) # get rid of the first index\n\t\t\t# print(' - pop:', pkt)\n\t\t\tcontinue\n\t\t# print(' > good packet')\n\t\tlength = (pkt[6] << 8) + pkt[5]\n\t\t# print(' > length', length)\n\t\tcrc_pos = 5 + length\n\t\tpkt_crc = pkt[crc_pos:crc_pos + 2]\n\t\tcrc = le(crc16(pkt[:crc_pos]))\n\t\t# if len(pkt) < (crc_pos + 1):\n\t\t# \tprint('<<< need more data for findPkt >>>')\n\n\t\t# print(' > calc crc', crc)\n\t\t# print(' > pkt crc', pkt_crc)\n\t\tif pkt_crc == crc:\n\t\t\tpkt_end = crc_pos+2\n\t\t\tret.append(pkt[:pkt_end])\n\t\t\t# print(' > found:', pkt[:pkt_end])\n\t\t\t# print(' > pkt size', pkt_end)\n\t\t\tdel pkt[:pkt_end]\n\t\t\t# print(' > remaining:', pkt)\n\t\telse:\n\t\t\tpkt_end = crc_pos+2\n\t\t\t# print(' - crap:', pkt[:pkt_end])\n\t\t\tdel pkt[:pkt_end]\n\t# print('findpkt ret:', ret)\n\treturn ret", "response": "findpkt - Search through a string of binary for a valid xl320 package."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef packetToDict(pkt):\n\n\td = {\n\t\t'id': pkt[4],\n\t\t'instruction': xl320.InstrToStr[pkt[7]],\n\t\t'length': (pkt[6] << 8) + pkt[5],\n\t\t'params': pkt[8:-2],\n\t\t'crc': pkt[-2:]\n\t}\n\n\treturn d", "response": "This function turns a packet into a dictionary"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef populate_all():\n ReceiptType.objects.populate()\n ConceptType.objects.populate()\n DocumentType.objects.populate()\n VatType.objects.populate()\n TaxType.objects.populate()\n CurrencyType.objects.populate()", "response": "Fetch and store all metadata from the AFIP."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef first_currency():\n ct = CurrencyType.objects.filter(code='PES').first()\n if ct:\n return ct.pk", "response": "Returns the id for the first currency in the not\n "} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\npopulates the database with types retrieved from the AFIP.", "response": "def populate(self, ticket=None):\n \"\"\"\n Populate the database with types retrieved from the AFIP.\n\n If no ticket is provided, the most recent available one will be used.\n \"\"\"\n ticket = ticket or AuthTicket.objects.get_any_active('wsfe')\n client = clients.get_client('wsfe', ticket.owner.is_sandboxed)\n service = getattr(client.service, self.__service_name)\n response_xml = service(serializers.serialize_ticket(ticket))\n\n check_response(response_xml)\n\n for result in getattr(response_xml.ResultGet, self.__type_name):\n self.get_or_create(\n code=result.Id,\n description=result.Desc,\n valid_from=parsers.parse_date(result.FchDesde),\n valid_to=parsers.parse_date(result.FchHasta),\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the certificate as an OpenSSL object.", "response": "def certificate_object(self):\n \"\"\"\n Returns the certificate as an OpenSSL object\n\n Returns the certificate as an OpenSSL object (rather than as a file\n object).\n \"\"\"\n if not self.certificate:\n return None\n self.certificate.seek(0)\n return crypto.parse_certificate(self.certificate.read())"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting the certificate expiration from the certificate file.", "response": "def get_certificate_expiration(self):\n \"\"\"\n Gets the certificate expiration from the certificate\n\n\n Gets the certificate expiration from the certificate file. Note that\n this value is stored into ``certificate_expiration`` when an instance\n is saved, so you should generally prefer that method (since this one\n requires reading and parsing the entire certificate).\n \"\"\"\n datestring = self.certificate_object.get_notAfter().decode()\n dt = datetime.strptime(datestring, '%Y%m%d%H%M%SZ')\n return dt.replace(tzinfo=timezone.utc)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngenerate a key file for this TaxPayer and saves it. Returns True if a key was created False otherwise.", "response": "def generate_key(self, force=False):\n \"\"\"\n Creates a key file for this TaxPayer\n\n Creates a key file for this TaxPayer if it does not have one, and\n immediately saves it.\n\n Returns True if and only if a key was created.\n \"\"\"\n if self.key and not force:\n logger.warning(\n 'Tried to generate key for a taxpayer that already had one'\n )\n return False\n\n with NamedTemporaryFile(suffix='.key') as file_:\n crypto.create_key(file_)\n self.key = File(file_, name='{}.key'.format(uuid.uuid4().hex))\n self.save()\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngenerating a CSR for this TaxPayer s key and returns it", "response": "def generate_csr(self, basename='djangoafip'):\n \"\"\"\n Creates a CSR for this TaxPayer's key\n\n Creates a file-like object that contains the CSR which can be used to\n request a new certificate from AFIP.\n \"\"\"\n csr = BytesIO()\n crypto.create_csr(\n self.key.file,\n self.name,\n '{}{}'.format(basename, int(datetime.now().timestamp())),\n 'CUIT {}'.format(self.cuit),\n csr,\n )\n csr.seek(0)\n return csr"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate an AuthTicket for a given service.", "response": "def create_ticket(self, service):\n \"\"\"Create an AuthTicket for a given service.\"\"\"\n ticket = AuthTicket(owner=self, service=service)\n ticket.authorize()\n return ticket"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_ticket(self, service):\n return self.auth_tickets \\\n .filter(expires__gt=datetime.now(timezone.utc), service=service) \\\n .last()", "response": "Return an existing AuthTicket for a given service."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfetch all point of sales from the WS and store them locally.", "response": "def fetch_points_of_sales(self, ticket=None):\n \"\"\"\n Fetch all point of sales objects.\n\n Fetch all point of sales from the WS and store (or update) them\n locally.\n\n Returns a list of tuples with the format (pos, created,).\n \"\"\"\n ticket = ticket or self.get_or_create_ticket('wsfe')\n\n client = clients.get_client('wsfe', self.is_sandboxed)\n response = client.service.FEParamGetPtosVenta(\n serializers.serialize_ticket(ticket),\n )\n check_response(response)\n\n results = []\n for pos_data in response.ResultGet.PtoVenta:\n results.append(PointOfSales.objects.update_or_create(\n number=pos_data.Nro,\n issuance_type=pos_data.EmisionTipo,\n owner=self,\n defaults={\n 'blocked': pos_data.Bloqueado == 'N',\n 'drop_date': parsers.parse_date(pos_data.FchBaja),\n }\n ))\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef authorize(self):\n request = self.__create_request_xml()\n request = self.__sign_request(request)\n request = b64encode(request).decode()\n\n client = clients.get_client('wsaa', self.owner.is_sandboxed)\n try:\n raw_response = client.service.loginCms(request)\n except Fault as e:\n if str(e) == 'Certificado expirado':\n raise exceptions.CertificateExpired(str(e)) from e\n if str(e) == 'Certificado no emitido por AC de confianza':\n raise exceptions.UntrustedCertificate(str(e)) from e\n raise exceptions.AuthenticationError(str(e)) from e\n response = etree.fromstring(raw_response.encode('utf-8'))\n\n self.token = response.xpath(self.TOKEN_XPATH)[0].text\n self.signature = response.xpath(self.SIGN_XPATH)[0].text\n\n self.save()", "response": "Send this ticket to AFIP for authorization."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _assign_numbers(self):\n first = self.select_related('point_of_sales', 'receipt_type').first()\n\n next_num = Receipt.objects.fetch_last_receipt_number(\n first.point_of_sales,\n first.receipt_type,\n ) + 1\n\n for receipt in self.filter(receipt_number__isnull=True):\n # Atomically update receipt number\n Receipt.objects.filter(\n pk=receipt.id,\n receipt_number__isnull=True,\n ).update(\n receipt_number=next_num,\n )\n next_num += 1", "response": "Assign numbers in preparation for validating these receipts."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncheck that all receipts returned by this queryset are groupable.", "response": "def check_groupable(self):\n \"\"\"\n Checks that all receipts returned by this queryset are groupable.\n\n \"Groupable\" means that they can be validated together: they have the\n same POS and receipt type.\n\n Returns the same queryset is all receipts are groupable, otherwise,\n raises :class:`~.CannotValidateTogether`.\n \"\"\"\n types = self.aggregate(\n poses=Count('point_of_sales_id', distinct=True),\n types=Count('receipt_type', distinct=True),\n )\n\n if set(types.values()) > {1}:\n raise exceptions.CannotValidateTogether()\n\n return self"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nvalidate all receipts matching this queryset and returns a list of validation errors.", "response": "def validate(self, ticket=None):\n \"\"\"\n Validates all receipts matching this queryset.\n\n Note that, due to how AFIP implements its numbering, this method is not\n thread-safe, or even multiprocess-safe.\n\n Because of this, it is possible that not all instances matching this\n queryset are validated properly. Obviously, only successfully validated\n receipts will be updated.\n\n Returns a list of errors as returned from AFIP's webservices. An\n exception is not raised because partial failures are possible.\n\n Receipts that succesfully validate will have a\n :class:`~.ReceiptValidation` object attatched to them with a validation\n date and CAE information.\n\n Already-validated receipts are ignored.\n\n Attempting to validate an empty queryset will simply return an empty\n list.\n \"\"\"\n # Skip any already-validated ones:\n qs = self.filter(validation__isnull=True).check_groupable()\n if qs.count() == 0:\n return []\n qs.order_by('issued_date', 'id')._assign_numbers()\n\n return qs._validate(ticket)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the number for the last validated receipt.", "response": "def fetch_last_receipt_number(self, point_of_sales, receipt_type):\n \"\"\"Returns the number for the last validated receipt.\"\"\"\n client = clients.get_client('wsfe', point_of_sales.owner.is_sandboxed)\n response_xml = client.service.FECompUltimoAutorizado(\n serializers.serialize_ticket(\n point_of_sales.owner.get_or_create_ticket('wsfe')\n ),\n point_of_sales.number,\n receipt_type.code,\n )\n check_response(response_xml)\n\n # TODO XXX: Error handling\n # (FERecuperaLastCbteResponse){\n # PtoVta = 0\n # CbteTipo = 0\n # CbteNro = 0\n # Errors =\n # (ArrayOfErr){\n # Err[] =\n # (Err){\n # Code = 601\n # Msg = \"CUIT representada no incluida en Token\"\n # },\n # }\n # }\n\n return response_xml.CbteNro"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the sum of all Vat objects.", "response": "def total_vat(self):\n \"\"\"Returns the sum of all Vat objects.\"\"\"\n q = Vat.objects.filter(receipt=self).aggregate(total=Sum('amount'))\n return q['total'] or 0"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef total_tax(self):\n q = Tax.objects.filter(receipt=self).aggregate(total=Sum('amount'))\n return q['total'] or 0", "response": "Returns the sum of all Tax objects."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning True if this instance is validated.", "response": "def is_validated(self):\n \"\"\"\n Returns True if this instance is validated.\n\n Note that resolving this property requires a DB query, so if you've a\n very large amount of receipts you should prefetch (see django's\n ``select_related``) the ``validation`` field. Even so, a DB query *may*\n be triggered.\n\n If you need a large list of validated receipts, you should actually\n filter them via a QuerySet::\n\n Receipt.objects.filter(validation__result==RESULT_APPROVED)\n\n :rtype: bool\n \"\"\"\n # Avoid the DB lookup if possible:\n if not self.receipt_number:\n return False\n\n try:\n return self.validation.result == ReceiptValidation.RESULT_APPROVED\n except ReceiptValidation.DoesNotExist:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nvalidate this receipt. This is a shortcut to :class:`~.ReceiptQuerySet`'s method of the same name. Calling this validates only this instance. :param AuthTicket ticket: Use this ticket. If None, one will be loaded or created automatically. :param bool raise_: If True, an exception will be raised when validation fails.", "response": "def validate(self, ticket=None, raise_=False):\n \"\"\"\n Validates this receipt.\n\n This is a shortcut to :class:`~.ReceiptQuerySet`'s method of the same\n name. Calling this validates only this instance.\n\n :param AuthTicket ticket: Use this ticket. If None, one will be loaded\n or created automatically.\n :param bool raise_: If True, an exception will be raised when\n validation fails.\n \"\"\"\n # XXX: Maybe actually have this sortcut raise an exception?\n rv = Receipt.objects.filter(pk=self.pk).validate(ticket)\n # Since we're operating via a queryset, this instance isn't properly\n # updated:\n self.refresh_from_db()\n if raise_ and rv:\n raise exceptions.ValidationError(rv[0])\n return rv"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create_for_receipt(self, receipt, **kwargs):\n try:\n profile = TaxPayerProfile.objects.get(\n taxpayer__points_of_sales__receipts=receipt,\n )\n except TaxPayerProfile.DoesNotExist:\n raise exceptions.DjangoAfipException(\n 'Cannot generate a PDF for taxpayer with no profile',\n )\n pdf = ReceiptPDF.objects.create(\n receipt=receipt,\n issuing_name=profile.issuing_name,\n issuing_address=profile.issuing_address,\n issuing_email=profile.issuing_email,\n vat_condition=profile.vat_condition,\n gross_income_condition=profile.gross_income_condition,\n sales_terms=profile.sales_terms,\n **kwargs\n )\n return pdf", "response": "Creates a ReceiptPDF object for a given receipt. Does not actually generate the related PDF file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef save_pdf(self, save_model=True):\n from django_afip.views import ReceiptPDFView\n\n if not self.receipt.is_validated:\n raise exceptions.DjangoAfipException(\n _('Cannot generate pdf for non-authorized receipt')\n )\n\n self.pdf_file = File(BytesIO(), name='{}.pdf'.format(uuid.uuid4().hex))\n render_pdf(\n template='receipts/code_{}.html'.format(\n self.receipt.receipt_type.code,\n ),\n file_=self.pdf_file,\n context=ReceiptPDFView.get_context_for_pk(self.receipt_id),\n )\n\n if save_model:\n self.save()", "response": "Save the receipt as a PDF related to this model instance."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef compute_amount(self):\n self.amount = self.base_amount * self.aliquot / 100\n return self.amount", "response": "Auto - assign and return the total amount for this tax."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the label - based random walk transition probability matrix.", "response": "def get_label_based_random_walk_matrix(adjacency_matrix, labelled_nodes, label_absorption_probability):\n \"\"\"\n Returns the label-absorbing random walk transition probability matrix.\n\n Input: - A: A sparse matrix that contains the adjacency matrix of the graph.\n\n Output: - W: A sparse matrix that contains the natural random walk transition probability matrix.\n \"\"\"\n # Turn to sparse.csr_matrix format for faster row access.\n rw_transition = sparse.csr_matrix(adjacency_matrix, dtype=np.float64)\n\n # Sum along the two axes to get out-degree and in-degree, respectively\n out_degree = rw_transition.sum(axis=1)\n in_degree = rw_transition.sum(axis=0)\n\n # Form the inverse of the diagonal matrix containing the out-degree\n for i in np.arange(rw_transition.shape[0]):\n rw_transition.data[rw_transition.indptr[i]: rw_transition.indptr[i + 1]] =\\\n rw_transition.data[rw_transition.indptr[i]: rw_transition.indptr[i + 1]]/out_degree[i]\n\n out_degree = np.array(out_degree).astype(np.float64).reshape(out_degree.size)\n in_degree = np.array(in_degree).astype(np.float64).reshape(in_degree.size)\n\n # When the random walk agent encounters a labelled node, there is a probability that it will be absorbed.\n diag = np.zeros_like(out_degree)\n diag[labelled_nodes] = 1.0\n diag = sparse.dia_matrix((diag, [0]), shape=(in_degree.size, in_degree.size))\n diag = sparse.csr_matrix(diag)\n\n rw_transition[labelled_nodes, :] = (1-label_absorption_probability)*rw_transition[labelled_nodes, :] + label_absorption_probability*diag[labelled_nodes, :]\n\n return rw_transition, out_degree, in_degree"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the natural random walk transition probability matrix given the adjacency matrix.", "response": "def get_natural_random_walk_matrix(adjacency_matrix, make_shared=False):\n \"\"\"\n Returns the natural random walk transition probability matrix given the adjacency matrix.\n\n Input: - A: A sparse matrix that contains the adjacency matrix of the graph.\n\n Output: - W: A sparse matrix that contains the natural random walk transition probability matrix.\n \"\"\"\n # Turn to sparse.csr_matrix format for faster row access.\n rw_transition = sparse.csr_matrix(adjacency_matrix, dtype=np.float64, copy=True)\n\n # Sum along the two axes to get out-degree and in-degree, respectively\n out_degree = rw_transition.sum(axis=1)\n in_degree = rw_transition.sum(axis=0)\n\n out_degree[out_degree == 0.0] = 1.0\n\n # Form the inverse of the diagonal matrix containing the out-degree\n for i in np.arange(rw_transition.shape[0]):\n rw_transition.data[rw_transition.indptr[i]: rw_transition.indptr[i + 1]] =\\\n rw_transition.data[rw_transition.indptr[i]: rw_transition.indptr[i + 1]]/out_degree[i]\n\n rw_transition.sort_indices()\n\n out_degree = np.array(out_degree).astype(np.float64).reshape(out_degree.size)\n in_degree = np.array(in_degree).astype(np.float64).reshape(in_degree.size)\n\n if make_shared:\n number_of_nodes = adjacency_matrix.shape[0]\n\n out_degree_c = mp.Array(c.c_double, number_of_nodes)\n in_degree_c = mp.Array(c.c_double, number_of_nodes)\n\n out_degree_shared = np.frombuffer(out_degree_c.get_obj(), dtype=np.float64, count=number_of_nodes)\n in_degree_shared = np.frombuffer(in_degree_c.get_obj(), dtype=np.float64, count=number_of_nodes)\n\n out_degree_shared[:] = out_degree[:]\n in_degree_shared[:] = in_degree[:]\n\n indices_c = mp.Array(c.c_int64, rw_transition.indices.size)\n indptr_c = mp.Array(c.c_int64, rw_transition.indptr.size)\n data_c = mp.Array(c.c_double, rw_transition.data.size)\n\n indices_shared = np.frombuffer(indices_c.get_obj(), dtype=np.int64, count=rw_transition.indices.size)\n indptr_shared = np.frombuffer(indptr_c.get_obj(), dtype=np.int64, count=rw_transition.indptr.size)\n data_shared = np.frombuffer(data_c.get_obj(), dtype=np.float64, count=rw_transition.data.size)\n\n indices_shared[:] = rw_transition.indices[:]\n indptr_shared[:] = rw_transition.indptr[:]\n data_shared[:] = rw_transition.data[:]\n\n rw_transition = sparse.csr_matrix((data_shared,\n indices_shared,\n indptr_shared),\n shape=rw_transition.shape)\n\n return rw_transition, out_degree, in_degree"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntest whether the function is a middleware", "response": "def is_middleware(func) -> bool:\n \"\"\"\n test whether it is a middleware\n :return: Boolean\n \"\"\"\n\n if inspect.isfunction(func):\n _check = func\n _name = func.__name__\n else:\n _check = func.__call__\n _name = func.__class__.__name__\n\n if not inspect.iscoroutinefunction(_check):\n raise UnknownMiddlewareException(\"Middleware {} should be async function\".format(_name))\n\n args = list(inspect.signature(_check).parameters.keys())\n\n if set(args) - MIDDLEWARE_PARAMETER_BOUNDARY:\n raise UnknownMiddlewareException(\"Parameters of middleware {} \"\n \"must be in list ['app', 'request', 'response', 'next']\".format(_name))\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nchecks the number valiaty for the Rond\u00f4nia state", "response": "def start(st_reg_number):\n \"\"\"Checks the number valiaty for the Rond\u00f4nia state\"\"\"\n\n divisor = 11\n\n weight = [6, 5, 4, 3, 2, 9, 8, 7, 6, 5, 4, 3, 2]\n\n if len(st_reg_number) != 9 and len(st_reg_number) != 14:\n return False\n\n if len(st_reg_number) == 9:\n sum_total = 0\n peso = 6\n for i in range(3, len(st_reg_number)-1):\n sum_total = sum_total + int(st_reg_number[i]) * peso\n peso = peso - 1\n\n rest_division = sum_total % divisor\n digit = divisor - rest_division\n\n if digit == 10:\n digit = 0\n\n if digit == 11:\n digit = 1\n\n return digit == int(st_reg_number[len(st_reg_number)-1])\n\n else:\n sum_total = 0\n for i in range(len(st_reg_number)-1):\n sum_total = sum_total + int(st_reg_number[i]) * weight[i]\n\n rest_division = sum_total % divisor\n digit = divisor - rest_division\n\n if digit == 10:\n digit = 0\n\n if digit == 11:\n digit = 1\n\n return digit == int(st_reg_number[len(st_reg_number)-1])"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef start(st_reg_number):\n\n divisor = 11\n\n verificador_one = int(st_reg_number[len(st_reg_number)-4])\n verificador_two = int(st_reg_number[len(st_reg_number)-1])\n weights_first = [1, 3, 4, 5, 6, 7, 8, 10]\n weights_secund = [3, 2, 10, 9, 8, 7, 6, 5, 4, 3, 2]\n\n if len(st_reg_number) > 13:\n return False\n\n if len(st_reg_number) < 12:\n return False\n\n if len(st_reg_number) == 12:\n sum_total = 0\n for i in range(len(st_reg_number)-4):\n sum_total = sum_total + int(st_reg_number[i]) * weights_first[i]\n\n rest_division = sum_total % divisor\n digit_first = rest_division\n\n if rest_division == 10:\n digit_first = 0\n\n num = 0\n mult = 10000000000\n for i in range(len(st_reg_number)-1):\n if i == 8:\n num = num + digit_first * mult\n else:\n num = num + int(st_reg_number[i]) * mult\n mult = mult/10\n\n if num < 10000000000:\n new_st = str('0') + str(num)\n else:\n new_st = str(num)\n\n sum_total = 0\n for i in range(len(new_st)):\n sum_total = sum_total + int(new_st[i]) * weights_secund[i]\n\n rest_division = sum_total % divisor\n digit_secund = rest_division\n\n if rest_division == 10:\n digit_secund = 0\n\n if digit_secund == verificador_two and digit_first == verificador_one:\n return True\n else:\n return False\n\n else:\n if st_reg_number[0] != \"P\":\n return False\n\n sum_total = 0\n for i in range(1, 9):\n sum_total = sum_total + int(st_reg_number[i]) * weights_first[i-1]\n\n digit = sum_total % divisor\n\n if digit == 10:\n digit = 0\n\n return digit == int(st_reg_number[len(st_reg_number)-4])", "response": "Checks the number valiaty for the S\u00e3o Paulo state"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef start(st_reg_number):\n divisor = 11\n\n if len(st_reg_number) > 9:\n return False\n\n if len(st_reg_number) < 9:\n return False\n\n sum_total = 0\n peso = 8\n for i in range(len(st_reg_number)-2):\n sum_total = sum_total + int(st_reg_number[i]) * peso\n peso = peso - 1\n\n rest_division = sum_total % divisor\n digit_first = divisor - rest_division\n\n if rest_division < 2:\n digit_first = 0\n\n if rest_division > 1:\n digit_first = 11 - rest_division\n\n num = 0\n peso = 9\n mult = 10000000\n for i in range(len(st_reg_number)-2):\n num = num + int(st_reg_number[i]) * mult\n mult = mult/10\n\n num = num + digit_first\n\n new_st = str(num)\n sum_total = 0\n peso = 9\n for i in range(len(new_st)-1):\n sum_total = sum_total + int(new_st[i]) * peso\n peso = peso - 1\n\n rest_division = sum_total % divisor\n\n if rest_division < 2:\n digit_secund = 0\n\n if rest_division > 1:\n digit_secund = divisor - rest_division\n\n if digit_secund == int(st_reg_number[len(st_reg_number)-1]) and digit_first == int(st_reg_number[len(st_reg_number)-2]):\n return True\n else:\n return False", "response": "Checks the number valiaty for the Pernanbuco state"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nset response header :param key: the key of header :param value: the value of header", "response": "def set_header(self, key: str, value: str) -> None:\n \"\"\"\n set response header\n :param key: the key of header\n :param value: the value of header\n \"\"\"\n self.headers[key] = value"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncheck the number valiaty for the Bahia state", "response": "def start(st_reg_number):\n \"\"\"Checks the number valiaty for the Bahia state\"\"\"\n\n weights_second_digit = range(len(st_reg_number)-1, 1, -1)\n weights_first_digit = range(len(st_reg_number), 1, -1)\n second_digits = st_reg_number[-1:]\n number_state_registration = st_reg_number[0:len(st_reg_number) - 2]\n digits_state_registration = st_reg_number[-2:]\n sum_second_digit = 0\n sum_first_digit = 0\n\n if len(st_reg_number) != 8 and len(st_reg_number) != 9:\n return False\n\n if st_reg_number[-8] in ['0', '1', '2', '3', '4', '5', '8']:\n\n for i in weights_second_digit:\n\n sum_second_digit = sum_second_digit + i*int(st_reg_number[-i-1])\n\n second_digits_check = 10 - (sum_second_digit % 10)\n\n if sum_second_digit % 10 == 0 or sum_second_digit % 11 == 1:\n\n second_digits_check = '0'\n\n if str(second_digits_check) != second_digits:\n\n return False\n\n digit_two = number_state_registration + str(second_digits_check)\n\n for i in weights_first_digit:\n\n sum_first_digit = sum_first_digit + i*int(digit_two[-i+1])\n\n first_digits_check = 10 - (sum_first_digit % 10)\n\n if sum_first_digit % 10 == 0 or sum_first_digit % 10 == 1:\n first_digits_check = '0'\n\n digits_calculated = str(first_digits_check) + str(second_digits_check)\n\n return digits_calculated == digits_state_registration\n\n elif st_reg_number[-8] in ['6', '7', '9']:\n\n for i in weights_second_digit:\n\n sum_second_digit = sum_second_digit + i*int(st_reg_number[-i-1])\n\n second_digits_check = 11 - (sum_second_digit % 11)\n\n if sum_second_digit % 11 == 0 or sum_second_digit % 11 == 1:\n second_digits_check = '0'\n\n if str(second_digits_check) != second_digits:\n return False\n\n digit_two = number_state_registration + str(second_digits_check)\n\n for i in weights_first_digit:\n\n sum_first_digit = sum_first_digit + i*int(digit_two[-i+1])\n\n first_digits_check = 11 - (sum_first_digit % 11)\n\n if sum_first_digit % 11 == 0 or sum_first_digit % 11 == 1:\n first_digits_check = '0'\n digits_calculated = str(first_digits_check) + str(second_digits_check)\n return digits_calculated == digits_state_registration"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\nasync def send_http_response(writer,\n http_code: int,\n headers: List[Tuple[str, str]],\n content: bytes,\n http_status: str= None\n ) -> None:\n \"\"\"\n generate http response payload and send to writer\n \"\"\"\n # generate response payload\n if not http_status:\n http_status = STATUS_CODES.get(http_code, 'Unknown')\n\n response: bytes = f'HTTP/1.1 {http_code} {http_status}\\r\\n'.encode()\n for k, v in headers:\n response += f'{k}: {v}\\r\\n'.encode()\n response += b'\\r\\n'\n response += content\n\n # send payload\n writer.write(response)\n await writer.drain()", "response": "Send an HTTP response to the writer."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nstart server with websocket", "response": "async def serve(http_handler: HTTP_WRAPPER_TYPE,\n websocket_handler=None,\n address: str='127.0.0.1',\n port: int=8000):\n \"\"\"\n start server\n \"\"\"\n return await asyncio.start_server(SocketWrapper(http_handler, websocket_handler), address, port)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nformats the body into different type", "response": "def __body_format(self, body):\n \"\"\"\n format body into different type\n :param body:\n :return:\n \"\"\"\n\n ctype, pdict = parse_header(self.content_type)\n if ctype == 'application/json':\n self.form = json.loads(body.decode())\n\n elif ctype == 'application/x-www-form-urlencoded':\n for key, value in parse_qs(body).items():\n self.form[key.decode()] = [one_value.decode() for one_value in value]\n\n elif ctype == \"multipart/form-data\":\n safe_env = {\n 'QUERY_STRING': '',\n 'REQUEST_METHOD': self.method.upper(),\n 'CONTENT_TYPE': self.content_type,\n 'CONTENT_LENGTH': self.headers.get('Content-Length', '-1')\n }\n fields = FieldStorage(io.BytesIO(body), environ=safe_env, keep_blank_values=True)\n\n fields = fields.list or []\n for field in fields:\n if field.filename:\n self.form[field.name] = File(field.filename, field.file)\n else:\n self.form[field.name] = field.value"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking the number valiaty for the Paraiba state", "response": "def start(st_reg_number):\n \"\"\"Checks the number valiaty for the Paraiba state\"\"\"\n #st_reg_number = str(st_reg_number)\n weights = [9, 8, 7, 6, 5, 4, 3, 2]\n digit_state_registration = st_reg_number[-1]\n\n if len(st_reg_number) != 9:\n return False\n\n sum_total = 0\n\n for i in range(0, 8):\n sum_total = sum_total + weights[i] * int(st_reg_number[i])\n\n if sum_total % 11 == 0:\n return digit_state_registration[-1] == '0'\n\n digit_check = 11 - sum_total % 11\n\n return str(digit_check) == digit_state_registration"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck the number valiaty for the Mato Grosso state", "response": "def start(st_reg_number):\n \"\"\"Checks the number valiaty for the Mato Grosso state\"\"\"\n #st_reg_number = str(st_reg_number)\n weights = [3, 2, 9, 8, 7, 6, 5, 4, 3, 2]\n digit_state_registration = st_reg_number[-1]\n\n if len(st_reg_number) != 11:\n return False\n\n sum = 0\n\n for i in range(0, 10):\n sum = sum + weights[i] * int(st_reg_number[i])\n\n if sum % 11 == 0:\n return digit_state_registration[-1] == '0'\n\n digit_check = 11 - sum % 11\n\n return str(digit_check) == digit_state_registration"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncheck the number valiaty for the Sergipe state", "response": "def start(st_reg_number):\n \"\"\"Checks the number valiaty for the Sergipe state\"\"\"\n divisor = 11\n\n if len(st_reg_number) > 9:\n return False\n\n if len(st_reg_number) < 9:\n return False\n\n sum_total = 0\n peso = 9\n\n for i in range(len(st_reg_number)-1):\n sum_total = sum_total + int(st_reg_number[i]) * peso\n peso = peso - 1\n\n rest_division = sum_total % divisor\n digit = divisor - rest_division\n\n if digit == 10 or digit == 11:\n digit = 0\n\n return digit == int(st_reg_number[len(st_reg_number)-1])"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef start(st_reg_number):\n\n weights = [9, 8, 7, 6, 5, 4, 3, 2]\n number_state_registration = st_reg_number[0:len(st_reg_number) - 1]\n digit_state_registration = st_reg_number[-1]\n\n if st_reg_number[0:2] not in ['10', '11', '12']:\n return False\n\n if len(st_reg_number) != 9:\n return False\n\n sum_total = 0\n\n for i in weights:\n sum_total = sum_total + i * (int(number_state_registration[-i+1]))\n\n check_number = number_state_registration <= 10119997\n\n if sum_total % 11 == 0:\n return '0' == digit_state_registration\n\n elif sum_total % 11 == 1 and (int(number_state_registration) >= 10103105 and check_number):\n return '1' == digit_state_registration\n\n elif sum_total % 11 == 1:\n return '0' == digit_state_registration\n\n else:\n digit_check = 11 - sum_total % 11\n return digit_state_registration == str(digit_check)\n\n if number_state_registration == '11094402' and (number_state_registration == '1' or number_state_registration == '0'):\n return True", "response": "Checks the number valiaty for the Espirito Santo state"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef start(st_reg_number):\n\n #st_reg_number = str(st_reg_number)\n weights = [4, 3, 2, 9, 8, 7, 6, 5, 4, 3, 2]\n digits = st_reg_number[:len(st_reg_number) - 2]\n check_digits = st_reg_number[-2:]\n divisor = 11\n\n if len(st_reg_number) > 13:\n return False\n\n sum_total = 0\n for i in range(len(digits)):\n sum_total = sum_total + int(digits[i]) * weights[i]\n\n rest_division = sum_total % divisor\n first_digit = divisor - rest_division\n\n if first_digit == 10 or first_digit == 11:\n first_digit = 0\n\n if str(first_digit) != check_digits[0]:\n return False\n\n digits = digits + str(first_digit)\n weights = [5] + weights\n\n sum_total = 0\n for i in range(len(digits)):\n sum_total = sum_total + int(digits[i]) * weights[i]\n\n rest_division = sum_total % divisor\n second_digit = divisor - rest_division\n\n if second_digit == 10 or second_digit == 11:\n second_digit = 0\n\n return str(first_digit) + str(second_digit) == check_digits", "response": "Checks the number valiaty for the Acre state"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load_from_object(self, object_name):\n for key in dir(object_name):\n if key.isupper():\n self[key] = getattr(object_name, key)", "response": "load all upper parameters of object as config parameters\n "} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef start(st_reg_number):\n #st_reg_number = str(st_reg_number)\n number_state_registration_first_digit = st_reg_number[0:3] + '0' + st_reg_number[3: len(st_reg_number)-2]\n weights_first_digit = [1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2]\n wights_second_digit = [3, 2, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2]\n first_digit = st_reg_number[-2]\n second_digit = st_reg_number[-1]\n sum_first_digit = 0\n sum_second_digit = 0\n sum_result_digit = ''\n sum_end = 0\n\n if len(st_reg_number) != 13:\n\n return False\n\n for i in range(0, 12):\n\n sum_first_digit = weights_first_digit[i] * int(number_state_registration_first_digit[i])\n\n sum_result_digit = sum_result_digit + str(sum_first_digit)\n\n for i in range(0, len(sum_result_digit)):\n\n sum_end = sum_end + int(sum_result_digit[i])\n\n if sum_end % 10 == 0:\n\n check_digit_one = 0\n\n elif sum_end < 10:\n\n check_digit_one = 10 - sum_end\n\n elif sum_end > 10:\n\n check_digit_one = (10 - sum_end % 10)\n\n if str(check_digit_one) != first_digit:\n\n return False\n\n number_state_registration_second_digit = st_reg_number + str(check_digit_one)\n\n for i in range(0, 12):\n\n sum_second_digit = sum_second_digit + wights_second_digit[i] * int(number_state_registration_second_digit[i])\n\n check_second_digit = 11 - sum_second_digit % 11\n\n if sum_second_digit == 1 or sum_second_digit == 0:\n\n return second_digit == '0'\n\n else:\n return str(check_second_digit) == second_digit", "response": "Checks the number valiaty for the Minas Gerais state"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef use(self, *middleware: MiddlewareType) -> None:\n for m in middleware:\n if is_middleware(m):\n self.middleware.append(m)", "response": "Register a set of middleware functions."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\nasync def handler(self, request: Request) -> Tuple[int, str, List[Tuple[str, str]], bytes]:\n response: 'Response' = Response()\n\n handler: Callable = empty\n chain_reverse = self.middleware[::-1]\n for middleware in chain_reverse:\n handler = map_context_to_middleware(middleware, self, request, response, handler)\n\n try:\n await handler()\n\n except HttpException as e:\n response.code = e.code\n response.content = e.body\n\n return response.code, response.status, response.header_as_list, response.output", "response": "The handler handling each request\nAttributeNames"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef run(self, host: str=\"localhost\", port: int=8000, debug: bool=False):\n self.debug = debug\n loop = asyncio.get_event_loop()\n\n try:\n loop.run_until_complete(self.start_server(host, port))\n loop.run_forever()\n except KeyboardInterrupt:\n loop.run_until_complete(self.signal_manager.activate('before_close'))\n loop.run_until_complete(self.close_server_async())\n loop.run_until_complete(self.signal_manager.activate('after_close'))\n loop.run_until_complete(asyncio.gather(*asyncio.Task.all_tasks()))\n loop.close()", "response": "Start the http server and run the loop until it is done."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncheck the number valiaty for the Amazonas state", "response": "def start(st_reg_number):\n \"\"\"Checks the number valiaty for the Amazonas state\"\"\"\n weights = range(2, 10)\n digits = st_reg_number[0:len(st_reg_number) - 1]\n control_digit = 11\n check_digit = st_reg_number[-1:]\n\n if len(st_reg_number) != 9:\n return False\n\n sum_total = 0\n\n for i in weights:\n sum_total = sum_total + i * int(digits[i-2])\n\n if sum_total < control_digit:\n control_digit = 11 - sum_total\n return str(digit_calculated) == check_digit\n\n elif sum_total % 11 <= 1:\n return '0' == check_digit\n\n else:\n digit_calculated = 11 - sum_total % 11\n return str(digit_calculated) == check_digit"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef start(state_registration_number, state_abbreviation):\n\n state_abbreviation = state_abbreviation.upper()\n states_validations = {\n 'AC': \"ac.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'AL': \"al.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'AM': \"am.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'AP': \"ap.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'BA': \"ba.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'CE': \"ce.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'DF': \"df.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'ES': \"es.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'GO': \"go.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'MA': \"ma.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'MG': \"mg.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'MS': \"ms.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'MT': \"mt.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'PA': \"pa.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'PB': \"pb.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'PE': \"pe.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'PI': \"pi.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'PR': \"pr.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'RJ': \"rj.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'RN': \"rn.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'RO': \"ro.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'RR': \"rr.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'RS': \"rs.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'SC': \"sc.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'SE': \"se.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'SP': \"sp.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\",\n 'TO': \"to.start(\" + \"\\\"\" + state_registration_number + \"\\\"\" + \")\"\n\n }\n\n exec('validity = ' + states_validations[state_abbreviation])\n return validity", "response": "This function is like a Facade to another modules that makes their own state validation. It is like a Facade to another modules that makes their own state validation. It is like a Facade to another modules that makes their own state validation."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking the number valiaty for the Alagoas state", "response": "def start(st_reg_number):\n \"\"\"Checks the number valiaty for the Alagoas state\"\"\"\n\n if len(st_reg_number) > 9:\n return False\n\n if len(st_reg_number) < 9:\n return False\n\n if st_reg_number[0:2] != \"24\":\n return False\n\n if st_reg_number[2] not in ['0', '3', '5', '7', '8']:\n return False\n\n aux = 9\n sum_total = 0\n for i in range(len(st_reg_number)-1):\n sum_total = sum_total + int(st_reg_number[i]) * aux\n aux -= 1\n\n product = sum_total * 10\n\n aux_2 = int(product/11)\n\n digit = product - aux_2 * 11\n\n if digit == 10:\n digit = 0\n\n return digit == int(st_reg_number[len(st_reg_number)-1])"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef start(st_reg_number):\n\n divisor = 11\n if len(st_reg_number) > 9:\n return False\n\n if len(st_reg_number) < 9:\n return False\n\n if st_reg_number[0:2] != \"03\":\n return False\n\n aux = int(st_reg_number[0:len(st_reg_number) - 1])\n\n if 3000000 < aux and aux < 3017001:\n control1 = 5\n control2 = 0\n\n if 3017000 < aux and aux < 3019023:\n control1 = 9\n control2 = 1\n\n if aux > 3019022:\n control1 = 0\n control2 = 0\n sum_total = 0\n peso = 9\n for i in range(len(st_reg_number)-1):\n sum_total = sum_total + int(st_reg_number[i]) * peso\n peso = peso - 1\n sum_total += control1\n\n rest_division = sum_total % divisor\n digit = divisor - rest_division\n\n if digit == 10:\n digit = 0\n\n if digit == 11:\n digit = control2\n\n return digit == int(st_reg_number[len(st_reg_number)-1])", "response": "Checks the number valiaty for the Alagoas state"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a dictionary representation of the error.", "response": "def to_dict(self):\n \"\"\"Return a dictionary representation of the error.\n\n :return: A dict with the keys:\n - attr: Attribute which contains the error, or \"\" if it refers to the schema root.\n - errors: A list of dictionary representations of the errors.\n \"\"\"\n def exception_to_dict(e):\n try:\n return e.to_dict()\n except AttributeError:\n return {\n \"type\": e.__class__.__name__,\n \"error\": str(e),\n }\n\n result = {\n \"errors\": [exception_to_dict(e) for e in self.errors]\n }\n if self.index is not None:\n result[\"index\"] = self.index\n else:\n result[\"attr\"] = self.attr if self.attr is not None else \"\"\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _filtered_actions(self, *a_types):\n for a in filter(lambda _: self.is_action(_, *a_types), self._actions):\n yield a", "response": "Get actions filtered on a list of action types."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nask the user for input of a single argument.", "response": "def _input_arg(self, a):\n \"\"\"\n Ask the user for input of a single argument.\n \n :param a: argparse.Action instance\n :return: the user input, asked according to the action\n \"\"\"\n # if action of an argument that suppresses any other, just return\n if a.dest == SUPPRESS or a.default == SUPPRESS:\n return\n # prepare the prompt\n prompt = (a.help or a.dest).capitalize()\n r = {'required': a.required}\n # now handle each different action\n if self.is_action(a, 'store', 'append'):\n return user_input(prompt, a.choices, a.default, **r)\n elif self.is_action(a, 'store_const', 'append_const'):\n return user_input(prompt, (\"(A)dd\", \"(D)iscard\"), \"d\", **r)\n elif self.is_action(a, 'store_true'):\n return user_input(prompt, (\"(Y)es\", \"(N)o\"), \"n\", **r)\n elif self.is_action(a, 'store_false'):\n return user_input(prompt, (\"(Y)es\", \"(N)o\"), \"n\", **r)\n elif self.is_action(a, 'count'):\n return user_input(prompt, is_pos_int, 0, \"positive integer\", **r)\n elif self.is_action(a, 'parsers'):\n pmap = a._name_parser_map\n _ = list(pmap.keys())\n return user_input(prompt, _, _[0], **r) if len(_) > 0 else None\n raise NotImplementedError(\"Unknown argparse action\")"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _set_arg(self, a, s=\"main\", c=False):\n # if action of an argument that suppresses any other, just return\n if a.dest is SUPPRESS or a.default is SUPPRESS:\n return\n # check if an option string is used for this action in sys.argv ;\n # if so, simply return as it will be parsed normally\n if any(o in sys.argv[1:] for o in a.option_strings):\n return\n # in case of non-null config, get the value from the config object\n default = a.default if a.default is None else str(a.default)\n if c:\n try:\n value = ArgumentParser._config.get(s, a.dest)\n except (NoOptionError, NoSectionError) as e:\n item = \"setting\" if isinstance(e, NoOptionError) else \"section\"\n # if the argument is required, just ask for the value\n value = self._input_arg(a) if a.required else default\n logger.debug(\"{} {} not present in config (set to {})\"\n .format(a.dest, item, value))\n # in case of null config, just ask for the value\n else:\n value = self._input_arg(a)\n # collect the option string before continuing\n try:\n ostr = a.option_strings[0]\n except IndexError: # occurs when positional argument\n ostr = None\n # now handle arguments regarding the action\n if self.is_action(a, 'store', 'append'):\n if value:\n if ostr:\n self._reparse_args['opt'].extend([ostr, value])\n else:\n self._reparse_args['pos'].extend([value])\n elif self.is_action(a, 'store_const', 'append_const'):\n if value.lower() == \"add\" or value != default:\n self._reparse_args['opt'].append(ostr)\n elif self.is_action(a, 'store_true'):\n if value.lower() in [\"y\", \"true\"]:\n self._reparse_args['opt'].append(ostr)\n elif self.is_action(a, 'store_false'):\n if value.lower() in [\"n\", \"false\"]:\n self._reparse_args['opt'].append(ostr)\n elif self.is_action(a, 'count'):\n v = int(value or 0)\n if v > 0:\n if ostr.startswith(\"--\"):\n new_arg = [ostr for i in range(v)]\n else:\n new_arg = [\"-{}\".format(v * ostr.strip('-'))]\n self._reparse_args['opt'].extend(new_arg)\n elif self.is_action(a, 'parsers'):\n if not value:\n value = self._input_arg(a)\n pmap = a._name_parser_map\n if c:\n pmap[value].config_args(a.dest)\n pmap[value]._reparse_args['pos'].insert(0, value)\n else:\n pmap[value].input_args()\n self._reparse_args['sub'].append(pmap[value])\n else:\n raise NotImplementedError(\"Unknown argparse action\")", "response": "Set a single argument."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _sorted_actions(self):\n for a in filter(lambda _: not _.last and \\\n not self.is_action(_, 'parsers'), self._actions):\n yield a\n for a in filter(lambda _: _.last and \\\n not self.is_action(_, 'parsers'), self._actions):\n yield a\n for a in filter(lambda _: self.is_action(_, 'parsers'),\n self._actions):\n yield a", "response": "Generate the sorted list of actions based on the last attribute."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef config_args(self, section=\"main\"):\n if self._config_parsed:\n return\n for a in self._filtered_actions(\"config\"):\n for o in a.option_strings:\n try:\n i = sys.argv.index(o)\n sys.argv.pop(i) # remove the option string\n sys.argv.pop(i) # remove the value that follows\n except ValueError:\n pass\n for a in self._sorted_actions():\n self._set_arg(a, section, True)\n self._config_parsed = True", "response": "Feeds the input arguments from a config file."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef demo_args(self):\n argv = random.choice(self.examples).replace(\"--demo\", \"\")\n self._reparse_args['pos'] = shlex.split(argv)", "response": "Additional method for replacing input arguments by demo ones."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef parse_args(self, args=None, namespace=None):\n if not namespace: # use the new Namespace class for handling _config\n namespace = Namespace(self)\n namespace = super(ArgumentParser, self).parse_args(args, namespace)\n if len(self._reparse_args['pos']) > 0 or \\\n len(self._reparse_args['opt']) > 0 or \\\n len(self._reparse_args['sub']) > 0:\n args = self._reset_args()\n namespace = super(ArgumentParser, self).parse_args(args, namespace)\n # process \"-hh...\" here, after having parsed the arguments\n help_level = getattr(namespace, \"help\", 0)\n if help_level > 0:\n self.print_help()\n self.print_extended_help(help_level)\n self.exit()\n return namespace", "response": "Reparses new arguments when the demo_args method is called."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef error(self, message):\n if all(len(x) == 0 for x in self._reparse_args.values()):\n # normal behavior with argparse\n self.print_usage(sys.stderr)\n self.exit(2, gt('%s: error: %s\\n') % (self.prog, message))", "response": "Prints a usage message incorporating the message to stderr and exits the program with an error code."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nadds a parameter to the shared ConfigParser object.", "response": "def add_to_config(cls, section, name, value):\n \"\"\"\n Add a parameter to the shared ConfigParser object.\n \n :param section: parameter's section\n :param name: parameter's name\n :param value: parameter's value\n \"\"\"\n if value:\n if not cls._config.has_section(section):\n cls._config.add_section(section)\n cls._config.set(section, name, str(value))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef output(f):\n def wrapper(self, *args, **kwargs):\n try:\n text = kwargs.get('text') or args[0]\n except IndexError:\n text = True\n _ = f(self, *args, **kwargs)\n if text:\n return _\n elif _ is not None and isinstance(_, string_types):\n filename = \"{}.{}\".format(self.filename, f.__name__)\n while exists(filename):\n name, ext = splitext(filename)\n try:\n name, i = name.split('-')\n i = int(i) + 1\n except ValueError:\n i = 2\n filename = \"{}-{}\".format(name, i) + ext\n with open(filename, 'w') as out:\n out.write(_)\n return wrapper", "response": "This is a decorator that can be used to save a file or text."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef html(self, text=TEXT):\n self.logger.debug(\"Generating the HTML report{}...\"\n .format([\"\", \" (text only)\"][text]))\n html = []\n for piece in self._pieces:\n if isinstance(piece, string_types):\n html.append(markdown2.markdown(piece, extras=[\"tables\"]))\n elif isinstance(piece, Element):\n html.append(piece.html())\n return \"\\n\\n\".join(html)", "response": "Generate an HTML file from the report data."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef pdf(self, text=TEXT):\n self.logger.debug(\"Generating the PDF report...\")\n html = HTML(string=self.html())\n css_file = self.css or join(dirname(abspath(__file__)),\n \"{}.css\".format(self.theme))\n css = [css_file, CSS(string=PAGE_CSS % self.__dict__)]\n html.write_pdf(\"{}.pdf\".format(self.filename), stylesheets=css)", "response": "Generate a PDF file from the report data."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngenerates a CSV table from the table data.", "response": "def csv(self, text=TEXT, sep=',', index=True, float_fmt=\"%.2g\"):\n \"\"\" Generate a CSV table from the table data. \"\"\"\n return self._data.to_csv(sep=sep, index=index, float_format=float_fmt)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef md(self, text=TEXT, float_format=\"%.2g\"):\n cols = self._data.columns\n hl = pd.DataFrame([[\"---\"] * len(cols)], index=[\"---\"], columns=cols)\n df = pd.concat([hl, self._data])\n return df.to_csv(sep='|', index=True, float_format=float_format)", "response": "Generate Markdown from the table data."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngenerates an XML output from the report data.", "response": "def xml(self, text=TEXT):\n \"\"\" Generate an XML output from the report data. \"\"\"\n def convert(line):\n xml = \" \\n\"\n for f in line.index:\n xml += \" %s\\n\" % (f, line[f])\n xml += \" \\n\"\n return xml\n return \"\\n\" + '\\n'.join(self._data.apply(convert, axis=1)) + \\\n \"\""} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef from_argspec(argspec):\n attributes = getattr(argspec, \"args\", []) + getattr(argspec, \"keywords\", [])\n defaults = argspec.defaults or []\n\n arguments, keywords = [], {}\n\n attribute_list = (\n attributes[: -len(defaults)] if len(defaults) != 0 else attributes[:]\n )\n for name in attribute_list:\n if name == \"self\":\n continue\n typ = argspec.annotations.get(name)\n arguments.append(Argument(name, NoDefault, typ))\n\n if len(defaults) != 0:\n for name, default in zip(attributes[-len(defaults) :], defaults):\n typ = argspec.annotations.get(name)\n keywords[name] = Argument(name, default, typ)\n\n return FunctionSignature(arguments, keywords)", "response": "retrieve a FunctionSignature object from the argspec and the annotations passed."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngives a dictionary of arguments split them into args and kwargs", "response": "def split_args(self, arg_dict):\n \"\"\"\n given a dictionary of arguments, split them into\n args and kwargs\n\n note: this destroys the arg_dict passed. if you need it,\n create a copy first.\n \"\"\"\n pos_args = []\n for arg in self.args:\n pos_args.append(arg_dict[arg.name])\n del arg_dict[arg.name]\n return pos_args, arg_dict"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef std_input(prompt=\"\", style=None):\n p = ansi_wrap(prompt, **(style or {}))\n try:\n return raw_input(p).strip()\n except NameError:\n return input(p).strip()", "response": "A simple Python 2. 3 - compatible input method."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef user_input(prompt=\"\", choices=None, default=None, choices_str=\"\",\n required=False):\n \"\"\"\n Python2/3-compatible input method handling choices and default value.\n \n :param prompt: prompt message\n :param choices: list of possible choices or lambda function\n :param default: default value\n :param required: make non-null user input mandatory\n :return: handled user input\n \"\"\"\n if type(choices) in [list, tuple, set]:\n choices = list(map(str, choices))\n choices_str = \" {%s}\" % (choices_str or \\\n '|'.join(list(map(str, choices))))\n # consider choices of the form [\"(Y)es\", \"(N)o\"] ;\n # in this case, we want the choices to be ['y', 'n'] for the sake of\n # simplicity for the user\n m = list(map(lambda x: CHOICE_REGEX.match(x), choices))\n choices = [x.group(1).lower() if x else c for x, c in zip(m, choices)]\n # this way, if using [\"Yes\", \"No\"], choices will remain so\n _check = lambda v: v in choices\n elif is_lambda(choices):\n _check = choices\n else:\n _check = lambda v: True\n prompt += \"{}{}\\n\".format(choices_str, [\" [{}]\".format(default), \"\"]\\\n [default is None and required])\n user_input, first = None, True\n while not user_input:\n user_input = std_input([\"\", prompt][first] + \" >> \")\n first = False\n if type(choices) in [list, tuple, set]:\n choices = list(map(lambda x: x.lower(), choices))\n user_input = user_input.lower()\n if user_input == \"\" and default is not None and _check(default):\n return str(default)\n if user_input != \"\" and _check(user_input):\n return user_input\n if not required:\n return", "response": "Python 2. 3 compatible user input method handling choices and default value."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntake the args and kwargs that are passed them and format in a prototype style.", "response": "def pretty_format_args(*args, **kwargs):\n \"\"\"\n Take the args, and kwargs that are passed them and format in a\n prototype style.\n \"\"\"\n args = list([repr(a) for a in args])\n for key, value in kwargs.items():\n args.append(\"%s=%s\" % (key, repr(value)))\n return \"(%s)\" % \", \".join([a for a in args])"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nvalidate an XML file.", "response": "def validate_xml(file):\n \"\"\"Validate an XML file.\"\"\"\n max_file_size = current_app.config.get(\n 'PREVIEWER_MAX_FILE_SIZE_BYTES', 1 * 1024 * 1024)\n if file.size > max_file_size:\n return False\n\n with file.open() as fp:\n try:\n content = fp.read().decode('utf-8')\n xml.dom.minidom.parseString(content)\n return True\n except:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef preview(file):\n return render_template(\n 'invenio_previewer/xml_prismjs.html',\n file=file,\n content=render(file),\n js_bundles=['previewer_prism_js'],\n css_bundles=['previewer_prism_css'],\n )", "response": "Render appropiate template with embed flag."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_context(argspec, kwargs):\n if argspec.keywords is not None:\n return kwargs\n return dict((arg, kwargs[arg]) for arg in argspec.args if arg in kwargs)", "response": "Prepare a context for the serialization function."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get(self, obj, **kwargs):\n assert self.getter is not None, \"Getter accessor is not specified.\"\n if callable(self.getter):\n return self.getter(obj, **_get_context(self._getter_argspec, kwargs))\n\n assert isinstance(self.getter, string_types), \"Accessor must be a function or a dot-separated string.\"\n\n for attr in self.getter.split(\".\"):\n if isinstance(obj, dict):\n obj = obj[attr]\n else:\n obj = getattr(obj, attr)\n\n if callable(obj):\n return obj()\n\n return obj", "response": "Get an attribute from an object."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nset value for obj s attribute.", "response": "def set(self, obj, value):\n \"\"\"Set value for obj's attribute.\n\n :param obj: Result object or dict to assign the attribute to.\n :param value: Value to be assigned.\n \"\"\"\n assert self.setter is not None, \"Setter accessor is not specified.\"\n if callable(self.setter):\n return self.setter(obj, value)\n\n assert isinstance(self.setter, string_types), \"Accessor must be a function or a dot-separated string.\"\n\n def _set(obj, attr, value):\n if isinstance(obj, dict):\n obj[attr] = value\n else:\n setattr(obj, attr, value)\n return value\n\n path = self.setter.split(\".\")\n for attr in path[:-1]:\n obj = _set(obj, attr, {})\n\n _set(obj, path[-1], value)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets an attribute s accessor with the getter and the setter.", "response": "def accessor(self):\n \"\"\"Get an attribute's accessor with the getter and the setter.\n\n :return: `Accessor` instance.\n \"\"\"\n if isinstance(self.attr, Accessor):\n return self.attr\n\n if callable(self.attr):\n return Accessor(getter=self.attr)\n\n attr = self.attr or self.name\n return Accessor(getter=attr, setter=attr)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nserializing the attribute of the input data.", "response": "def serialize(self, value, **kwargs):\n \"\"\"Serialize the attribute of the input data.\n\n Gets the attribute value with accessor and converts it using the\n type serialization. Schema will place this serialized value into\n corresponding compartment of the HAL structure with the name of the\n attribute as a key.\n\n :param value: Value to get the attribute value from.\n :return: Serialized attribute value.\n \"\"\"\n if types.Type.is_type(self.attr_type):\n try:\n value = self.accessor.get(value, **kwargs)\n except (AttributeError, KeyError):\n if not hasattr(self, \"default\") and self.required:\n raise\n value = self.default() if callable(self.default) else self.default\n\n return self.attr_type.serialize(value, **_get_context(self._attr_type_serialize_argspec, kwargs))\n\n return self.attr_type"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef deserialize(cls, value, output=None, **kwargs):\n errors = []\n result = {}\n for attr in cls.__attrs__.values():\n try:\n result[attr.name] = attr.deserialize(value, **kwargs)\n except NotImplementedError:\n # Links don't support deserialization\n continue\n except ValueError as e:\n errors.append(exceptions.ValidationError(e, attr.name))\n except exceptions.ValidationError as e:\n e.attr = attr.name\n errors.append(e)\n except (KeyError, AttributeError):\n if attr.required:\n errors.append(exceptions.ValidationError(\"Missing attribute.\", attr.name))\n\n if errors:\n raise exceptions.ValidationError(errors)\n\n if output is None:\n return result\n for attr in cls.__attrs__.values():\n if attr.name in result:\n attr.accessor.set(output, result[attr.name])", "response": "Deserialize the HAL structure into the output value."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nparsing a 9 bytes packet in the Temperature format and returns a dictionary containing the data extracted.", "response": "def parse(self, data):\n \"\"\"Parse a 9 bytes packet in the Temperature format and return a\n dictionary containing the data extracted. An example of a return value\n would be:\n\n .. code-block:: python\n\n {\n 'id': \"0x2EB2\",\n 'packet_length': 8,\n 'packet_type': 80,\n 'packet_type_name': 'Temperature sensors',\n 'sequence_number': 0,\n 'packet_subtype': 1,\n 'packet_subtype_name': \"THR128/138, THC138\",\n 'temperature': 21.3,\n 'signal_level': 9,\n 'battery_level': 6,\n }\n\n :param data: bytearray to be parsed\n :type data: bytearray\n\n :return: Data dictionary containing the parsed values\n :rtype: dict\n \"\"\"\n\n self.validate_packet(data)\n\n id_ = self.dump_hex(data[4:6])\n # channel = data[5] TBC\n\n temperature = ((data[6] & 0x7f) * 256 + data[7]) / 10\n signbit = data[6] & 0x80\n if signbit != 0:\n temperature = -temperature\n\n sensor_specific = {\n 'id': id_,\n # 'channel': channel, TBC\n 'temperature': temperature\n }\n\n results = self.parse_header_part(data)\n results.update(RfxPacketUtils.parse_signal_and_battery(data[8]))\n results.update(sensor_specific)\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nparse a comma - separated list of ints.", "response": "def ints(l, ifilter=lambda x: x, idescr=None):\n \"\"\" Parses a comma-separated list of ints. \"\"\"\n if isinstance(l, string_types):\n if l[0] == '[' and l[-1] == ']':\n l = l[1:-1]\n l = list(map(lambda x: x.strip(), l.split(',')))\n try:\n l = list(map(ifilter, list(map(int, l))))\n except:\n raise ValueError(\"Bad list of {}integers\"\n .format(\"\" if idescr is None else idescr + \" \"))\n return l"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngiving a list of IP addresses return a list of all hosts in the network.", "response": "def ip_address_list(ips):\n \"\"\" IP address range validation and expansion. \"\"\"\n # first, try it as a single IP address\n try:\n return ip_address(ips)\n except ValueError:\n pass\n # then, consider it as an ipaddress.IPv[4|6]Network instance and expand it\n return list(ipaddress.ip_network(u(ips)).hosts())"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nvalidates and return a valid port number.", "response": "def port_number(port):\n \"\"\" Port number validation. \"\"\"\n try:\n port = int(port)\n except ValueError:\n raise ValueError(\"Bad port number\")\n if not 0 <= port < 2 ** 16:\n raise ValueError(\"Bad port number\")\n return port"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef port_number_range(prange):\n # first, try it as a normal port number\n try:\n return port_number(prange)\n except ValueError:\n pass\n # then, consider it as a range with the format \"x-y\" and expand it\n try:\n bounds = list(map(int, re.match(r'^(\\d+)\\-(\\d+)$', prange).groups()))\n if bounds[0] > bounds[1]:\n raise AttributeError()\n except (AttributeError, TypeError):\n raise ValueError(\"Bad port number range\")\n return list(range(bounds[0], bounds[1] + 1))", "response": "Validate and expansion of a port number range."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nextracts and converts the RFX common header part of all valid packets to a plain dictionary.", "response": "def parse_header_part(self, data):\n \"\"\"Extracts and converts the RFX common header part of all valid\n packets to a plain dictionary. RFX header part is the 4 bytes prior\n the sensor vendor specific data part.\n\n The RFX common header part contains respectively:\n - packet length\n - packet type\n - packet sub-type\n - sequence number\n\n :param data: bytearray of received data\n :type data: bytearray\n\n \"\"\"\n packet_length = data[0]\n packet_type = data[1]\n packet_subtype = data[2]\n sequence_number = data[3]\n return {\n 'packet_length': packet_length,\n 'packet_type': packet_type,\n 'packet_type_name': self.PACKET_TYPES.get(packet_type),\n 'packet_subtype': packet_subtype,\n 'packet_subtype_name': self.PACKET_SUBTYPES.get(packet_subtype),\n 'sequence_number': sequence_number\n }"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef load(self, data):\n self.loaded_at = datetime.utcnow()\n self.raw = data\n self.data = self.parse(data)\n return self.data", "response": "This method is used to load the raw data and start parsing the data."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef validate_packet(self, data):\n\n # Validate length.\n # The first byte in the packet should be equal to the number of\n # remaining bytes (i.e. length excluding the first byte).\n expected_length = data[0] + 1\n\n if len(data) != expected_length:\n raise InvalidPacketLength(\n \"Expected packet length to be %s bytes but it was %s bytes\"\n % (expected_length, len(data))\n )\n\n # Validate minimal length.\n # The packet contains at least the RFX header:\n # packet_length (1 byte) + packet_type (1 byte)\n # + packet_subtype (1 byte) + sequence_number (1 byte)\n if expected_length < 4:\n raise MalformedPacket(\n \"Expected packet length to be larger than 4 bytes but \\\n it was %s bytes\"\n % (len(data))\n )\n\n # Validate Packet Type.\n # This specifies the family of devices.\n # Check it is one of the supported packet types\n packet_type = data[1]\n\n if self.PACKET_TYPES and packet_type not in self.PACKET_TYPES:\n types = \",\".join(\"0x{:02x}\".format(pt) for pt in self.PACKET_TYPES)\n raise UnknownPacketType(\n \"Expected packet type to be one of [%s] but recieved %s\"\n % (types, packet_type)\n )\n\n # Validate Packet Subtype.\n # This specifies the sub-family of devices.\n # Check it is one of the supported packet subtypes for current type\n sub_type = data[2]\n\n if self.PACKET_SUBTYPES and sub_type not in self.PACKET_SUBTYPES:\n types = \\\n \",\".join(\"0x{:02x}\".format(pt) for pt in self.PACKET_SUBTYPES)\n raise UnknownPacketSubtype(\n \"Expected packet type to be one of [%s] but recieved %s\"\n % (types, sub_type))\n\n return True", "response": "Validate a packet against this packet handler and determine if it meets the requirements."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nenforcing an instance of the schematics class.", "response": "def _enforce_instance(model_or_class):\n \"\"\"\n It's a common mistake to not initialize a\n schematics class. We should handle that by just\n calling the default constructor.\n \"\"\"\n if isinstance(model_or_class, type) and issubclass(model_or_class, BaseType):\n return model_or_class()\n return model_or_class"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef render(file):\n fp = file.open()\n content = fp.read()\n fp.close()\n\n notebook = nbformat.reads(content.decode('utf-8'), as_version=4)\n\n html_exporter = HTMLExporter()\n html_exporter.template_file = 'basic'\n (body, resources) = html_exporter.from_notebook_node(notebook)\n return body, resources", "response": "Generate the result HTML."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef preview(file):\n body, resources = render(file)\n default_ipython_style = resources['inlining']['css'][1]\n return render_template(\n 'invenio_previewer/ipynb.html',\n file=file,\n content=body,\n style=default_ipython_style\n )", "response": "Render the IPython Notebook."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef transmute_route(app, fn, context=default_context):\n transmute_func = TransmuteFunction(fn)\n routes, handler = create_routes_and_handler(transmute_func, context)\n for r in routes:\n \"\"\"\n the route being attached is a great place to start building up a\n swagger spec. the SwaggerSpec object handles creating the\n swagger spec from transmute routes for you.\n\n almost all web frameworks provide some app-specific context\n that one can add values to. It's recommended to attach\n and retrieve the swagger spec from there.\n \"\"\"\n if not hasattr(app, SWAGGER_ATTR_NAME):\n setattr(app, SWAGGER_ATTR_NAME, SwaggerSpec())\n swagger_obj = getattr(app, SWAGGER_ATTR_NAME)\n swagger_obj.add_func(transmute_func, context)\n app.route(r, methods=transmute_func.methods)(handler)", "response": "This function will handle the transmute route for all the web frameworks. It will create the routes and handler for all the web frameworks."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_routes_and_handler(transmute_func, context):\n @wraps(transmute_func.raw_func)\n def handler():\n exc, result = None, None\n try:\n args, kwargs = ParamExtractorFlask().extract_params(\n context, transmute_func, request.content_type\n )\n result = transmute_func(*args, **kwargs)\n except Exception as e:\n exc = e\n \"\"\"\n attaching the traceack is done for you in Python 3, but\n in Python 2 the __traceback__ must be\n attached to the object manually.\n \"\"\"\n exc.__traceback__ = sys.exc_info()[2]\n \"\"\"\n transmute_func.process_result handles converting\n the response from the function into the response body,\n the status code that should be returned, and the\n response content-type.\n \"\"\"\n response = transmute_func.process_result(\n context, result, exc, request.content_type\n )\n return Response(\n response[\"body\"],\n status=response[\"code\"],\n mimetype=response[\"content-type\"],\n headers=response[\"headers\"]\n )\n return (\n _convert_paths_to_flask(transmute_func.paths),\n handler\n )", "response": "Creates a route and handler for the given Transmute function."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _convert_paths_to_flask(transmute_paths):\n paths = []\n for p in transmute_paths:\n paths.append(p.replace(\"{\", \"<\").replace(\"}\", \">\"))\n return paths", "response": "converts a list of transmute - core s path syntax into flask s path syntax"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nadds a swagger page to the app.", "response": "def add_swagger(app, json_route, html_route, **kwargs):\n \"\"\"\n add a swagger html page, and a swagger.json generated\n from the routes added to the app.\n \"\"\"\n spec = getattr(app, SWAGGER_ATTR_NAME)\n if spec:\n spec = spec.swagger_definition(**kwargs)\n else:\n spec = {}\n encoded_spec = json.dumps(spec).encode(\"UTF-8\")\n\n @app.route(json_route)\n def swagger():\n return Response(\n encoded_spec,\n # we allow CORS, so this can be requested at swagger.io\n headers={\"Access-Control-Allow-Origin\": \"*\"},\n content_type=\"application/json\",\n )\n\n # add the statics\n static_root = get_swagger_static_root()\n swagger_body = generate_swagger_html(\n STATIC_PATH, json_route\n ).encode(\"utf-8\")\n\n @app.route(html_route)\n def swagger_ui():\n return Response(swagger_body, content_type=\"text/html\")\n\n # the blueprint work is the easiest way to integrate a static\n # directory into flask.\n blueprint = Blueprint('swagger', __name__, static_url_path=STATIC_PATH,\n static_folder=static_root)\n app.register_blueprint(blueprint)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _log_enabled_protocols(self, flags, protocols):\n\n enabled, disabled = [], []\n\n for procol, flag in sorted(zip(protocols, flags)):\n\n if flag == '1':\n enabled.append(procol)\n status = 'Enabled'\n else:\n disabled.append(procol)\n status = 'Disabled'\n\n message = \"{0:21}: {1}\".format(procol, status)\n self.log.info(message)\n\n return enabled, disabled", "response": "Log the status of each protocol where 1 is the enabled and 0 is the disabled ones."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef parse(self, data):\n\n self.validate_packet(data)\n\n packet_length = data[0]\n packet_type = data[1]\n sub_type = data[2]\n sequence_number = data[3]\n command_type = data[4]\n transceiver_type = data[5]\n transceiver_type_text = _MSG1_RECEIVER_TYPE.get(data[5])\n firmware_version = data[6]\n\n flags = self._int_to_binary_list(data[7])\n flags.extend(self._int_to_binary_list(data[8]))\n flags.extend(self._int_to_binary_list(data[9]))\n\n enabled, disabled = self._log_enabled_protocols(flags, PROTOCOLS)\n\n return {\n 'packet_length': packet_length,\n 'packet_type': packet_type,\n 'packet_type_name': self.PACKET_TYPES.get(packet_type),\n 'sequence_number': sequence_number,\n 'sub_type': sub_type,\n 'sub_type_name': self.PACKET_SUBTYPES.get(sub_type),\n 'command_type': command_type,\n 'transceiver_type': transceiver_type,\n 'transceiver_type_text': transceiver_type_text,\n 'firmware_version': firmware_version,\n 'enabled_protocols': enabled,\n 'disabled_protocols': disabled,\n }", "response": "Parse a 13 byte packet in the Status format."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set_time_items(glob):\n a = glob['args']\n l = glob['logger']\n\n class __TimeManager(object):\n def __init__(self):\n c = a._collisions\n self._stats = getattr(a, c.get(\"stats\") or \"stats\", False)\n self._timings = getattr(a, c.get(\"timings\") or \"timings\", False)\n self.enabled = self._stats or self._timings\n self.last = self.start = time.time()\n self.times = []\n \n def stats(self):\n end = time.time()\n b = \"\"\n for d, s, e in self.times:\n b += \"\\n{}\\n> {} seconds\".format(d, e - s)\n l.time(\"Total time: {} seconds{}\".format(end - self.start, b))\n \n glob['time_manager'] = manager = __TimeManager()\n\n def _take_time(start=None, descr=None):\n t = manager.last = time.time()\n if start is not None and descr is not None:\n manager.times.append((descr, float(start), float(t)))\n return t - (start or 0)\n\n class Timer(object):\n class TimeoutError(Exception):\n pass # TimeoutError is not handled in Python 2\n \n def __init__(self, description=None, message=TO_MSG, timeout=None,\n fail_on_timeout=False):\n self.fail = fail_on_timeout\n self.id = len(manager.times)\n self.descr = \"#\" + str(self.id) + \\\n (\": \" + (description or \"\")).rstrip(\": \")\n self.message = message\n self.start = _take_time()\n self.timeout = timeout\n\n def __enter__(self):\n if manager.enabled:\n if self.timeout is not None:\n signal.signal(signal.SIGALRM, self._handler)\n signal.alarm(self.timeout)\n if manager._timings and self.descr:\n l.time(self.descr)\n return self\n \n def __exit__(self, exc_type, exc_value, exc_traceback):\n if manager.enabled:\n d = _take_time(self.start, self.descr)\n if manager._timings:\n l.time(\"> Time elapsed: {} seconds\".format(d))\n if self.timeout is not None:\n if self.fail and exc_type is Timer.TimeoutError:\n return True\n\n def _handler(self, signum, frame):\n raise Timer.TimeoutError(self.message)\n \n glob['Timer'] = Timer\n \n def get_time(message=None, start=manager.start):\n if manager._timings:\n l.time(\"> {}: {} seconds\".format(message or \"Time elapsed since \"\n \"execution start\", _take_time(start)))\n \n glob['get_time'] = get_time\n \n def get_time_since_last(message=None):\n get_time(message or \"Time elapsed since last measure\", manager.last)\n \n glob['get_time_since_last'] = get_time_since_last", "response": "This function prepares the benchmark items for inclusion in main script s global scope dictionary."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncapturing the variable in the parameters dictionary", "response": "def _capture_variable(iterator, parameters):\n \"\"\"\n return the replacement string.\n this assumes the preceeding {{ has already been\n popped off.\n \"\"\"\n key = \"\"\n next_c = next(iterator)\n while next_c != \"}\":\n key += next_c\n next_c = next(iterator)\n # remove the final \"}\"\n next(iterator)\n return parameters[key]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef rgb_distance(rgb1, rgb2):\n '''\n Calculate the distance between two RGB sequences.\n '''\n return sum(map(lambda c: (c[0] - c[1]) ** 2,\n zip(rgb1, rgb2)))", "response": "Calculate the distance between two RGB sequences."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef rgb_reduce(r, g, b, mode=8):\n '''\n Convert an RGB colour to 8 or 16 colour ANSI graphics.\n '''\n colours = ANSI_COLOURS[:mode]\n matches = [(rgb_distance(c, map(int, [r, g, b])), i)\n for i, c in enumerate(colours)]\n matches.sort()\n return sequence('m')(str(30 + matches[0][1]))", "response": "Convert an RGB colour to 8 or 16 colour ANSI graphics.\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef rgb256(r, g, b):\n '''\n Convert an RGB colour to 256 colour ANSI graphics.\n '''\n grey = False\n poss = True\n step = 2.5\n\n while poss: # As long as the colour could be grey scale\n if r < step or g < step or b < step:\n grey = r < step and g < step and b < step\n poss = False\n\n step += 42.5\n\n if grey:\n colour = 232 + int(float(sum([r, g, b]) / 33.0))\n else:\n colour = sum([16] + [int (6 * float(val) / 256) * mod\n for val, mod in ((r, 36), (g, 6), (b, 1))])\n\n return sequence('m', fields=3)(38, 5, colour)", "response": "Convert an RGB colour to 256 colour ANSI graphics."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef uri(self):\n return url_for(\n '.{0}_files'.format(self.pid.pid_type),\n pid_value=self.pid.pid_value,\n filename=self.file.key)", "response": "Get file download link."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if the file has one of the extensions.", "response": "def has_extensions(self, *exts):\n \"\"\"Check if file has one of the extensions.\"\"\"\n file_ext = splitext(self.filename)[1]\n file_ext = file_ext.lower()\n\n for e in exts:\n if file_ext == e:\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef render(file):\n with file.open() as fp:\n encoding = detect_encoding(fp, default='utf-8')\n file_content = fp.read().decode(encoding)\n json_data = json.loads(file_content, object_pairs_hook=OrderedDict)\n return json.dumps(json_data, indent=4, separators=(',', ': '))", "response": "Pretty print the JSON file for rendering."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef validate_json(file):\n max_file_size = current_app.config.get(\n 'PREVIEWER_MAX_FILE_SIZE_BYTES', 1 * 1024 * 1024)\n if file.size > max_file_size:\n return False\n\n with file.open() as fp:\n try:\n json.loads(fp.read().decode('utf-8'))\n return True\n except:\n return False", "response": "Validate a JSON file."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef optspace(edm_missing, rank, niter=500, tol=1e-6, print_out=False):\n from .opt_space import opt_space\n N = edm_missing.shape[0]\n X, S, Y, __ = opt_space(edm_missing, r=rank, niter=niter,\n tol=tol, print_out=print_out)\n edm_complete = X.dot(S.dot(Y.T))\n edm_complete[range(N), range(N)] = 0.0\n return edm_complete", "response": "Complete and denoise EDM using OptSpace algorithm."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncomplete and denoise EDM using rank alternation.", "response": "def rank_alternation(edm_missing, rank, niter=50, print_out=False, edm_true=None):\n \"\"\"Complete and denoise EDM using rank alternation.\n\n Iteratively impose rank and strucutre to complete marix entries\n\n :param edm_missing: EDM with 0 where no measurement was taken.\n :param rank: expected rank of complete EDM.\n :param niter: maximum number of iterations.\n :param edm: if given, the relative EDM error is tracked. \n\n :return: Completed matrix and array of errors (empty if no true edm is given).\n The matrix is of the correct structure, but might not have the right measured entries.\n\n \"\"\"\n from pylocus.basics import low_rank_approximation\n errs = []\n N = edm_missing.shape[0]\n edm_complete = edm_missing.copy()\n edm_complete[edm_complete == 0] = np.mean(edm_complete[edm_complete > 0])\n for i in range(niter):\n # impose matrix rank\n edm_complete = low_rank_approximation(edm_complete, rank)\n\n # impose known entries\n edm_complete[edm_missing > 0] = edm_missing[edm_missing > 0]\n\n # impose matrix structure\n edm_complete[range(N), range(N)] = 0.0\n edm_complete[edm_complete < 0] = 0.0\n edm_complete = 0.5 * (edm_complete + edm_complete.T)\n\n if edm_true is not None:\n err = np.linalg.norm(edm_complete - edm_true)\n errs.append(err)\n return edm_complete, errs"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncompletes and denoise EDM using semidefinite relaxation.", "response": "def semidefinite_relaxation(edm_missing, lamda, W=None, print_out=False, **kwargs):\n \"\"\"Complete and denoise EDM using semidefinite relaxation.\n\n Returns solution to the relaxation of the following problem: \n D = argmin || W * (D - edm_missing) || \n s.t. D is EDM\n where edm_missing is measured matrix, W is a weight matrix, and * is pointwise multiplication. \n\n Refer to paper \"Euclidean Distance Matrices - Essential Theory, Algorithms and Applications\", \n Algorithm 5, for details. (https://www.doi.org/%2010.1109/MSP.2015.2398954)\n\n :param edm_missing: EDM with 0 where no measurement was taken. \n :param lamda: Regularization parameter. \n :param W: Optional mask. If no mask is given, a binary mask is created based on missing elements of edm_missing. \n If mask is given \n :param kwargs: more options passed to the solver. See cvxpy documentation for all options. \n \"\"\" \n from .algorithms import reconstruct_mds\n\n def kappa(gram):\n n = len(gram)\n e = np.ones(n)\n return np.outer(np.diag(gram), e) + np.outer(e, np.diag(gram).T) - 2 * gram\n\n def kappa_cvx(gram, n):\n e = np.ones((n, 1))\n return reshape(diag(gram), (n, 1)) * e.T + e * reshape(diag(gram), (1, n)) - 2 * gram\n\n method = kwargs.pop('method', 'maximize')\n options = {'solver': 'CVXOPT'}\n options.update(kwargs)\n\n if W is None:\n W = (edm_missing > 0)\n else:\n W[edm_missing == 0] = 0.0\n\n n = edm_missing.shape[0]\n V = np.c_[-np.ones((n - 1, 1)) / np.sqrt(n), np.eye(n - 1) -\n np.ones((n - 1, n - 1)) / (n + np.sqrt(n))].T\n\n H = Variable((n - 1, n - 1), PSD=True)\n G = V * H * V.T # * is overloaded\n edm_optimize = kappa_cvx(G, n)\n\n if method == 'maximize':\n obj = Maximize(trace(H) - lamda *\n norm(multiply(W, (edm_optimize - edm_missing)), p=1))\n # TODO: add a reference to paper where \"minimize\" is used instead of maximize. \n elif method == 'minimize':\n obj = Minimize(trace(H) + lamda *\n norm(multiply(W, (edm_optimize - edm_missing)), p=1))\n\n prob = Problem(obj)\n\n total = prob.solve(**options)\n if print_out:\n print('total cost:', total)\n print('SDP status:', prob.status)\n\n if H.value is not None:\n Gbest = V.dot(H.value).dot(V.T)\n if print_out:\n print('eigenvalues:', np.sum(np.linalg.eigvals(Gbest)[2:]))\n edm_complete = kappa(Gbest)\n else:\n edm_complete = edm_missing\n\n if (print_out):\n if H.value is not None:\n print('trace of H:', np.trace(H.value))\n print('other cost:', lamda *\n norm(multiply(W, (edm_complete - edm_missing)), p=1).value)\n\n return np.array(edm_complete)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef completion_acd(edm, X0, W=None, tol=1e-6, sweeps=3):\n from .algorithms import reconstruct_acd\n Xhat, costs = reconstruct_acd(edm, X0, W, tol=tol, sweeps=sweeps)\n return get_edm(Xhat)", "response": "Complete an arbitrary denoise EDM using an iterative decent algorithm."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef completion_dwmds(edm, X0, W=None, tol=1e-10, sweeps=100):\n from .algorithms import reconstruct_dwmds\n Xhat, costs = reconstruct_dwmds(edm, X0, W, n=1, tol=tol, sweeps=sweeps)\n return get_edm(Xhat)", "response": "Complete an EDM using DWMDS."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts the transmute_path to a tornado compatible regex and return a tornado. web. URLSpec object.", "response": "def url_spec(transmute_path, handler, *args, **kwargs):\n \"\"\"\n convert the transmute_path\n to a tornado compatible regex,\n and return a tornado url object.\n \"\"\"\n p = _to_tornado_pattern(transmute_path)\n for m in METHODS:\n method = getattr(handler, m)\n if hasattr(method, \"transmute_func\"):\n method.transmute_func.paths.add(transmute_path)\n return tornado.web.URLSpec(\n p, handler,\n *args, **kwargs\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ninitialize points according to predefined modes.", "response": "def set_points(self, mode='', points=None, range_=RANGE, size=1):\n \"\"\" Initialize points according to predefined modes.\n\n :param range_:[xmin, xmax, ymin, ymax], range of point sets\n \"\"\"\n if mode == 'last':\n if points is None:\n print('Error: empty last point specification given.')\n return\n tol = 0.1\n [i, j, k] = points\n alpha = 2.0 * np.random.rand(1) + 0.5\n beta = 2.0 * np.random.rand(1) + 0.5\n if i >= 0 and j < 0 and k < 0:\n # at one corner of triangle\n assert i < 3\n other = np.delete(np.arange(3), i)\n u = (self.points[i, :] - self.points[other[0], :]\n ) / np.linalg.norm(\n self.points[i, :] - self.points[other[0], :])\n v = (self.points[i, :] - self.points[other[1], :]\n ) / np.linalg.norm(\n self.points[i, :] - self.points[other[1], :])\n self.points[-1, :] = self.points[i, :] + alpha * u + beta * v\n elif i >= 0 and j >= 0 and k < 0:\n found = False\n safety_it = 0\n while not found:\n alpha = np.random.uniform(tol, 1 - tol)\n beta = 1.0 - alpha\n gamma = 2 * np.random.rand(1) + tol\n assert j < 3\n other = np.delete(np.arange(3), (i, j))\n u = (\n self.points[i, :] - self.points[other, :]\n ) # /np.linalg.norm(self.points[i,:] - self.points[other,:])\n v = (\n self.points[j, :] - self.points[other, :]\n ) # /np.linalg.norm(self.points[j,:] - self.points[other,:])\n self.points[-1, :] = (1.0 + gamma) * (\n self.points[other, :] + alpha * u + beta * v)\n #check if new direction lies between u and v.\n new_direction = self.points[-1, :] - self.points[other, :]\n new_direction = new_direction.reshape(\n (-1, )) / np.linalg.norm(new_direction)\n u = u.reshape((-1, )) / np.linalg.norm(u)\n v = v.reshape((-1, )) / np.linalg.norm(v)\n #print('{} + {} = {}'.format(acos(np.dot(new_direction,u)),acos(np.dot(new_direction,v)),acos(np.dot(u,v))))\n if abs(\n acos(np.dot(new_direction, u)) +\n acos(np.dot(new_direction, v)) -\n acos(np.dot(u, v))) < 1e-10:\n found = True\n safety_it += 1\n if safety_it > 100:\n print('Error: nothing found after 100 iterations.')\n return\n elif i >= 0 and j >= 0 and k >= 0:\n # inside triangle\n assert k < 3\n found = False\n safety_it = 0\n while not found:\n alpha = np.random.rand(1) + tol\n beta = np.random.rand(1) + tol\n other = np.delete(np.arange(3), i)\n u = self.points[other[0], :] - self.points[i, :]\n v = self.points[other[1], :] - self.points[i, :]\n temptative_point = self.points[i, :] + alpha * u + beta * v\n vjk = self.points[other[1], :] - self.points[other[0], :]\n njk = [vjk[1], -vjk[0]]\n if (np.dot(self.points[j, :] - self.points[i, :], njk) >\n 0) != (np.dot(temptative_point - self.points[j, :],\n njk) > 0):\n self.points[-1, :] = temptative_point\n found = True\n safety_it += 1\n if safety_it > 100:\n print('Error: nothing found after 100 iterations.')\n return\n elif i < 0 and j < 0 and k < 0:\n x = range_[0] + (\n range_[1] - range_[0]) * np.random.rand(1)\n y = range_[2] + (\n range_[1] - range_[0]) * np.random.rand(1)\n self.points[-1, :] = [x, y]\n else:\n print(\"Error: non-valid arguments.\")\n elif mode == 'random':\n \"\"\" Create N uniformly distributed points in [0, size] x [0, size]\n \"\"\"\n self.points = np.random.uniform(0, size, (self.N, self.d))\n elif mode == 'normal':\n self.points = np.random.normal(0, size, (self.N, self.d))\n elif mode == 'circle':\n from math import cos, sin\n x_range = size / 2.0\n y_range = size / 2.0\n c = np.array((x_range, y_range))\n r = 0.9 * min(x_range, y_range)\n theta = 2 * pi / self.N\n for i in range(self.N):\n theta_tot = i * theta\n self.points[i, :] = c + np.array(\n (r * cos(theta_tot), r * sin(theta_tot)))\n elif mode == 'set':\n \"\"\"\n Place points according to hard coded rule.\n \"\"\"\n if self.N == 3:\n x = [-1.0, 1.0, 0.0]\n y = [-1.0, -1.0, 1.0]\n elif self.N == 4:\n x = [-1.0, 1.0, 0.0, 0.0]\n y = [-1.0, -1.0, 1.0, 0.0]\n elif self.N == 5:\n x = [-0.0, 1.5, 1.5, -0.0, -1.0]\n y = [-1.0, -1.0, 1.0, 1.0, 0.0]\n else:\n print(\"Error: No rule defined for N = \", self.N)\n return\n self.points = np.c_[x, y]\n elif mode == 'geogebra':\n if self.N == 4:\n self.points = np.array(((1.5, 1.8), (7.9, 2.5), (2.3, 5.1),\n (3.34, -1.36)))\n elif self.N == 5:\n self.points = np.array(((1.5, 1.8), (7.9, 2.5), (2.3, 5.1),\n (3.34, -1.36), (5, 1.4)))\n else:\n print(\"Error: No rule defined for N = \", self.N)\n elif mode == '':\n if points is None:\n raise NotImplementedError(\"Need to give either mode or points.\")\n else:\n self.points = points\n self.N, self.d = points.shape\n\n self.init()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the set of inner angles between 0 and pi and the corners corresponding to each entry of theta.", "response": "def create_theta(self):\n \"\"\"\n Returns the set of inner angles (between 0 and pi)\n reconstructed from point coordinates.\n Also returns the corners corresponding to each entry of theta.\n \"\"\"\n import itertools\n from pylocus.basics_angles import from_0_to_pi\n theta = np.empty((self.M, ))\n corners = np.empty((self.M, 3))\n k = 0\n indices = np.arange(self.N)\n for triangle in itertools.combinations(indices, 3):\n for counter, idx in enumerate(triangle):\n corner = idx\n other = np.delete(triangle, counter)\n corners[k, :] = [corner, other[0], other[1]]\n theta[k] = self.get_inner_angle(corner, other)\n theta[k] = from_0_to_pi(theta[k])\n if DEBUG:\n print(self.abs_angles[corner, other[0]],\n self.abs_angles[corner, other[1]])\n print('theta', corners[k, :], theta[k])\n k = k + 1\n inner_angle_sum = theta[k - 1] + theta[k - 2] + theta[k - 3]\n assert abs(inner_angle_sum - pi) < 1e-10, \\\n 'inner angle sum: {} {} {}'.format(\n triangle, inner_angle_sum, (theta[k - 1], theta[k - 2], theta[k - 3]))\n self.theta = theta\n self.corners = corners\n return theta, corners"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating angles theta_ik and theta_jk theta produce point Pk.", "response": "def get_orientation(k, i, j):\n from pylocus.basics_angles import from_0_to_2pi\n \"\"\"calculate angles theta_ik and theta_jk theta produce point Pk.\n Should give the same as get_absolute_angle! \"\"\"\n theta_ij = own.abs_angles[i, j]\n theta_ji = own.abs_angles[j, i]\n\n # complicated\n xi = own.points[i, 0]\n xj = own.points[j, 0]\n yi = own.points[i, 1]\n yj = own.points[j, 1]\n w = np.array([yi - yj, xj - xi])\n test = np.dot(own.points[k, :] - own.points[i, :], w) > 0\n\n # more elegant\n theta_ik = truth.abs_angles[i, k]\n diff = from_0_to_2pi(theta_ik - theta_ij)\n test2 = (diff > 0 and diff < pi)\n assert (test == test2), \"diff: %r, scalar prodcut: %r\" % (diff, np.dot(\n own.points[k, :] - own.points[i, :], w))\n\n thetai_jk = truth.get_theta(i, j, k)\n thetaj_ik = truth.get_theta(j, i, k)\n if test:\n theta_ik = theta_ij + thetai_jk\n theta_jk = theta_ji - thetaj_ik\n else:\n theta_ik = theta_ij - thetai_jk\n theta_jk = theta_ji + thetaj_ik\n theta_ik = from_0_to_2pi(theta_ik)\n theta_jk = from_0_to_2pi(theta_jk)\n return theta_ik, theta_jk"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_indices(self, k):\n indices_rays = []\n indices_triangles = []\n corners_rays = []\n angles_rays = []\n for t, triangle in enumerate(self.corners):\n if triangle[0] == k:\n indices_rays.append(t)\n corners_rays.append(triangle)\n angles_rays.append(self.theta[t])\n else:\n indices_triangles.append(t)\n np_corners_rays = np.vstack(corners_rays)\n np_angles_rays = np.vstack(angles_rays).reshape((-1, ))\n return indices_rays, indices_triangles, np_corners_rays, np_angles_rays", "response": "Get indices of theta vector that have k as first corner."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_G(self, k, add_noise=True):\n G = np.ones((self.N - 1, self.N - 1))\n if (add_noise):\n noise = pi * 0.1 * np.random.rand(\n (self.N - 1) * (self.N - 1)).reshape((self.N - 1, self.N - 1))\n other_indices = np.delete(range(self.N), k)\n for idx, i in enumerate(other_indices):\n for jdx, j in enumerate(other_indices):\n if (add_noise and\n i != j): # do not add noise on diagonal elements.\n thetak_ij = self.get_inner_angle(k,\n (i, j)) + noise[idx, jdx]\n else:\n thetak_ij = self.get_inner_angle(k, (i, j))\n G[idx, jdx] = cos(thetak_ij)\n G[jdx, idx] = cos(thetak_ij)\n return G", "response": "get G matrix from angles."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget all convex polygones of size m", "response": "def get_convex_polygons(self, m, print_out=False):\n \"\"\"\n :param m: size of polygones (number of corners)\n \n :return: (ordered) indices of all convex polygones of size m.\n \"\"\"\n convex_polygons = []\n for corners in itertools.combinations(np.arange(self.N), m):\n p = np.zeros(m, np.uint)\n p[0] = corners[0]\n left = corners[1:]\n # loop through second corners\n for i, second in enumerate(corners[1:m - 1]):\n p[1] = second\n left = np.delete(corners, (0, i + 1))\n for j, last in enumerate(corners[i + 2:]):\n left = np.delete(corners, (0, i + 1, j + i + 2))\n p[-1] = last\n # loop through all permutations of left corners.\n for permut in itertools.permutations(left):\n p[2:-1] = permut\n sum_theta = 0\n # sum over all inner angles.\n for k in range(m):\n sum_theta += self.get_inner_angle(\n p[1], (p[0], p[2]))\n p = np.roll(p, 1)\n angle = sum_theta\n sum_angle = (m - 2) * pi\n if (abs(angle - sum_angle) < 1e-14 or\n abs(angle) < 1e-14):\n if (print_out):\n print(\"convex polygon found: \", p)\n convex_polygons.append(p.copy())\n # elif (angle < sum_angle):\n # if (print_out): print(\"non convex polygon found:\",p,angle)\n elif (angle > sum_angle):\n if (print_out):\n print(\"oops\")\n return convex_polygons"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget the constraints for the polygones", "response": "def get_polygon_constraints(self,\n range_polygones=range(3, 5),\n print_out=False):\n \"\"\"\n :param range_polygones: list of numbers of polygones to test.\n \n :return A, b: the constraints on the theta-vector of the form A*theta = b\n \"\"\"\n rows_A = []\n rows_b = []\n for m in range_polygones:\n if (print_out):\n print('checking {}-polygones'.format(m))\n polygons = self.get_convex_polygons(m)\n row_A, row_b = self.get_polygon_constraints_m(polygons, print_out)\n rows_A.append(row_A)\n rows_b.append(row_b)\n self.A = np.vstack(rows_A)\n self.b = np.hstack(rows_b)\n return self.A, self.b"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_polygon_constraints_m(self, polygons_m, print_out=False):\n rows_b = []\n rows_A = []\n m = len(polygons_m[0])\n rows_b.append((m - 2) * pi * np.ones(\n len(polygons_m), ))\n for p in polygons_m:\n row = np.zeros((self.theta.shape[0], ))\n for k in range(m):\n index = get_index(self.corners, p[1], (p[0], p[2]))\n row[index] = 1\n p = np.roll(p, 1)\n assert np.sum(row) == m\n rows_A.append(row)\n A = np.vstack(rows_A)\n b = np.hstack(rows_b)\n num_constraints = A.shape[0]\n A_repeat = np.repeat(A.astype(bool), 3).reshape((1, -1))\n corners = self.corners.reshape((1, -1))\n corners_tiled = np.tile(corners, num_constraints)\n if (print_out):\n print('shape of A {}'.format(A.shape))\n if (print_out):\n print('chosen angles m={}:\\n{}'.format(m, (corners_tiled)[A_repeat]\n .reshape((-1, m * 3))))\n if (print_out):\n print('{}-polygones: {}'.format(m, rows_A))\n self.A = A\n self.b = b\n return A, b", "response": "returns the constraints A b for each of the given set of polygones."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets linear constraints on KE matrix.", "response": "def get_KE_constraints(self):\n \"\"\"Get linear constraints on KE matrix.\n \"\"\"\n C2 = np.eye(self.m)\n C2 = C2[:self.m - 2, :]\n to_be_deleted = []\n for idx_vij_1 in range(self.m - 2):\n idx_vij_2 = idx_vij_1 + 1\n C2[idx_vij_1, idx_vij_2] = -1\n i1 = np.where(self.C[idx_vij_1, :] == 1)[0][0]\n i2 = np.where(self.C[idx_vij_2, :] == 1)[0][0]\n j = np.where(self.C[idx_vij_1, :] == -1)[0][0]\n if i1 == i2:\n i = i1\n k = np.where(self.C[idx_vij_2, :] == -1)[0][0]\n i_indices = self.C[:, j] == 1\n j_indices = self.C[:, k] == -1\n idx_vij_3 = np.where(np.bitwise_and(\n i_indices, j_indices))[0][0]\n #print('v{}{}, v{}{}, v{}{}\\n{} {} {}'.format(j,i,k,i,k,j,idx_vij_1,idx_vij_2,idx_vij_3))\n C2[idx_vij_1, idx_vij_3] = 1\n else:\n #print('v{}{}, v{}{} not considered.'.format(j,i1,j,i2))\n to_be_deleted.append(idx_vij_1)\n C2 = np.delete(C2, to_be_deleted, axis=0)\n b = np.zeros((C2.shape[0], 1))\n return C2, b"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nparse a 12 byte packet in the Lighting2 format and returns a dictionary containing the parsed data.", "response": "def parse(self, data):\n \"\"\"Parse a 12 bytes packet in the Lighting2 format and return a\n dictionary containing the data extracted. An example of a return value\n would be:\n\n .. code-block:: python\n\n {\n 'id': \"0x111F342\",\n 'packet_length': 10,\n 'packet_type': 17,\n 'packet_type_name': 'Humidity sensors',\n 'sequence_number': 19,\n 'packet_subtype': 0,\n 'packet_subtype_name': \"AC\",\n 'unit_code': 10,\n 'command': 1,\n 'command_text': \"Off\",\n 'level': 7,\n 'signal_level': 9,\n }\n\n :param data: bytearray to be parsed\n :type data: bytearray\n\n :return: Data dictionary containing the parsed values\n :rtype: dict\n \"\"\"\n\n self.validate_packet(data)\n\n results = self.parse_header_part(data)\n sub_type = results['packet_subtype']\n\n id_ = self.dump_hex(data[4:8])\n\n unit_code = data[8]\n command = data[9]\n command_text = SUB_TYPE_COMMANDS.get(sub_type, {}).get(command)\n dim_level = DIM_LEVEL_TO_PERCENT.get(data[10], '--??--')\n\n sensor_specific = {\n 'id': id_,\n 'unit_code': unit_code,\n 'command': command,\n 'command_text': command_text,\n 'dim_level': dim_level\n }\n\n results.update(RfxPacketUtils.parse_signal_upper(data[11]))\n results.update(sensor_specific)\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts an array of 6 bytes to a 48bit integer.", "response": "def _bytes_to_uint_48(self, bytes_):\n \"\"\"Converts an array of 6 bytes to a 48bit integer.\n\n :param data: bytearray to be converted to a 48bit integer\n :type data: bytearray\n\n :return: the integer\n :rtype: int\n \"\"\"\n return ((bytes_[0] * pow(2, 40)) + (bytes_[1] * pow(2, 32)) +\n (bytes_[2] * pow(2, 24)) + (bytes_[3] << 16) +\n (bytes_[4] << 8) + bytes_[4])"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef parse(self, data):\n\n self.validate_packet(data)\n\n TOTAL_DIVISOR = 223.666\n\n id_ = self.dump_hex(data[4:6])\n count = data[6]\n instant = data[7:11]\n total = data[11:16]\n\n current_watts = self._bytes_to_uint_32(instant)\n total_watts = self._bytes_to_uint_48(total) / TOTAL_DIVISOR\n\n sensor_specific = {\n 'count': count,\n 'current_watts': current_watts,\n 'id': id_,\n 'total_watts': total_watts\n }\n\n results = self.parse_header_part(data)\n results.update(RfxPacketUtils.parse_signal_and_battery(data[17]))\n results.update(sensor_specific)\n\n return results", "response": "Parses a 18 bytes packet in the Electricity format and returns a dictionary containing the data extracted."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_key_from_path_and_site(path, site_id):\n '''\n cache key has to be < 250 chars to avoid memcache.Client.MemcachedKeyLengthError\n The best algoritm is SHA-224 whose output (224 chars) respects this limitations\n '''\n key = '{}_{}'.format(path, site_id)\n key = hashlib.sha224(key.encode('utf-8')).hexdigest()\n return key", "response": "Get the cache key from a path and site ID."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef validate_csv(file):\n try:\n # Detect encoding and dialect\n with file.open() as fp:\n encoding = detect_encoding(fp, default='utf-8')\n sample = fp.read(\n current_app.config.get('PREVIEWER_CSV_VALIDATION_BYTES', 1024))\n delimiter = csv.Sniffer().sniff(sample.decode(encoding)).delimiter\n is_valid = True\n except Exception as e:\n current_app.logger.debug(\n 'File {0} is not valid CSV: {1}'.format(file.uri, e))\n encoding = ''\n delimiter = ''\n is_valid = False\n\n return {\n 'delimiter': delimiter,\n 'encoding': encoding,\n 'is_valid': is_valid\n }", "response": "Return dialect information about given csv file."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nrenders appropiate template with embed flag.", "response": "def preview(file):\n \"\"\"Render appropiate template with embed flag.\"\"\"\n file_info = validate_csv(file)\n return render_template(\n 'invenio_previewer/csv_bar.html',\n file=file,\n delimiter=file_info['delimiter'],\n encoding=file_info['encoding'],\n js_bundles=current_previewer.js_bundles + ['previewer_csv_js'],\n css_bundles=current_previewer.css_bundles,\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef make_tree(file):\n max_files_count = current_app.config.get('PREVIEWER_ZIP_MAX_FILES', 1000)\n tree = {'type': 'folder', 'id': -1, 'children': {}}\n\n try:\n with file.open() as fp:\n zf = zipfile.ZipFile(fp)\n # Detect filenames encoding.\n sample = ' '.join(zf.namelist()[:max_files_count])\n if not isinstance(sample, binary_type):\n sample = sample.encode('utf-16be')\n encoding = chardet.detect(sample).get('encoding', 'utf-8')\n for i, info in enumerate(zf.infolist()):\n if i > max_files_count:\n raise BufferError('Too many files inside the ZIP file.')\n comps = info.filename.split(os.sep)\n node = tree\n for c in comps:\n if not isinstance(c, text_type):\n c = c.decode(encoding)\n if c not in node['children']:\n if c == '':\n node['type'] = 'folder'\n continue\n node['children'][c] = {\n 'name': c,\n 'type': 'item',\n 'id': 'item{0}'.format(i),\n 'children': {}\n }\n node = node['children'][c]\n node['size'] = info.file_size\n except BufferError:\n return tree, True, None\n except (zipfile.LargeZipFile):\n return tree, False, 'Zipfile is too large to be previewed.'\n except Exception as e:\n current_app.logger.warning(str(e), exc_info=True)\n return tree, False, 'Zipfile is not previewable.'\n\n return tree, False, None", "response": "Create tree structure from a ZIP archive."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning appropriate template and pass the file and an embed flag.", "response": "def preview(file):\n \"\"\"Return appropriate template and pass the file and an embed flag.\"\"\"\n tree, limit_reached, error = make_tree(file)\n list = children_to_list(tree)['children']\n return render_template(\n \"invenio_previewer/zip.html\",\n file=file,\n tree=list,\n limit_reached=limit_reached,\n error=error,\n js_bundles=current_previewer.js_bundles + ['previewer_fullscreen_js'],\n css_bundles=current_previewer.css_bundles,\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create_noisy_edm(edm, noise, n=None):\n N = edm.shape[0]\n if n is None:\n n = N\n found = False\n max_it = 100\n i = 0\n while not found:\n i += 1\n dm = np.sqrt(edm) + np.random.normal(scale=noise, size=edm.shape)\n dm = np.triu(dm)\n edm_noisy = np.power(dm + dm.T, 2)\n edm_noisy[range(N), range(N)] = 0.0\n edm_noisy[n:, n:] = edm[n:, n:]\n if (edm_noisy >= 0).all():\n found = True\n if i > max_it:\n print('create_noisy_edm: last EDM', edm_noisy)\n raise RuntimeError(\n 'Could not generate all positive edm in {} iterations.'.format(max_it))\n return edm_noisy", "response": "Create noisy version of the input EDM."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create_mask(N, method='all', nmissing=0):\n\n weights = np.ones((N, N))\n weights[range(N), range(N)] = 0\n\n if method == 'none':\n return weights\n\n # create indices object to choose from\n elif method == 'all':\n all_indices = np.triu_indices(N, 1)\n elif method == 'first':\n all_indices = [np.zeros(N - 1).astype(np.int),\n np.arange(1, N).astype(np.int)]\n ntotal = len(all_indices[0])\n # randomly choose from indices and set to 0\n choice = np.random.choice(ntotal, nmissing, replace=False)\n chosen = [all_indices[0][choice], all_indices[1][choice]]\n weights[chosen] = 0\n weights[chosen[1], chosen[0]] = 0\n return weights", "response": "Create weight mask according to method."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef describe(**kwargs):\n # if we have a single method, make it a list.\n if isinstance(kwargs.get(\"paths\"), string_type):\n kwargs[\"paths\"] = [kwargs[\"paths\"]]\n if isinstance(kwargs.get(\"methods\"), string_type):\n kwargs[\"methods\"] = [kwargs[\"methods\"]]\n attrs = TransmuteAttributes(**kwargs)\n\n def decorator(f):\n if hasattr(f, \"transmute\"):\n f.transmute = f.transmute | attrs\n else:\n f.transmute = attrs\n return f\n\n return decorator", "response": "This is a decorator that can customize the rest API\n that transmute generates."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef add_swagger(app, json_route, html_route):\n app.router.add_route('GET', json_route, create_swagger_json_handler(app))\n add_swagger_api_route(app, html_route, json_route)", "response": "add a swagger. json route and a swagger. html route"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nadding a route to the swagger api.", "response": "def add_swagger_api_route(app, target_route, swagger_json_route):\n \"\"\"\n mount a swagger statics page.\n\n app: the aiohttp app object\n target_route: the path to mount the statics page.\n swagger_json_route: the path where the swagger json definitions is\n expected to be.\n \"\"\"\n static_root = get_swagger_static_root()\n swagger_body = generate_swagger_html(\n STATIC_ROOT, swagger_json_route\n ).encode(\"utf-8\")\n\n async def swagger_ui(request):\n return web.Response(body=swagger_body, content_type=\"text/html\")\n\n app.router.add_route(\"GET\", target_route, swagger_ui)\n app.router.add_static(STATIC_ROOT, static_root)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate a handler that returns the swagger definition for an application.", "response": "def create_swagger_json_handler(app, **kwargs):\n \"\"\"\n Create a handler that returns the swagger definition\n for an application.\n\n This method assumes the application is using the\n TransmuteUrlDispatcher as the router.\n \"\"\"\n\n spec = get_swagger_spec(app).swagger_definition(**kwargs)\n encoded_spec = json.dumps(spec).encode(\"UTF-8\")\n\n async def swagger(request):\n return web.Response(\n # we allow CORS, so this can be requested at swagger.io\n headers={\n \"Access-Control-Allow-Origin\": \"*\"\n },\n body=encoded_spec,\n content_type=\"application/json\",\n )\n\n return swagger"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates detail plots for the single - term experiment.", "response": "def create_multispan_plots(tag_ids):\n \"\"\"Create detail plots (first row) and total block(second row) of experiments.\n \n Args:\n tag_ids: list of tag-dictionaries, where the dictionaries must have fields 'name' (used for naming) \n and 'id' (used for numbering axis_dict)\n \n Returns:\n Figure element fig, ax_dict containing the first row plots (accessed via id) and ax_total containing the \n second row block. \n \"\"\"\n import matplotlib.gridspec as gridspec\n fig = plt.figure()\n nrows = 1\n if len(tag_ids) > 1:\n nrows = 2\n fig.set_size_inches(10, 5*nrows)\n\n gs = gridspec.GridSpec(nrows, len(tag_ids))\n ax_list = [fig.add_subplot(g) for g in gs]\n ax_dict = {}\n for i, tag_dict in enumerate(tag_ids):\n ax_dict[tag_dict['id']] = ax_list[i]\n ax_dict[tag_dict['id']].set_title(\n 'System {} (id {})'.format(tag_dict['name'], tag_dict['id']))\n\n if nrows > 1:\n ax_total = plt.subplot(gs[1, :])\n\n title = 'Combined {}'.format(tag_ids[0]['name'])\n for i in range(1, len(tag_ids)):\n title = title + ' and {}'.format(tag_ids[i]['name'])\n ax_total.set_title(title)\n gs.tight_layout(fig, rect=[0, 0.03, 1, 0.95])\n return fig, ax_dict, ax_total\n gs.tight_layout(fig, rect=[0, 0.03, 1, 0.95])\n return fig, ax_dict, None"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef change_angles(method, theta, tol=1e-10):\n try:\n theta_new = np.zeros(theta.shape)\n for i, thet in enumerate(theta):\n try:\n # theta is vector\n theta_new[i] = method(thet, tol)\n except:\n # theta is matrix\n for j, th in enumerate(thet):\n try:\n theta_new[i, j] = method(th, tol)\n except:\n # theta is tensor\n for k, t in enumerate(th):\n theta_new[i, j, k] = method(t, tol)\n return theta_new\n except:\n return method(theta, tol)", "response": "Function used by all angle conversion functions"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncalculates coordinates of point Pk given two points Pi Pj and inner angles.", "response": "def get_point(theta_ik, theta_jk, Pi, Pj):\n \"\"\" Calculate coordinates of point Pk given two points Pi, Pj and inner angles. :param theta_ik: Inner angle at Pi to Pk.\n :param theta_jk: Inner angle at Pj to Pk.\n :param Pi: Coordinates of point Pi.\n :param Pj: Coordinates of point Pj.\n\n :return: Coordinate of point Pk.\n \"\"\"\n A = np.array([[sin(theta_ik), -cos(theta_ik)],\n [sin(theta_jk), -cos(theta_jk)]])\n B = np.array([[sin(theta_ik), -cos(theta_ik), 0, 0],\n [0, 0, sin(theta_jk), -cos(theta_jk)]])\n p = np.r_[Pi, Pj]\n Pk = np.linalg.solve(A, np.dot(B, p))\n return Pk"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef keys(self):\n return_value = []\n for s in self.serializers:\n return_value += s.content_type\n return return_value", "response": "return a list of the content types this set supports"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngive a function and a TransmuteAttributes object and a set of parameters that can be used to generate the transmute parameters for the current language.", "response": "def get_parameters(signature, transmute_attrs, arguments_to_ignore=None):\n \"\"\"given a function, categorize which arguments should be passed by\n what types of parameters. The choices are:\n\n * query parameters: passed in as query parameters in the url\n * body parameters: retrieved from the request body (includes forms)\n * header parameters: retrieved from the request header\n * path parameters: retrieved from the uri path\n\n The categorization is performed for an argument \"arg\" by:\n\n 1. examining the transmute parameters attached to the function (e.g.\n func.transmute_query_parameters), and checking if \"arg\" is mentioned. If so,\n it is added to the category.\n\n 2. If the argument is available in the path, it will be added\n as a path parameter.\n\n 3. If the method of the function is GET and only GET, then \"arg\" will be\n be added to the expected query parameters. Otherwise, \"arg\" will be added as\n a body parameter.\n \"\"\"\n params = Parameters()\n used_keys = set(arguments_to_ignore or [])\n # examine what variables are categorized first.\n for key in [\"query\", \"header\", \"path\"]:\n param_set = getattr(params, key)\n explicit_parameters = getattr(transmute_attrs, key + \"_parameters\")\n used_keys |= load_parameters(\n param_set, explicit_parameters, signature, transmute_attrs\n )\n\n body_parameters = transmute_attrs.body_parameters\n if isinstance(body_parameters, str):\n name = body_parameters\n params.body = Param(\n argument_name=name,\n description=transmute_attrs.parameter_descriptions.get(name),\n arginfo=signature.get_argument(name),\n )\n used_keys.add(name)\n else:\n used_keys |= load_parameters(\n params.body, transmute_attrs.body_parameters, signature, transmute_attrs\n )\n\n # extract the parameters from the paths\n for name in _extract_path_parameters_from_paths(transmute_attrs.paths):\n params.path[name] = Param(\n argument_name=name,\n description=transmute_attrs.parameter_descriptions.get(name),\n arginfo=signature.get_argument(name),\n )\n used_keys.add(name)\n\n # check the method type, and decide if the parameters should be extracted\n # from query parameters or the body\n default_param_key = \"query\" if transmute_attrs.methods == set([\"GET\"]) else \"body\"\n default_params = getattr(params, default_param_key)\n\n # parse all positional params\n for arginfo in signature:\n if arginfo.name in used_keys:\n continue\n used_keys.add(arginfo.name)\n default_params[arginfo.name] = Param(\n arginfo.name,\n description=transmute_attrs.parameter_descriptions.get(arginfo.name),\n arginfo=arginfo,\n )\n\n return params"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nextracts the parameters present in a list of paths.", "response": "def _extract_path_parameters_from_paths(paths):\n \"\"\"\n from a list of paths, return back a list of the\n arguments present in those paths.\n\n the arguments available in all of the paths must match: if not,\n an exception will be raised.\n \"\"\"\n params = set()\n for path in paths:\n parts = PART_REGEX.split(path)\n for p in parts:\n match = PARAM_REGEX.match(p)\n if match:\n params.add(match.group(\"name\"))\n return params"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nbuilding the schema for the body parameter.", "response": "def _build_body_schema(serializer, body_parameters):\n \"\"\" body is built differently, since it's a single argument no matter what. \"\"\"\n description = \"\"\n if isinstance(body_parameters, Param):\n schema = serializer.to_json_schema(body_parameters.arginfo.type)\n description = body_parameters.description\n required = True\n else:\n if len(body_parameters) == 0:\n return None\n required = set()\n body_properties = {}\n for name, param in body_parameters.items():\n arginfo = param.arginfo\n body_properties[name] = serializer.to_json_schema(arginfo.type)\n body_properties[name][\"description\"] = param.description\n if arginfo.default is NoDefault:\n required.add(name)\n schema = {\n \"type\": \"object\",\n \"required\": list(required),\n \"properties\": body_properties,\n }\n required = len(required) > 0\n return BodyParameter(\n {\n \"name\": \"body\",\n \"description\": description,\n \"required\": required,\n \"schema\": schema,\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse(self, data):\n\n self.validate_packet(data)\n\n results = self.parse_header_part(data)\n sub_type = results['packet_subtype']\n\n id_ = self.dump_hex(data[4:6])\n\n direction = data[6] * 256 + data[7]\n if sub_type != 0x05:\n av_speed = (data[8] * 256 + data[9]) * 0.1\n else:\n av_speed = '--??--'\n gust = (data[10] * 256 + data[11]) * 0.1\n if sub_type == 0x04:\n temperature = ((data[12] & 0x7f) * 256 + data[13]) / 10\n signbit = data[12] & 0x80\n if signbit != 0:\n temperature = -temperature\n else:\n temperature = '--??--'\n if sub_type == 0x04:\n wind_chill = ((data[14] & 0x7f) * 256 + data[15]) / 10\n signbit = data[14] & 0x80\n if signbit != 0:\n wind_chill = -wind_chill\n else:\n wind_chill = '--??--'\n\n sensor_specific = {\n 'id': id_,\n 'direction': direction,\n 'wind_gust': gust\n }\n if av_speed != '--??--':\n sensor_specific['av_speed'] = av_speed\n if temperature != '--??--':\n sensor_specific['temperature'] = temperature\n if wind_chill != '--??--':\n sensor_specific['wind_chill'] = wind_chill\n\n results.update(RfxPacketUtils.parse_signal_and_battery(data[16]))\n results.update(sensor_specific)\n\n return results", "response": "Parses a 17 bytes packet in the Wind format and returns a dictionary containing the parsed values."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef procrustes(anchors, X, scale=True, print_out=False):\n def centralize(X):\n n = X.shape[0]\n ones = np.ones((n, 1))\n return X - np.multiply(1 / n * np.dot(ones.T, X), ones)\n m = anchors.shape[0]\n N, d = X.shape\n assert m >= d, 'Have to give at least d anchor nodes.'\n X_m = X[N - m:, :]\n ones = np.ones((m, 1))\n\n mux = 1 / m * np.dot(ones.T, X_m)\n muy = 1 / m * np.dot(ones.T, anchors)\n sigmax = 1 / m * np.linalg.norm(X_m - mux)**2\n sigmaxy = 1 / m * np.dot((anchors - muy).T, X_m - mux)\n try:\n U, D, VT = np.linalg.svd(sigmaxy)\n except np.LinAlgError:\n print('strange things are happening...')\n print(sigmaxy)\n print(np.linalg.matrix_rank(sigmaxy))\n #this doesn't work and doesn't seem to be necessary! (why?)\n # S = np.eye(D.shape[0])\n # if (np.linalg.det(U)*np.linalg.det(VT.T) < 0):\n # print('switching')\n # S[-1,-1] = -1.0\n # else:\n # print('not switching')\n # c = np.trace(np.dot(np.diag(D),S))/sigmax\n # R = np.dot(U, np.dot(S,VT))\n if (scale):\n c = np.trace(np.diag(D)) / sigmax\n else:\n c = np.trace(np.diag(D)) / sigmax\n if (print_out):\n print('Optimal scale would be: {}. Setting it to 1 now.'.format(c))\n c = 1.0\n R = np.dot(U, VT)\n t = muy.T - c * np.dot(R, mux.T)\n X_transformed = (c * np.dot(R, (X - mux).T) + muy.T).T\n return X_transformed, R, t, c", "response": "Fit X to an anchor set by applying optimal translation rotation and reflection."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreconstructs point set using E ( dge ) - MDS.", "response": "def reconstruct_emds(edm, Om, all_points, method=None, **kwargs):\n \"\"\" Reconstruct point set using E(dge)-MDS.\n \"\"\"\n from .point_set import dm_from_edm\n N = all_points.shape[0]\n d = all_points.shape[1]\n dm = dm_from_edm(edm)\n if method is None:\n from .mds import superMDS\n Xhat, __ = superMDS(all_points[0, :], N, d, Om=Om, dm=dm)\n else:\n C = kwargs.get('C', None)\n b = kwargs.get('b', None)\n if C is None or b is None:\n raise NameError(\n 'Need constraints C and b for reconstruct_emds in iterative mode.')\n KE_noisy = np.multiply(np.outer(dm, dm), Om)\n if method == 'iterative':\n from .mds import iterativeEMDS\n Xhat, __ = iterativeEMDS(\n all_points[0, :], N, d, KE=KE_noisy, C=C, b=b)\n elif method == 'relaxed':\n from .mds import relaxedEMDS\n Xhat, __ = relaxedEMDS(\n all_points[0, :], N, d, KE=KE_noisy, C=C, b=b)\n else:\n raise NameError('Undefined method', method)\n Y, R, t, c = procrustes(all_points, Xhat, scale=False)\n return Y"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef reconstruct_cdm(dm, absolute_angles, all_points, W=None):\n from pylocus.point_set import dmi_from_V, sdm_from_dmi, get_V\n from pylocus.mds import signedMDS\n\n N = all_points.shape[0]\n\n V = get_V(absolute_angles, dm)\n\n dmx = dmi_from_V(V, 0)\n dmy = dmi_from_V(V, 1)\n\n sdmx = sdm_from_dmi(dmx, N)\n sdmy = sdm_from_dmi(dmy, N)\n\n points_x = signedMDS(sdmx, W)\n points_y = signedMDS(sdmy, W)\n\n Xhat = np.c_[points_x, points_y]\n Y, R, t, c = procrustes(all_points, Xhat, scale=False)\n return Y", "response": "Reconstruct point set from angle and distance measurements using coordinate difference matrices."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreconstructs point set using MDS and matrix completion algorithms.", "response": "def reconstruct_mds(edm, all_points, completion='optspace', mask=None, method='geometric', print_out=False, n=1):\n \"\"\" Reconstruct point set using MDS and matrix completion algorithms.\n \"\"\"\n from .point_set import dm_from_edm\n from .mds import MDS\n N = all_points.shape[0]\n d = all_points.shape[1]\n if mask is not None:\n edm_missing = np.multiply(edm, mask)\n if completion == 'optspace':\n from .edm_completion import optspace\n edm_complete = optspace(edm_missing, d + 2)\n elif completion == 'alternate':\n from .edm_completion import rank_alternation\n edm_complete, errs = rank_alternation(\n edm_missing, d + 2, print_out=False, edm_true=edm)\n else:\n raise NameError('Unknown completion method {}'.format(completion))\n if (print_out):\n err = np.linalg.norm(edm_complete - edm)**2 / \\\n np.linalg.norm(edm)**2\n print('{}: relative error:{}'.format(completion, err))\n edm = edm_complete\n Xhat = MDS(edm, d, method, False).T\n Y, R, t, c = procrustes(all_points[n:], Xhat, True)\n #Y, R, t, c = procrustes(all_points, Xhat, True)\n return Y"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef reconstruct_sdp(edm, all_points, W=None, print_out=False, lamda=1000, **kwargs):\n from .edm_completion import semidefinite_relaxation\n edm_complete = semidefinite_relaxation(\n edm, lamda=lamda, W=W, print_out=print_out, **kwargs)\n Xhat = reconstruct_mds(edm_complete, all_points, method='geometric')\n return Xhat, edm_complete", "response": "Reconstruct point set using semi - definite rank relaxation."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreconstruct the point set using S ( squared R ) L ( squares ) method.", "response": "def reconstruct_srls(edm, all_points, W=None, print_out=False, n=1, rescale=False,\n z=None):\n \"\"\" Reconstruct point set using S(quared)R(ange)L(east)S(quares) method.\n \"\"\"\n from .lateration import SRLS, get_lateration_parameters\n Y = all_points.copy()\n indices = range(n)\n for index in indices:\n anchors, w, r2 = get_lateration_parameters(all_points, indices, index,\n edm, W)\n if print_out:\n print('SRLS parameters:')\n print('anchors', anchors)\n print('w', w)\n print('r2', r2)\n\n srls = SRLS(anchors, w, r2, rescale, z, print_out)\n if rescale:\n srls = srls[0] # second element of output is the scale\n Y[index, :] = srls\n return Y"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreconstructs the acd from the given ELM matrix X0 using alternating coordinate descent.", "response": "def reconstruct_acd(edm, X0, W=None, print_out=False, tol=1e-10, sweeps=10):\n \"\"\" Reconstruct point set using alternating coordinate descent.\n\n :param X0: Nxd matrix of starting points.\n :param tol: Stopping criterion: if the stepsize in all coordinate directions \n is less than tol for 2 consecutive sweeps, we stop. \n :param sweep: Maximum number of sweeps. One sweep goes through all coordintaes and points once. \n \"\"\"\n\n def get_unique_delta(delta, i, coord, print_out=False):\n delta_unique = delta.copy()\n ftest = []\n for de in delta_unique:\n X_ktest = X_k.copy()\n X_ktest[i, coord] += de\n ftest.append(f(X_ktest, edm, W))\n # choose delta_unique with lowest cost.\n if print_out:\n print(ftest)\n print(delta_unique)\n delta_unique = delta_unique[ftest == min(ftest)]\n if print_out:\n print(delta_unique)\n if len(delta_unique) > 1:\n # if multiple delta_uniques give the same cost, choose the biggest step.\n delta_unique = max(delta_unique)\n return delta_unique\n\n def sweep():\n for i in range(N):\n loop_point(i)\n\n def loop_coordinate(coord, i):\n delta = get_step_size(i, coord, X_k, edm, W)\n if len(delta) > 1:\n delta_unique = get_unique_delta(delta, i, coord)\n elif len(delta) == 1:\n delta_unique = delta[0]\n else:\n print('Warning: did not find delta!', delta)\n delta_unique = 0.0\n try:\n X_k[i, coord] += delta_unique\n except:\n get_unique_delta(delta, i, coord, print_out=True)\n raise\n\n cost_this = f(X_k, edm, W)\n costs.append(cost_this)\n\n if delta_unique <= tol:\n coordinates_converged[i, coord] += 1\n else:\n if print_out:\n print('======= coordinate {} of {} not yet converged'.format(coord, i))\n coordinates_converged[i, coord] = 0\n\n def loop_point(i):\n coord_counter = 0\n while not (coordinates_converged[i] >= 2).all():\n if print_out:\n print('==== point {}'.format(i))\n coord_counter += 1\n for coord in range(d):\n if print_out:\n print('======= coord {}'.format(coord))\n loop_coordinate(coord, i)\n if coord_counter > coord_n_it:\n break\n from .distributed_mds import get_step_size, f\n\n N, d = X0.shape\n if W is None:\n W = np.ones((N, N)) - np.eye(N)\n\n X_k = X0.copy()\n\n costs = []\n coordinates_converged = np.zeros(X_k.shape)\n coord_n_it = 3\n for sweep_counter in range(sweeps):\n if print_out:\n print('= sweep', sweep_counter)\n sweep()\n if (coordinates_converged >= 2).all():\n if (print_out):\n print('acd: all coordinates converged after {} sweeps.'.format(\n sweep_counter))\n return X_k, costs\n if (print_out):\n print('acd: did not converge after {} sweeps'.format(sweep_counter + 1))\n return X_k, costs"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreconstructing point set using d ( istributed ) w (eighted ) MDS.", "response": "def reconstruct_dwmds(edm, X0, W=None, n=None, r=None, X_bar=None, print_out=False, tol=1e-10, sweeps=100):\n \"\"\" Reconstruct point set using d(istributed)w(eighted) MDS.\n\n Refer to paper \"Distributed Weighted-Multidimensional Scaling for Node Localization in Sensor Networks\" for \n implementation details (doi.org/10.1145/1138127.1138129)\n\n :param X0: Nxd matrix of starting points.\n :param n: Number of points of unknown position. The first n points in X0 and edm are considered unknown. \n :param tol: Stopping criterion: when the cost is below this level, we stop. \n :param sweeps: Maximum number of sweeps. \n \"\"\"\n from .basics import get_edm\n from .distributed_mds import get_b, get_Si\n\n N, d = X0.shape\n if r is None and n is None:\n raise ValueError('either r or n have to be given.')\n elif n is None:\n n = r.shape[0]\n\n if W is None:\n W = np.ones((N, N)) - np.eye(N)\n\n X_k = X0.copy()\n\n costs = []\n # don't have to ignore i=j, because W[i,i] is zero.\n a = np.sum(W[:n, :n], axis=1).flatten() + 2 * \\\n np.sum(W[:n, n:], axis=1).flatten()\n if r is not None:\n a += r.flatten()\n for k in range(sweeps):\n S = 0\n for i in range(n):\n edm_estimated = get_edm(X_k)\n bi = get_b(i, edm_estimated, W, edm, n)\n if r is not None and X_bar is not None:\n X_k[i] = 1 / a[i] * (r[i] * X_bar[i, :] +\n X_k.T.dot(bi).flatten())\n Si = get_Si(i, edm_estimated, edm, W, n, r, X_bar[i], X_k[i])\n else:\n X_k[i] = 1 / a[i] * X_k.T.dot(bi).flatten()\n Si = get_Si(i, edm_estimated, edm, W, n)\n S += Si\n costs.append(S)\n if k > 1 and abs(costs[-1] - costs[-2]) < tol:\n if (print_out):\n print('dwMDS: converged after', k)\n break\n return X_k, costs"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nparsing a 12 byte packet in the Rain format and returns a dictionary containing the parsed data.", "response": "def parse(self, data):\n \"\"\"Parse a 12 bytes packet in the Rain format and return a\n dictionary containing the data extracted. An example of a return value\n would be:\n\n .. code-block:: python\n\n {\n 'id': \"0x2EB2\",\n 'packet_length': 11,\n 'packet_type': 85,\n 'packet_type_name': 'Rain sensors',\n 'sequence_number': 0,\n 'packet_subtype': 1,\n 'packet_subtype_name': \"RGR126/682/918\",\n 'rain_rate': 5,\n 'rain_total': 130.0,\n 'signal_level': 9,\n 'battery_level': 6,\n }\n\n :param data: bytearray to be parsed\n :type data: bytearray\n\n :return: Data dictionary containing the parsed values\n :rtype: dict\n \"\"\"\n\n self.validate_packet(data)\n\n results = self.parse_header_part(data)\n sub_type = results['packet_subtype']\n\n id_ = self.dump_hex(data[4:6])\n\n rain_rate_high = data[6]\n rain_rate_low = data[7]\n if sub_type == 0x01:\n rain_rate = rain_rate_high * 0x100 + rain_rate_low\n elif sub_type == 0x02:\n rain_rate = float(rain_rate_high * 0x100 + rain_rate_low) / 100\n else:\n rain_rate = '--??--'\n rain_total1 = data[8]\n rain_total2 = data[9]\n rain_total3 = data[10]\n if sub_type != 0x06:\n rain_total = float(\n rain_total1 * 0x1000 + rain_total2 * 0x100 + rain_total3) / 10\n else:\n rain_total = '--??--'\n\n sensor_specific = {\n 'id': id_\n }\n if rain_rate != '--??--':\n sensor_specific['rain_rate'] = rain_rate\n if rain_total != '--??--':\n sensor_specific['rain_total'] = rain_total\n\n results.update(RfxPacketUtils.parse_signal_and_battery(data[11]))\n results.update(sensor_specific)\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_lateration_parameters(all_points, indices, index, edm, W=None):\n if W is None:\n W = np.ones(edm.shape)\n\n # delete points that are not considered anchors\n anchors = np.delete(all_points, indices, axis=0)\n r2 = np.delete(edm[index, :], indices)\n w = np.delete(W[index, :], indices)\n\n # set w to zero where measurements are invalid\n if np.isnan(r2).any():\n nan_measurements = np.where(np.isnan(r2))[0]\n r2[nan_measurements] = 0.0\n w[nan_measurements] = 0.0\n if np.isnan(w).any():\n nan_measurements = np.where(np.isnan(w))[0]\n r2[nan_measurements] = 0.0\n w[nan_measurements] = 0.0\n\n # delete anchors where weight is zero to avoid ill-conditioning\n missing_anchors = np.where(w == 0.0)[0]\n w = np.asarray(np.delete(w, missing_anchors))\n r2 = np.asarray(np.delete(r2, missing_anchors))\n w.resize(edm.shape[0] - len(indices) - len(missing_anchors), 1)\n r2.resize(edm.shape[0] - len(indices) - len(missing_anchors), 1)\n anchors = np.delete(anchors, missing_anchors, axis=0)\n assert w.shape[0] == anchors.shape[0]\n assert np.isnan(w).any() == False\n assert np.isnan(r2).any() == False\n return anchors, w, r2", "response": "Get parameters relevant for lateration from full all_points edm and W."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsquaring range least squares algorithm written by A. Beck P. Stoica in Approximate and Exact solutions of Source Localization Problems.", "response": "def SRLS(anchors, w, r2, rescale=False, z=None, print_out=False):\n '''Squared range least squares (SRLS)\n\n Algorithm written by A.Beck, P.Stoica in \"Approximate and Exact solutions of Source Localization Problems\".\n\n :param anchors: anchor points (Nxd)\n :param w: weights for the measurements (Nx1)\n :param r2: squared distances from anchors to point x. (Nx1)\n :param rescale: Optional parameter. When set to True, the algorithm will\n also identify if there is a global scaling of the measurements. Such a\n situation arise for example when the measurement units of the distance is\n unknown and different from that of the anchors locations (e.g. anchors are\n in meters, distance in centimeters).\n :param z: Optional parameter. Use to fix the z-coordinate of localized point.\n :param print_out: Optional parameter, prints extra information.\n\n :return: estimated position of point x.\n '''\n def y_hat(_lambda):\n lhs = ATA + _lambda * D\n assert A.shape[0] == b.shape[0]\n assert A.shape[1] == f.shape[0], 'A {}, f {}'.format(A.shape, f.shape)\n rhs = (np.dot(A.T, b) - _lambda * f).reshape((-1,))\n assert lhs.shape[0] == rhs.shape[0], 'lhs {}, rhs {}'.format(\n lhs.shape, rhs.shape)\n try:\n return np.linalg.solve(lhs, rhs)\n except:\n return np.zeros((lhs.shape[1],))\n\n def phi(_lambda):\n yhat = y_hat(_lambda).reshape((-1, 1))\n sol = np.dot(yhat.T, np.dot(D, yhat)) + 2 * np.dot(f.T, yhat)\n return sol.flatten()\n\n def phi_prime(_lambda):\n # TODO: test this.\n B = np.linalg.inv(ATA + _lambda * D)\n C = A.T.dot(b) - _lambda*f\n y_prime = -B.dot(D.dot(B.dot(C)) - f)\n y = y_hat(_lambda)\n return 2*y.T.dot(D).dot(y_prime) + 2*f.T.dot(y_prime)\n\n from scipy import optimize\n from scipy.linalg import sqrtm\n\n n, d = anchors.shape\n assert r2.shape[1] == 1 and r2.shape[0] == n, 'r2 has to be of shape Nx1'\n assert w.shape[1] == 1 and w.shape[0] == n, 'w has to be of shape Nx1'\n if z is not None:\n assert d == 3, 'Dimension of problem has to be 3 for fixing z.'\n\n if rescale and z is not None:\n raise NotImplementedError('Cannot run rescale for fixed z.')\n if rescale and n < d + 2:\n raise ValueError(\n 'A minimum of d + 2 ranges are necessary for rescaled ranging.')\n elif n < d + 1 and z is None:\n raise ValueError(\n 'A minimum of d + 1 ranges are necessary for ranging.')\n elif n < d:\n raise ValueError(\n 'A minimum of d ranges are necessary for ranging.')\n\n Sigma = np.diagflat(np.power(w, 0.5))\n\n if rescale:\n A = np.c_[-2 * anchors, np.ones((n, 1)), -r2]\n else:\n if z is None:\n A = np.c_[-2 * anchors, np.ones((n, 1))]\n else:\n A = np.c_[-2 * anchors[:, :2], np.ones((n, 1))]\n\n A = Sigma.dot(A)\n\n if rescale:\n b = - np.power(np.linalg.norm(anchors, axis=1), 2).reshape(r2.shape)\n else:\n b = r2 - np.power(np.linalg.norm(anchors, axis=1), 2).reshape(r2.shape)\n if z is not None:\n b = b + 2 * anchors[:, 2].reshape((-1, 1)) * z - z**2\n b = Sigma.dot(b)\n\n ATA = np.dot(A.T, A)\n\n if rescale:\n D = np.zeros((d + 2, d + 2))\n D[:d, :d] = np.eye(d)\n else:\n if z is None:\n D = np.zeros((d + 1, d + 1))\n else:\n D = np.zeros((d, d))\n D[:-1, :-1] = np.eye(D.shape[0]-1)\n\n if rescale:\n f = np.c_[np.zeros((1, d)), -0.5, 0.].T\n elif z is None:\n f = np.c_[np.zeros((1, d)), -0.5].T\n else:\n f = np.c_[np.zeros((1, 2)), -0.5].T\n\n eig = np.sort(np.real(eigvalsh(a=D, b=ATA)))\n if (print_out):\n print('ATA:', ATA)\n print('rank:', np.linalg.matrix_rank(A))\n print('eigvals:', eigvals(ATA))\n print('condition number:', np.linalg.cond(ATA))\n print('generalized eigenvalues:', eig)\n\n #eps = 0.01\n if eig[-1] > 1e-10:\n lower_bound = - 1.0 / eig[-1] \n else:\n print('Warning: biggest eigenvalue is zero!')\n lower_bound = -1e-5\n\n inf = 1e5\n xtol = 1e-12\n\n lambda_opt = 0\n # We will look for the zero of phi between lower_bound and inf. \n # Therefore, the two have to be of different signs. \n if (phi(lower_bound) > 0) and (phi(inf) < 0): \n # brentq is considered the best rootfinding routine. \n try: \n lambda_opt = optimize.brentq(phi, lower_bound, inf, xtol=xtol)\n except:\n print('SRLS error: brentq did not converge even though we found an estimate for lower and upper bonud. Setting lambda to 0.')\n else: \n try: \n lambda_opt = optimize.newton(phi, lower_bound, fprime=phi_prime, maxiter=1000, tol=xtol, verbose=True)\n assert phi(lambda_opt) < xtol, 'did not find solution of phi(lambda)=0:={}'.format(phi(lambda_opt))\n except:\n print('SRLS error: newton did not converge. Setting lambda to 0.')\n\n if (print_out):\n print('phi(lower_bound)', phi(lower_bound))\n print('phi(inf)', phi(inf))\n print('phi(lambda_opt)', phi(lambda_opt))\n pos_definite = ATA + lambda_opt*D\n eig = np.sort(np.real(eigvals(pos_definite)))\n print('should be strictly bigger than 0:', eig)\n\n # Compute best estimate\n yhat = y_hat(lambda_opt)\n\n if print_out and rescale:\n print('Scaling factor :', yhat[-1])\n\n if rescale:\n return yhat[:d], yhat[-1]\n elif z is None:\n return yhat[:d]\n else:\n return np.r_[yhat[0], yhat[1], z]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef PozyxLS(anchors, W, r2, print_out=False):\n ''' Algorithm used by pozyx (https://www.pozyx.io/Documentation/how_does_positioning_work)\n\n :param anchors: anchor points\n :param r2: squared distances from anchors to point x.\n\n :returns: estimated position of point x.\n '''\n N = anchors.shape[0]\n anchors_term = np.sum(np.power(anchors[:-1], 2), axis=1)\n last_term = np.sum(np.power(anchors[-1], 2))\n b = r2[:-1] - anchors_term + last_term - r2[-1]\n A = -2 * (anchors[:-1] - anchors[-1])\n x, res, rank, s = np.linalg.lstsq(A, b)\n return x", "response": "Algorithm used by pozyx. io"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nrange least squares algorithm.", "response": "def RLS(anchors, W, r, print_out=False, grid=None, num_points=10):\n \"\"\" Range least squares (RLS) using grid search.\n\n Algorithm written by A.Beck, P.Stoica in \"Approximate and Exact solutions of Source Localization Problems\".\n\n :param anchors: anchor points\n :param r2: squared distances from anchors to point x.\n :param grid: where to search for solution. (min, max) where min and max are \n lists of d elements, d being the dimension of the setup. If not given, the \n search is conducted within the space covered by the anchors.\n :param num_points: number of grid points per direction.\n\n :return: estimated position of point x.\n \"\"\"\n def cost_function(arr):\n X = np.c_[arr]\n r_measured = np.linalg.norm(anchors - X)\n mse = np.linalg.norm(r_measured - r)**2\n return mse\n\n if grid is None:\n grid = [np.min(anchors, axis=0), np.max(anchors, axis=0)]\n\n d = anchors.shape[1]\n x = np.linspace(grid[0][0], grid[1][0], num_points)\n y = np.linspace(grid[0][1], grid[1][1], num_points)\n if d == 2:\n errors_test = np.zeros((num_points, num_points))\n for i, xs in enumerate(x):\n for j, ys in enumerate(y):\n errors_test[i, j] = cost_function((xs, ys))\n min_idx = errors_test.argmin()\n min_idx_multi = np.unravel_index(min_idx, errors_test.shape)\n xhat = np.c_[x[min_idx_multi[0]], y[min_idx_multi[1]]]\n elif d == 3:\n z = np.linspace(grid[0][2], grid[1][2], num_points)\n errors_test = np.zeros((num_points, num_points, num_points))\n\n # TODO: make this more efficient.\n #xx, yy, zz= np.meshgrid(x, y, z)\n for i, xs in enumerate(x):\n for j, ys in enumerate(y):\n for k, zs in enumerate(z):\n errors_test[i, j, k] = cost_function((xs, ys, zs))\n min_idx = errors_test.argmin()\n min_idx_multi = np.unravel_index(min_idx, errors_test.shape)\n xhat = np.c_[x[min_idx_multi[0]],\n y[min_idx_multi[1]], z[min_idx_multi[2]]]\n else:\n raise ValueError('Non-supported number of dimensions.')\n return xhat[0]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nrange least squares algorithm cited by A. Beck P. Stoica in Approximate and Exact solutions of Source Localization Problems.", "response": "def RLS_SDR(anchors, W, r, print_out=False):\n \"\"\" Range least squares (RLS) using SDR.\n\n Algorithm cited by A.Beck, P.Stoica in \"Approximate and Exact solutions of Source Localization Problems\".\n\n :param anchors: anchor points\n :param r2: squared distances from anchors to point x.\n\n :return: estimated position of point x.\n \"\"\"\n from pylocus.basics import low_rank_approximation, eigendecomp\n from pylocus.mds import x_from_eigendecomp\n\n m = anchors.shape[0]\n d = anchors.shape[1]\n\n G = Variable(m + 1, m + 1)\n X = Variable(d + 1, d + 1)\n constraints = [G[m, m] == 1.0,\n X[d, d] == 1.0,\n G >> 0, X >> 0,\n G == G.T, X == X.T]\n for i in range(m):\n Ci = np.eye(d + 1)\n Ci[:-1, -1] = -anchors[i]\n Ci[-1, :-1] = -anchors[i].T\n Ci[-1, -1] = np.linalg.norm(anchors[i])**2\n constraints.append(G[i, i] == trace(Ci * X))\n\n obj = Minimize(trace(G) - 2 * sum_entries(mul_elemwise(r, G[m, :-1].T)))\n prob = Problem(obj, constraints)\n\n ## Solution\n total = prob.solve(verbose=True)\n rank_G = np.linalg.matrix_rank(G.value)\n rank_X = np.linalg.matrix_rank(X.value)\n if rank_G > 1:\n u, s, v = np.linalg.svd(G.value, full_matrices=False)\n print('optimal G is not of rank 1!')\n print(s)\n if rank_X > 1:\n u, s, v = np.linalg.svd(X.value, full_matrices=False)\n print('optimal X is not of rank 1!')\n print(s)\n\n factor, u = eigendecomp(X.value, 1)\n xhat = x_from_eigendecomp(factor, u, 1)\n return xhat"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\njoin parameters from the lhs to the rhs", "response": "def _join_parameters(base, nxt):\n \"\"\" join parameters from the lhs to the rhs, if compatible. \"\"\"\n if nxt is None:\n return base\n if isinstance(base, set) and isinstance(nxt, set):\n return base | nxt\n else:\n return nxt"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget the swagger_schema operation representation.", "response": "def get_swagger_operation(self, context=default_context):\n \"\"\"\n get the swagger_schema operation representation.\n \"\"\"\n consumes = produces = context.contenttype_serializers.keys()\n parameters = get_swagger_parameters(self.parameters, context)\n responses = {\n \"400\": Response(\n {\n \"description\": \"invalid input received\",\n \"schema\": Schema(\n {\n \"title\": \"FailureObject\",\n \"type\": \"object\",\n \"properties\": {\n \"success\": {\"type\": \"boolean\"},\n \"result\": {\"type\": \"string\"},\n },\n \"required\": [\"success\", \"result\"],\n }\n ),\n }\n )\n }\n\n for code, details in self.response_types.items():\n responses[str(code)] = details.swagger_definition(context)\n\n return Operation(\n {\n \"summary\": self.summary,\n \"description\": self.description,\n \"consumes\": consumes,\n \"produces\": produces,\n \"parameters\": parameters,\n \"responses\": responses,\n \"operationId\": self.raw_func.__name__,\n \"tags\": self.tags,\n }\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef process_result(self, context, result_body, exc, content_type):\n return process_result(self, context, result_body, exc, content_type)", "response": "Process a result object and return the appropriate result object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nparsing the response types from the given parameters and return back the response types dictionary.", "response": "def _parse_response_types(argspec, attrs):\n \"\"\"\n from the given parameters, return back the response type dictionaries.\n \"\"\"\n return_type = argspec.annotations.get(\"return\") or None\n type_description = attrs.parameter_descriptions.get(\"return\", \"\")\n response_types = attrs.response_types.copy()\n if return_type or len(response_types) == 0:\n response_types[attrs.success_code] = ResponseType(\n type=return_type,\n type_description=type_description,\n description=\"success\",\n )\n return response_types"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef generate_swagger_html(swagger_static_root, swagger_json_url):\n tmpl = _get_template(\"swagger.html\")\n return tmpl.render(\n swagger_root=swagger_static_root, swagger_json_url=swagger_json_url\n )", "response": "Generate a swagger. html file from the given swagger statics and swagger json path."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd a transmute function to the spec", "response": "def add_func(self, transmute_func, transmute_context):\n \"\"\" add a transmute function's swagger definition to the spec \"\"\"\n swagger_path = transmute_func.get_swagger_path(transmute_context)\n for p in transmute_func.paths:\n self.add_path(p, swagger_path)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef add_path(self, path, path_item):\n if path not in self._swagger:\n self._swagger[path] = path_item\n else:\n for method, definition in path_item.items():\n if definition is not None:\n setattr(self._swagger[path], method, definition)", "response": "Add the path items to the swagger file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a valid swagger spec with the values passed.", "response": "def swagger_definition(self, base_path=None, **kwargs):\n \"\"\"\n return a valid swagger spec, with the values passed.\n \"\"\"\n return Swagger(\n {\n \"info\": Info(\n {\n key: kwargs.get(key, self.DEFAULT_INFO.get(key))\n for key in Info.fields.keys()\n if key in kwargs or key in self.DEFAULT_INFO\n }\n ),\n \"paths\": self.paths,\n \"swagger\": \"2.0\",\n \"basePath\": base_path,\n }\n ).to_primitive()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _call_spy(self, *args, **kwargs):\n '''\n Wrapper to call the spied-on function. Operates similar to\n Expectation.test.\n '''\n if self._spy_side_effect:\n if self._spy_side_effect_args or self._spy_side_effect_kwargs:\n self._spy_side_effect(\n *self._spy_side_effect_args,\n **self._spy_side_effect_kwargs)\n else:\n self._spy_side_effect(*args, **kwargs)\n\n return_value = self._stub.call_orig(*args, **kwargs)\n if self._spy_return:\n self._spy_return(return_value)\n\n return return_value", "response": "Wrapper to call the spied - on function. Operates similar to the test function. Operates similar to test."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef side_effect(self, func, *args, **kwargs):\n '''\n Wrap side effects for spies.\n '''\n self._spy_side_effect = func\n self._spy_side_effect_args = args\n self._spy_side_effect_kwargs = kwargs\n return self", "response": "Wrap side effects for spies."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_interact_items(glob):\n a, l = glob['args'], glob['logger']\n enabled = getattr(a, a._collisions.get(\"interact\") or \"interact\", False)\n if enabled:\n readline.parse_and_bind('tab: complete')\n\n # InteractiveConsole as defined in the code module, but handling a banner\n # using the logging of tinyscript\n class InteractiveConsole(BaseInteractiveConsole, object):\n def __init__(self, banner=None, namespace=None, filename='',\n exitmsg=None):\n if enabled:\n self.banner = banner\n self.exitmsg = exitmsg\n ns = glob\n ns.update(namespace or {})\n super(InteractiveConsole, self).__init__(locals=ns,\n filename=filename)\n \n def __enter__(self):\n if enabled and self.banner is not None:\n l.interact(self.banner)\n return self\n \n def __exit__(self, *args):\n if enabled and self.exitmsg is not None:\n l.interact(self.exitmsg)\n \n def interact(self, *args, **kwargs):\n if enabled:\n super(InteractiveConsole, self).interact(*args, **kwargs)\n\n glob['InteractiveConsole'] = InteractiveConsole\n \n def interact(banner=None, readfunc=None, namespace=None, exitmsg=None):\n if enabled:\n if banner is not None:\n l.interact(banner)\n ns = glob\n ns.update(namespace or {})\n base_interact(readfunc=readfunc, local=ns)\n if exitmsg is not None:\n l.interact(exitmsg)\n\n glob['interact'] = interact\n \n glob['compile_command'] = compile_command if enabled else \\\n lambda *a, **kw: None\n\n # ConsoleSocket for handling duplicating std*** to a socket for the\n # RemoteInteractiveConsole\n host = getattr(a, a._collisions.get(\"host\") or \"host\", None)\n port = getattr(a, a._collisions.get(\"port\") or \"port\", None)\n\n # custom socket, for handling the bindings of stdXXX through a socket\n class ConsoleSocket(socket.socket):\n def readline(self, nbytes=2048):\n return self.recv(nbytes)\n\n def write(self, *args, **kwargs):\n return self.send(*args, **kwargs)\n\n # RemoteInteractiveConsole as defined in the code module, but handling \n # interaction through a socket\n class RemoteInteractiveConsole(InteractiveConsole):\n def __init__(self, *args, **kwargs):\n if enabled:\n # open a socket\n self.socket = ConsoleSocket()\n self.socket.connect((str(host), port))\n # save STDIN, STDOUT and STDERR\n self.__stdin = sys.stdin\n self.__stdout = sys.stdout\n self.__stderr = sys.stderr\n # rebind STDIN, STDOUT and STDERR to the socket\n sys.stdin = sys.stdout = sys.stderr = self.socket\n # now initialize the interactive console\n super(RemoteInteractiveConsole, self).__init__(*args, **kwargs)\n \n def __exit__(self, *args):\n if enabled:\n super(RemoteInteractiveConsole, self).__exit__(*args)\n self.socket.close()\n self.close()\n \n def close(self):\n if enabled:\n # restore STDIN, STDOUT and STDERR\n sys.stdin = self.__stdin\n sys.stdout = self.__stdout\n sys.stderr = self.__stderr\n\n glob['RemoteInteractiveConsole'] = RemoteInteractiveConsole", "response": "This function prepares the interaction items for inclusion in main script s global scope dictionary."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconfigure the logger for the given dictionary.", "response": "def configure_logger(glob, multi_level,\n relative=False, logfile=None, syslog=False):\n \"\"\"\n Logger configuration function for setting either a simple debug mode or a\n multi-level one.\n \n :param glob: globals dictionary\n :param multi_level: boolean telling if multi-level debug is to be considered\n :param relative: use relative time for the logging messages\n :param logfile: log file to be saved (None means do not log to file)\n :param syslog: enable logging to /var/log/syslog\n \"\"\"\n levels = [logging.ERROR, logging.WARNING, logging.INFO, logging.DEBUG] \\\n if multi_level else [logging.INFO, logging.DEBUG]\n try:\n verbose = min(int(glob['args'].verbose), 3)\n except AttributeError:\n verbose = 0\n glob['args']._debug_level = dl = levels[verbose]\n logger.handlers = []\n glob['logger'] = logger\n handler = logging.StreamHandler()\n formatter = logging.Formatter(glob['LOG_FORMAT'], glob['DATE_FORMAT'])\n handler.setFormatter(formatter)\n glob['logger'].addHandler(handler)\n glob['logger'].setLevel(dl)\n if relative:\n coloredlogs.ColoredFormatter = RelativeTimeColoredFormatter\n coloredlogs.install(dl,\n logger=glob['logger'],\n fmt=glob['LOG_FORMAT'],\n datefmt=glob['DATE_FORMAT'], \n milliseconds=glob['TIME_MILLISECONDS'],\n syslog=syslog,\n stream=logfile)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef process_result(transmute_func, context, result, exc, content_type):\n if isinstance(result, Response):\n response = result\n else:\n response = Response(\n result=result, code=transmute_func.success_code, success=True\n )\n if exc:\n if isinstance(exc, APIException):\n response.result = str(exc)\n response.success = False\n response.code = exc.code\n else:\n reraise(type(exc), exc, getattr(exc, \"__traceback__\", None))\n else:\n return_type = transmute_func.get_response_by_code(response.code)\n if return_type:\n response.result = context.serializers.dump(return_type, response.result)\n try:\n content_type = str(content_type)\n serializer = context.contenttype_serializers[content_type]\n except NoSerializerFound:\n serializer = context.contenttype_serializers.default\n content_type = serializer.main_type\n if response.success:\n result = context.response_shape.create_body(attr.asdict(response))\n response.result = result\n else:\n response.result = attr.asdict(response)\n body = serializer.dump(response.result)\n # keeping the return type a dict to\n # reduce performance overhead.\n return {\n \"body\": body,\n \"code\": response.code,\n \"content-type\": content_type,\n \"headers\": response.headers,\n }", "response": "process a result of a transmute_func call"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a new array that is a subset of the matrix M_E.", "response": "def opt_space(M_E, r=None, niter=50, tol=1e-6, print_out=False):\r\n '''\r\n Implementation of the OptSpace matrix completion algorithm.\r\n An algorithm for Matrix Reconstruction from a partially revealed set.\r\n Sparse treatment of matrices are removed because of indexing problems in Python.\r\n Args:\r\n M_E: 2D numpy array; The partially revealed matrix.\r\n Matrix with zeroes at the unrevealed indices.\r\n r: The rank to be used for reconstruction. If left empty, the rank is guessed in the program.\r\n niter: Maximum number of iterations.\r\n tol: Stop iterations if norm((XSY' - M_E) * E, 'fro') / sqrt(|E|) < tol, where\r\n E_{ij} = 1 if M_{ij} is revealed and zero otherwise,\r\n |E| is the size of the revealed set.\r\n Returns: The following\r\n X: A M_E.shape[0]xr numpy array\r\n S: An rxr numpy array\r\n Y: A M_E.shape[1]xr numpy matrix such that M_hat = X*S*Y'\r\n errs: A vector containing norm((XSY' - M_E) * E, 'fro') / sqrt(|E|) at each iteration.\r\n '''\r\n n, m = M_E.shape\r\n # construct the revealed set\r\n E = np.zeros(M_E.shape)\r\n E[np.nonzero(M_E)] = 1\r\n eps = np.count_nonzero(E) / np.sqrt(m * n)\r\n\r\n if r is None:\r\n print('Rank not specified. Trying to guess ...')\r\n r = guess_rank(M_E)\r\n print('Using Rank : %d' % r)\r\n\r\n m0 = 10000\r\n rho = 0\r\n\r\n rescal_param = np.sqrt(np.count_nonzero(\r\n E) * r / np.linalg.norm(M_E, ord='fro')**2)\r\n M_E = M_E * rescal_param\r\n\r\n if print_out:\r\n print('Trimming ...')\r\n\r\n M_Et = copy.deepcopy(M_E)\r\n d = E.sum(0)\r\n d_ = np.mean(d)\r\n\r\n for col in range(m):\r\n if E[:, col].sum() > 2 * d_:\r\n nonzero_ind_list = np.nonzero(E[:, col])\r\n p = np.random.permutation(len(nonzero_ind_list))\r\n M_Et[nonzero_ind_list[p[np.ceil(2 * d_):]], col] = 0\r\n\r\n d = E.sum(1)\r\n d_ = np.mean(d)\r\n for row in range(n):\r\n if E[row, :].sum() > 2 * d_:\r\n nonzero_ind_list = np.nonzero(E[row, :])\r\n p = np.random.permutation(len(nonzero_ind_list))\r\n M_Et[nonzero_ind_list[row, p[np.ceil(2 * d_):]]] = 0\r\n\r\n if print_out:\r\n print('Sparse SVD ...')\r\n\r\n X0, S0, Y0 = svds_descending(M_Et, r)\r\n del M_Et\r\n\r\n # Initial Guess\r\n X0 = X0 * np.sqrt(n)\r\n Y0 = Y0 * np.sqrt(m)\r\n S0 = S0 / eps\r\n\r\n if print_out:\r\n print('Iteration\\tFit Error')\r\n\r\n # Gradient Descent\r\n X = copy.deepcopy(X0)\r\n Y = copy.deepcopy(Y0)\r\n S = getoptS(X, Y, M_E, E)\r\n\r\n errs = [None] * (niter + 1)\r\n errs[0] = np.linalg.norm(\r\n (M_E - np.dot(np.dot(X, S), Y.T)) * E, ord='fro') / np.sqrt(np.count_nonzero(E))\r\n\r\n if print_out:\r\n print('0\\t\\t\\t%e' % errs[0])\r\n\r\n for i in range(niter):\r\n # Compute the Gradient\r\n W, Z = gradF_t(X, Y, S, M_E, E, m0, rho)\r\n\r\n # Line search for the optimum jump length\r\n t = getoptT(X, W, Y, Z, S, M_E, E, m0, rho)\r\n X = X + t * W\r\n Y = Y + t * Z\r\n S = getoptS(X, Y, M_E, E)\r\n\r\n # Compute the distortion\r\n errs[i + 1] = np.linalg.norm((M_E - np.dot(np.dot(X, S), Y.T))\r\n * E, ord='fro') / np.sqrt(np.count_nonzero(E))\r\n if print_out:\r\n print('%d\\t\\t\\t%e' % (i + 1, errs[i + 1]))\r\n\r\n if abs(errs[i + 1] - errs[i]) < tol:\r\n break\r\n S = S / rescal_param\r\n\r\n return X, S, Y, errs"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef svds_descending(M, k):\r\n '''\r\n In contrast to MATLAB, numpy's svds() arranges the singular\r\n values in ascending order. In order to have matching codes,\r\n we wrap it around by a function which re-sorts the singular\r\n values and singular vectors.\r\n Args:\r\n M: 2D numpy array; the matrix whose SVD is to be computed.\r\n k: Number of singular values to be computed.\r\n\r\n Returns:\r\n u, s, vt = svds(M, k=k)\r\n '''\r\n u, s, vt = svds(M, k=k)\r\n # reverse columns of u\r\n u = u[:, ::-1]\r\n # reverse s\r\n s = s[::-1]\r\n # reverse rows of vt\r\n vt = vt[::-1, :]\r\n return u, np.diag(s), vt.T", "response": "Returns the singular values of the given size in descending order."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef guess_rank(M_E):\r\n '''Guess the rank of the incomplete matrix '''\r\n n, m = M_E.shape\r\n epsilon = np.count_nonzero(M_E) / np.sqrt(m * n)\r\n _, S0, _ = svds_descending(M_E, min(100, max(M_E.shape) - 1))\r\n S0 = np.diag(S0)\r\n\r\n S1 = S0[:-1] - S0[1:]\r\n S1_ = S1 / np.mean(S1[-10:])\r\n r1 = 0\r\n lam = 0.05\r\n cost = [None] * len(S1_)\r\n while r1 <= 0:\r\n for idx in range(len(S1_)):\r\n cost[idx] = lam * max(S1_[idx:]) + idx\r\n i2 = np.argmin(cost)\r\n r1 = np.max(i2)\r\n lam += 0.05\r\n\r\n cost = [None] * (len(S0) - 1)\r\n for idx in range(len(S0) - 1):\r\n cost[idx] = (S0[idx + 1] + np.sqrt(idx * epsilon)\r\n * S0[0] / epsilon) / S0[idx]\r\n i2 = np.argmin(cost)\r\n r2 = np.max(i2 + 1)\r\n r = max([r1, r2])\r\n\r\n return r", "response": "Guess the rank of the incomplete matrix M_E."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef F_t(X, Y, S, M_E, E, m0, rho):\r\n ''' Compute the distortion '''\r\n r = X.shape[1]\r\n out1 = (((np.dot(np.dot(X, S), Y.T) - M_E) * E)**2).sum() / 2\r\n out2 = rho * G(Y, m0, r)\r\n out3 = rho * G(X, m0, r)\r\n\r\n return out1 + out2 + out3", "response": "Compute the distortion of the current state of the tree"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef gradF_t(X, Y, S, M_E, E, m0, rho):\r\n ''' Compute the gradient.\r\n '''\r\n n, r = X.shape\r\n m, r = Y.shape\r\n\r\n XS = np.dot(X, S)\r\n YS = np.dot(Y, S.T)\r\n XSY = np.dot(XS, Y.T)\r\n\r\n Qx = np.dot(np.dot(X.T, ((M_E - XSY) * E)), YS) / n\r\n Qy = np.dot(np.dot(Y.T, ((M_E - XSY) * E).T), XS) / m\r\n\r\n W = np.dot((XSY - M_E) * E, YS) + np.dot(X, Qx) + rho * Gp(X, m0, r)\r\n Z = np.dot(((XSY - M_E) * E).T, XS) + np.dot(Y, Qy) + rho * Gp(Y, m0, r)\r\n\r\n return W, Z", "response": "Compute the gradient of the F_t function."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfinding Sopt given X Y M_E and E", "response": "def getoptS(X, Y, M_E, E):\r\n ''' Find Sopt given X, Y\r\n '''\r\n n, r = X.shape\r\n C = np.dot(np.dot(X.T, M_E), Y)\r\n C = C.flatten()\r\n\r\n A = np.zeros((r * r, r * r))\r\n for i in range(r):\r\n for j in range(r):\r\n ind = j * r + i\r\n temp = np.dot(\r\n np.dot(X.T, np.dot(X[:, i, None], Y[:, j, None].T) * E), Y)\r\n A[:, ind] = temp.flatten()\r\n\r\n S = np.linalg.solve(A, C)\r\n\r\n return np.reshape(S, (r, r)).T"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nrecovers points from euclidean distance matrix using classic MDS algorithm.", "response": "def MDS(D, dim, method='simple', theta=False):\n \"\"\" recover points from euclidean distance matrix using classic MDS algorithm. \n \"\"\"\n N = D.shape[0]\n if method == 'simple':\n d1 = D[0, :]\n # buf_ = d1 * np.ones([1, N]).T + (np.ones([N, 1]) * d1).T\n buf_ = np.broadcast_to(d1, D.shape) + np.broadcast_to(d1[:, np.newaxis], D.shape)\n np.subtract(D, buf_, out=buf_)\n G = buf_ # G = (D - d1 * np.ones([1, N]).T - (np.ones([N, 1]) * d1).T)\n elif method == 'advanced':\n # s1T = np.vstack([np.ones([1, N]), np.zeros([N - 1, N])])\n s1T = np.zeros_like(D)\n s1T[0, :] = 1\n np.subtract(np.identity(N), s1T, out = s1T)\n G = np.dot(np.dot(s1T.T, D), s1T)\n elif method == 'geometric':\n J = np.identity(N) + np.full((N, N), -1.0 / float(N))\n G = np.dot(np.dot(J, D), J)\n else:\n print('Unknown method {} in MDS'.format(method))\n G *= -0.5\n factor, u = eigendecomp(G, dim)\n if (theta):\n return theta_from_eigendecomp(factor, u)\n else:\n return x_from_eigendecomp(factor, u, dim)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfinds the set of points from an edge kernel.", "response": "def superMDS(X0, N, d, **kwargs):\n \"\"\" Find the set of points from an edge kernel.\n \"\"\"\n Om = kwargs.get('Om', None)\n dm = kwargs.get('dm', None)\n if Om is not None and dm is not None:\n KE = kwargs.get('KE', None)\n if KE is not None:\n print('superMDS: KE and Om, dm given. Continuing with Om, dm')\n factor, u = eigendecomp(Om, d)\n uhat = u[:, :d]\n lambdahat = np.diag(factor[:d])\n diag_dm = np.diag(dm)\n Vhat = np.dot(diag_dm, np.dot(uhat, lambdahat))\n elif Om is None or dm is None:\n KE = kwargs.get('KE', None)\n if KE is None:\n raise NameError('Either KE or Om and dm have to be given.')\n factor, u = eigendecomp(KE, d)\n lambda_ = np.diag(factor)\n Vhat = np.dot(u, lambda_)[:, :d]\n\n C_inv = -np.eye(N)\n C_inv[0, 0] = 1.0\n C_inv[:, 0] = 1.0\n b = np.zeros((C_inv.shape[1], d))\n b[0, :] = X0\n b[1:, :] = Vhat[:N - 1, :]\n Xhat = np.dot(C_inv, b)\n return Xhat, Vhat"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfind the set of points from an edge kernel with geometric constraints, using iterative projection", "response": "def iterativeEMDS(X0, N, d, C, b, max_it=10, print_out=False, **kwargs):\n \"\"\" Find the set of points from an edge kernel with geometric constraints, using iterative projection \n \"\"\"\n from pylocus.basics import mse, projection\n KE = kwargs.get('KE', None)\n KE_projected = KE.copy()\n d = len(X0)\n for i in range(max_it):\n # projection on constraints\n KE_projected, cost, __ = projection(KE_projected, C, b)\n rank = np.linalg.matrix_rank(KE_projected)\n\n # rank 2 decomposition\n Xhat_KE, Vhat_KE = superMDS(X0, N, d, KE=KE_projected)\n KE_projected = Vhat_KE.dot(Vhat_KE.T)\n\n error = mse(C.dot(KE_projected), b)\n\n if (print_out):\n print('cost={:2.2e},error={:2.2e}, rank={}'.format(\n cost, error, rank))\n if cost < 1e-20 and error < 1e-20 and rank == d:\n if (print_out):\n print('converged after {} iterations'.format(i))\n return Xhat_KE, Vhat_KE\n print('iterativeMDS did not converge!')\n return None, None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef relaxedEMDS(X0, N, d, C, b, KE, print_out=False, lamda=10):\n E = C.shape[1]\n X = Variable((E, E), PSD=True)\n\n constraints = [C[i, :] * X == b[i] for i in range(C.shape[0])]\n\n obj = Minimize(trace(X) + lamda * norm(KE - X))\n prob = Problem(obj, constraints)\n\n try:\n # CVXOPT is more accurate than SCS, even though slower.\n total_cost = prob.solve(solver='CVXOPT', verbose=print_out)\n except:\n try:\n print('CVXOPT with default cholesky failed. Trying kktsolver...')\n # kktsolver is more robust than default (cholesky), even though slower.\n total_cost = prob.solve(\n solver='CVXOPT', verbose=print_out, kktsolver=\"robust\")\n except:\n try:\n print('CVXOPT with robust kktsovler failed. Trying SCS...')\n # SCS is fast and robust, but inaccurate (last choice).\n total_cost = prob.solve(solver='SCS', verbose=print_out)\n except:\n print('SCS and CVXOPT solver with default and kktsolver failed .')\n if print_out:\n print('status:', prob.status)\n Xhat_KE, Vhat_KE = superMDS(X0, N, d, KE=X.value)\n return Xhat_KE, Vhat_KE", "response": "This function is used to find the set of points from an edge kernel with geometric constraints using convex rank relaxation."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef signedMDS(cdm, W=None):\n\n N = cdm.shape[0]\n\n D_sym = (cdm - cdm.T)\n D_sym /= 2\n\n if W is None:\n x_est = np.mean(D_sym, axis=1)\n x_est -= np.min(x_est)\n return x_est\n\n W_sub = W[1:, 1:]\n sum_W = np.sum(W[1:, :], axis=1)\n\n # A = np.eye(N, N-1, k=-1) - W_sub.astype(np.int) / sum_W[:, None]\n A = np.eye(N - 1) - W_sub.astype(np.int) / 1. / sum_W[:, None]\n d = (np.sum(D_sym[1:, :] * W[1:, :], axis=1) / 1. / sum_W)\n\n x_est = np.linalg.lstsq(A, d)[0]\n x_est = np.r_[[0], x_est]\n x_est -= np.min(x_est)\n\n return x_est, A, np.linalg.pinv(A)", "response": "Find the set of points from a cdm."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _stub_attr(obj, attr_name):\n '''\n Stub an attribute of an object. Will return an existing stub if\n there already is one.\n '''\n # Annoying circular reference requires importing here. Would like to see\n # this cleaned up. @AW\n from .mock import Mock\n\n # Check to see if this a property, this check is only for when dealing\n # with an instance. getattr will work for classes.\n is_property = False\n\n if not inspect.isclass(obj) and not inspect.ismodule(obj):\n # It's possible that the attribute is defined after initialization, and\n # so is not on the class itself.\n attr = getattr(obj.__class__, attr_name, None)\n if isinstance(attr, property):\n is_property = True\n\n if not is_property:\n attr = getattr(obj, attr_name)\n\n # Return an existing stub\n if isinstance(attr, Stub):\n return attr\n\n # If a Mock object, stub its __call__\n if isinstance(attr, Mock):\n return stub(attr.__call__)\n\n if isinstance(attr, property):\n return StubProperty(obj, attr_name)\n\n # Sadly, builtin functions and methods have the same type, so we have to\n # use the same stub class even though it's a bit ugly\n if inspect.ismodule(obj) and isinstance(attr, (types.FunctionType,\n types.BuiltinFunctionType,\n types.BuiltinMethodType)):\n return StubFunction(obj, attr_name)\n\n # In python3 unbound methods are treated as functions with no reference\n # back to the parent class and no im_* fields. We can still make unbound\n # methods work by passing these through to the stub\n if inspect.isclass(obj) and isinstance(attr, types.FunctionType):\n return StubUnboundMethod(obj, attr_name)\n\n # I thought that types.UnboundMethodType differentiated these cases but\n # apparently not.\n if isinstance(attr, types.MethodType):\n # Handle differently if unbound because it's an implicit \"any instance\"\n if getattr(attr, 'im_self', None) is None:\n # Handle the python3 case and py2 filter\n if hasattr(attr, '__self__'):\n if attr.__self__ is not None:\n return StubMethod(obj, attr_name)\n if sys.version_info.major == 2:\n return StubUnboundMethod(attr)\n else:\n return StubMethod(obj, attr_name)\n\n if isinstance(attr, (types.BuiltinFunctionType, types.BuiltinMethodType)):\n return StubFunction(obj, attr_name)\n\n # What an absurd type this is ....\n if type(attr).__name__ == 'method-wrapper':\n return StubMethodWrapper(attr)\n\n # This is also slot_descriptor\n if type(attr).__name__ == 'wrapper_descriptor':\n return StubWrapperDescriptor(obj, attr_name)\n\n raise UnsupportedStub(\n \"can't stub %s(%s) of %s\", attr_name, type(attr), obj)", "response": "Stub an attribute of an object. Will return an existing stub if there already is one."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nstubbing an object directly.", "response": "def _stub_obj(obj):\n '''\n Stub an object directly.\n '''\n # Annoying circular reference requires importing here. Would like to see\n # this cleaned up. @AW\n from .mock import Mock\n\n # Return an existing stub\n if isinstance(obj, Stub):\n return obj\n\n # If a Mock object, stub its __call__\n if isinstance(obj, Mock):\n return stub(obj.__call__)\n\n # If passed-in a type, assume that we're going to stub out the creation.\n # See StubNew for the awesome sauce.\n # if isinstance(obj, types.TypeType):\n if hasattr(types, 'TypeType') and isinstance(obj, types.TypeType):\n return StubNew(obj)\n elif hasattr(__builtins__, 'type') and \\\n isinstance(obj, __builtins__.type):\n return StubNew(obj)\n elif inspect.isclass(obj):\n return StubNew(obj)\n\n # I thought that types.UnboundMethodType differentiated these cases but\n # apparently not.\n if isinstance(obj, types.MethodType):\n # Handle differently if unbound because it's an implicit \"any instance\"\n if getattr(obj, 'im_self', None) is None:\n # Handle the python3 case and py2 filter\n if hasattr(obj, '__self__'):\n if obj.__self__ is not None:\n return StubMethod(obj)\n if sys.version_info.major == 2:\n return StubUnboundMethod(obj)\n else:\n return StubMethod(obj)\n\n # These aren't in the types library\n if type(obj).__name__ == 'method-wrapper':\n return StubMethodWrapper(obj)\n\n if type(obj).__name__ == 'wrapper_descriptor':\n raise UnsupportedStub(\n \"must call stub(obj,'%s') for slot wrapper on %s\",\n obj.__name__, obj.__objclass__.__name__)\n\n # (Mostly) Lastly, look for properties.\n # First look for the situation where there's a reference back to the\n # property.\n prop = obj\n if isinstance(getattr(obj, '__self__', None), property):\n obj = prop.__self__\n\n # Once we've found a property, we have to figure out how to reference\n # back to the owning class. This is a giant pain and we have to use gc\n # to find out where it comes from. This code is dense but resolves to\n # something like this:\n # >>> gc.get_referrers( foo.x )\n # [{'__dict__': ,\n # 'x': ,\n # '__module__': '__main__',\n # '__weakref__': ,\n # '__doc__': None}]\n if isinstance(obj, property):\n klass, attr = None, None\n for ref in gc.get_referrers(obj):\n if klass and attr:\n break\n if isinstance(ref, dict) and ref.get('prop', None) is obj:\n klass = getattr(\n ref.get('__dict__', None), '__objclass__', None)\n for name, val in getattr(klass, '__dict__', {}).items():\n if val is obj:\n attr = name\n break\n # In the case of PyPy, we have to check all types that refer to\n # the property, and see if any of their attrs are the property\n elif isinstance(ref, type):\n # Use dir as a means to quickly walk through the class tree\n for name in dir(ref):\n if getattr(ref, name) == obj:\n klass = ref\n attr = name\n break\n\n if klass and attr:\n rval = stub(klass, attr)\n if prop != obj:\n return stub(rval, prop.__name__)\n return rval\n\n # If a function and it has an associated module, we can mock directly.\n # Note that this *must* be after properties, otherwise it conflicts with\n # stubbing out the deleter methods and such\n # Sadly, builtin functions and methods have the same type, so we have to\n # use the same stub class even though it's a bit ugly\n if isinstance(obj, (types.FunctionType, types.BuiltinFunctionType,\n types.BuiltinMethodType)) and hasattr(obj, '__module__'):\n return StubFunction(obj)\n\n raise UnsupportedStub(\"can't stub %s\", obj)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef unmet_expectations(self):\n '''\n Assert that all expectations on the stub have been met.\n '''\n unmet = []\n for exp in self._expectations:\n if not exp.closed(with_counts=True):\n unmet.append(ExpectationNotSatisfied(exp))\n return unmet", "response": "Assert that all expectations on the stub have been met."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncleaning up all expectations and restore the original attribute of the mocked object.", "response": "def teardown(self):\n '''\n Clean up all expectations and restore the original attribute of the\n mocked object.\n '''\n if not self._torn:\n self._expectations = []\n self._torn = True\n self._teardown()"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef expect(self):\n '''\n Add an expectation to this stub. Return the expectation.\n '''\n exp = Expectation(self)\n self._expectations.append(exp)\n return exp", "response": "Add an expectation to this stub. Return the expectation."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef spy(self):\n '''\n Add a spy to this stub. Return the spy.\n '''\n spy = Spy(self)\n self._expectations.append(spy)\n return spy", "response": "Add a spy to this stub. Return the spy."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef call_orig(self, *args, **kwargs):\n '''\n Calls the original function.\n '''\n if hasattr(self._obj, '__self__') and \\\n inspect.isclass(self._obj.__self__) and \\\n self._obj.__self__ is self._instance:\n return self._obj.__func__(self._instance, *args, **kwargs)\n elif hasattr(self._obj, 'im_self') and \\\n inspect.isclass(self._obj.im_self) and \\\n self._obj.im_self is self._instance:\n return self._obj.im_func(self._instance, *args, **kwargs)\n else:\n return self._obj(*args, **kwargs)", "response": "Calls the original function."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nputting the original method back in place. This will also handle the special case when it putting back a class method. The following code snippet best describe why it fails using settar, the class method would be replaced with a bound method not a class method. >>> class Example(object): ... @classmethod ... def a_classmethod(self): ... pass ... >>> Example.__dict__['a_classmethod'] >>> orig = getattr(Example, 'a_classmethod') >>> orig > >>> setattr(Example, 'a_classmethod', orig) >>> Example.__dict__['a_classmethod'] > The only way to figure out if this is a class method is to check and see if the bound method im_self is a class, if so then we need to wrap the function object (im_func) with class method before setting it back on the class.", "response": "def _teardown(self):\n '''\n Put the original method back in place. This will also handle the\n special case when it putting back a class method.\n\n The following code snippet best describe why it fails using settar,\n the class method would be replaced with a bound method not a class\n method.\n\n >>> class Example(object):\n ... @classmethod\n ... def a_classmethod(self):\n ... pass\n ...\n >>> Example.__dict__['a_classmethod']\n \n >>> orig = getattr(Example, 'a_classmethod')\n >>> orig\n >\n >>> setattr(Example, 'a_classmethod', orig)\n >>> Example.__dict__['a_classmethod']\n >\n\n The only way to figure out if this is a class method is to check and\n see if the bound method im_self is a class, if so then we need to wrap\n the function object (im_func) with class method before setting it back\n on the class.\n '''\n # Figure out if this is a class method and we're unstubbing it on the\n # class to which it belongs. This addresses an edge case where a\n # module can expose a method of an instance. e.g gevent.\n if hasattr(self._obj, '__self__') and \\\n inspect.isclass(self._obj.__self__) and \\\n self._obj.__self__ is self._instance:\n setattr(\n self._instance, self._attr, classmethod(self._obj.__func__))\n elif hasattr(self._obj, 'im_self') and \\\n inspect.isclass(self._obj.im_self) and \\\n self._obj.im_self is self._instance:\n # Wrap it and set it back on the class\n setattr(self._instance, self._attr, classmethod(self._obj.im_func))\n else:\n setattr(self._instance, self._attr, self._obj)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _teardown(self):\n '''\n Replace the original method.\n '''\n if not self._was_object_method:\n setattr(self._instance, self._attr, self._obj)\n else:\n delattr(self._instance, self._attr)", "response": "Replace the original method.\n Replace the original method.\n Delete the original object."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncalls the original function. Simulates __new__ and __init__ together.", "response": "def call_orig(self, *args, **kwargs):\n '''\n Calls the original function. Simulates __new__ and __init__ together.\n '''\n rval = super(StubNew, self).call_orig(self._type)\n rval.__init__(*args, **kwargs)\n return rval"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\noverloading so that we can clear out the cache after a test run.", "response": "def _teardown(self):\n '''\n Overload so that we can clear out the cache after a test run.\n '''\n # __new__ is a super-special case in that even when stubbing a class\n # which implements its own __new__ and subclasses object, the\n # \"Class.__new__\" reference is a staticmethod and not a method (or\n # function). That confuses the \"was_object_method\" logic in\n # StubFunction which then fails to delattr and from then on the class\n # is corrupted. So skip that teardown and use a __new__-specific case.\n setattr(self._instance, self._attr, staticmethod(self._new))\n StubNew._cache.pop(self._type)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef call_orig(self, *args, **kwargs):\n '''\n Calls the original function.\n '''\n return self._orig(self._obj, *args, **kwargs)", "response": "Calls the original function."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nperform the RFXtrx initialisation protocol in a Future. Currently this is the rough workflow of the interactions with the RFXtrx. We also do a few extra things - flush the buffer, and attach readers/writers to the asyncio loop. 1. Write a RESET packet (write all zeros) 2. Wait at least 50ms and less than 9000ms 3. Write the STATUS packet to verify the device is up. 4. Receive status response 5. Write the MODE packet to enable or disabled the required protocols.", "response": "def _setup(self):\n \"\"\"Performs the RFXtrx initialisation protocol in a Future.\n\n Currently this is the rough workflow of the interactions with the\n RFXtrx. We also do a few extra things - flush the buffer, and attach\n readers/writers to the asyncio loop.\n\n 1. Write a RESET packet (write all zeros)\n 2. Wait at least 50ms and less than 9000ms\n 3. Write the STATUS packet to verify the device is up.\n 4. Receive status response\n 5. Write the MODE packet to enable or disabled the required protocols.\n \"\"\"\n self.log.info(\"Adding reader to prepare to receive.\")\n self.loop.add_reader(self.dev.fd, self.read)\n\n self.log.info(\"Flushing the RFXtrx buffer.\")\n self.flushSerialInput()\n\n self.log.info(\"Writing the reset packet to the RFXtrx. (blocking)\")\n yield from self.sendRESET()\n\n self.log.info(\"Wating 0.4s\")\n yield from asyncio.sleep(0.4)\n\n self.log.info(\"Write the status packet (blocking)\")\n yield from self.sendSTATUS()\n\n # TODO receive status response, compare it with the needed MODE and\n # request a new MODE if required. Currently MODE is always sent.\n\n self.log.info(\"Adding mode packet to the write queue (blocking)\")\n yield from self.sendMODE()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding the callback to the event loop.", "response": "def do_callback(self, pkt):\n \"\"\"Add the callback to the event loop, we use call soon because we just\n want it to be called at some point, but don't care when particularly.\n \"\"\"\n callback, parser = self.get_callback_parser(pkt)\n\n if asyncio.iscoroutinefunction(callback):\n self.loop.call_soon_threadsafe(self._do_async_callback,\n callback, parser)\n else:\n self.loop.call_soon(callback, parser)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef read(self):\n\n data = self.dev.read()\n\n if len(data) == 0:\n self.log.warning(\"READ : Nothing received\")\n return\n\n if data == b'\\x00':\n self.log.warning(\"READ : Empty packet (Got \\\\x00)\")\n return\n\n pkt = bytearray(data)\n data = self.dev.read(pkt[0])\n pkt.extend(bytearray(data))\n\n self.log.info(\"READ : %s\" % self.format_packet(pkt))\n self.do_callback(pkt)\n return pkt", "response": "Read a single packet from the device and pass to the callback."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nexecutes an HTTP DELETE request for the given URL.", "response": "def delete(self, url, params=None):\n \"\"\" Executes an HTTP DELETE request for the given URL.\n\n ``params`` should be a dictionary\n \"\"\"\n response = self.http.delete(url,\n params=params,\n **self.requests_params)\n return self.process(response)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get(self, url, data=None):\n response = self.http.get(url,\n headers=self.headers,\n params=data,\n **self.requests_params)\n return self.process(response)", "response": "Executes an HTTP GET request for the given URL."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nexecutes an HTTP POST request for the given URL.", "response": "def post(self, url, body=None):\n \"\"\" Executes an HTTP POST request for the given URL. \"\"\"\n response = self.http.post(url,\n headers=self.headers,\n data=body,\n **self.requests_params)\n\n return self.process(response)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nexecute an HTTP PUT request for the given URL.", "response": "def put(self, url, data=None, body=None):\n \"\"\" Executes an HTTP PUT request for the given URL. \"\"\"\n response = self.http.put(url,\n headers=self.headers,\n data=body,\n params=data,\n **self.requests_params)\n\n return self.process(response)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef process(self, response):\n\n try:\n code = response.status_code\n\n # 204 - No Content\n if code == 204:\n body = None\n # add an error message to 402 errors\n elif code == 402:\n body = {\n \"message\": \"Payment Required\",\n \"status\": \"error\"\n }\n else:\n body = response.json()\n\n return Response(code, body, response.content, response)\n except ValueError:\n raise ZencoderResponseError(response, response.content)", "response": "Returns HTTP backend agnostic Response data."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create(self, email, tos=1, options=None):\n data = {'email': email,\n 'terms_of_service': str(tos)}\n if options:\n data.update(options)\n\n return self.post(self.base_url, body=json.dumps(data))", "response": "Creates an account with Zencoder no API Key necessary."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create(self, input=None, live_stream=False, outputs=None, options=None):\n data = {\"input\": input, \"test\": self.test}\n if outputs:\n data['outputs'] = outputs\n\n if options:\n data.update(options)\n\n if live_stream:\n data['live_stream'] = live_stream\n\n return self.post(self.base_url, body=json.dumps(data))", "response": "Creates a new transcoding job."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef resubmit(self, job_id):\n url = self.base_url + '/%s/resubmit' % str(job_id)\n return self.put(url)", "response": "Resubmits the given job_id."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncanceling the given job_id.", "response": "def cancel(self, job_id):\n \"\"\" Cancels the given ``job_id``.\n\n https://app.zencoder.com/docs/api/jobs/cancel\n\n \"\"\"\n if self.version == 'v1':\n verb = self.get\n else:\n verb = self.put\n\n url = self.base_url + '/%s/cancel' % str(job_id)\n return verb(url)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef minutes(self, start_date=None, end_date=None, grouping=None):\n\n data = self.__format(start_date, end_date)\n\n url = self.base_url + '/minutes'\n return self.get(url, data=data)", "response": "Gets a detailed report of encoded minutes and billable minutes for a specific date range."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef route(app_or_blueprint, context=default_context, **kwargs):\n def decorator(fn):\n fn = describe(**kwargs)(fn)\n transmute_func = TransmuteFunction(fn)\n routes, handler = create_routes_and_handler(transmute_func, context)\n for r in routes:\n # push swagger info.\n if not hasattr(app_or_blueprint, SWAGGER_ATTR_NAME):\n setattr(app_or_blueprint, SWAGGER_ATTR_NAME, SwaggerSpec())\n swagger_obj = getattr(app_or_blueprint, SWAGGER_ATTR_NAME)\n swagger_obj.add_func(transmute_func, context)\n app_or_blueprint.route(r, methods=transmute_func.methods)(handler)\n return handler\n return decorator", "response": "attach a transmute route."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nstubbing an object. If attr is not None, will attempt to stub that attribute on the object. Only required for modules and other rare cases where we can't determine the binding from the object.", "response": "def stub(self, obj, attr=None):\n '''\n Stub an object. If attr is not None, will attempt to stub that\n attribute on the object. Only required for modules and other rare\n cases where we can't determine the binding from the object.\n '''\n s = stub(obj, attr)\n if s not in self._stubs:\n self._stubs.append(s)\n return s"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef mock(self, obj=None, attr=None, **kwargs):\n '''\n Return a mock object.\n '''\n rval = Mock(**kwargs)\n if obj is not None and attr is not None:\n rval._object = obj\n rval._attr = attr\n\n if hasattr(obj, attr):\n orig = getattr(obj, attr)\n self._mocks.append((obj, attr, orig))\n setattr(obj, attr, rval)\n else:\n self._mocks.append((obj, attr))\n setattr(obj, attr, rval)\n return rval", "response": "Return a mock object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating an error from a validation exception.", "response": "def from_validation_exception(cls, exception, **kwargs):\n \"\"\"Create an error from validation exception.\"\"\"\n errors = []\n\n def flatten(error, path=\"\"):\n if isinstance(error, halogen.exceptions.ValidationError):\n if not path.endswith(\"/\"):\n path += \"/\"\n if error.attr is not None:\n path += error.attr\n elif error.index is not None:\n path += six.text_type(error.index)\n\n for e in error.errors:\n flatten(e, path)\n else:\n message = error\n if isinstance(error, Exception):\n try:\n message = error.message\n except AttributeError:\n message = six.text_type(error)\n # TODO: i18n\n errors.append(Error(message=message, path=path))\n\n flatten(exception)\n message = kwargs.pop(\"message\", \"Validation error.\")\n return cls(message=message, errors=sorted(errors, key=lambda error: error.path or \"\"), **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngive a bytes object should return a base python data structure that represents the object.", "response": "def load(raw_bytes):\n \"\"\"\n given a bytes object, should return a base python data\n structure that represents the object.\n \"\"\"\n try:\n return yaml.load(raw_bytes)\n except yaml.scanner.ScannerError as e:\n raise SerializationException(str(e))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add_transmute_route(self, *args):\n if len(args) == 1:\n fn = args[0]\n elif len(args) == 3:\n methods, paths, fn = args\n fn = describe(methods=methods, paths=paths)(fn)\n else:\n raise ValueError(\n \"expected one or three arguments for add_transmute_route!\"\n )\n\n add_route(self._app, fn, context=self._transmute_context)", "response": "add a transmute route to the current application"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn True if the structure class can be used to handle the given class.", "response": "def can_handle(self, cls):\n \"\"\"\n this will theoretically be compatible with everything,\n as cattrs can handle many basic types as well.\n \"\"\"\n # cattrs uses a Singledispatch like function\n # under the hood.\n f = self._cattrs_converter._structure_func.dispatch(cls)\n return f != self._cattrs_converter._structure_default"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef load(self, model, value):\n try:\n return self._cattrs_converter.structure(value, model)\n except (ValueError, TypeError) as e:\n raise SerializationException(str(e))", "response": "Converts unstructured data into structured data recursively."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts attrs data into unstructured data with basic types recursively.", "response": "def dump(self, model, value):\n \"\"\"\n Convert attrs data into unstructured data with basic types, recursively:\n\n - attrs classes => dictionaries\n - Enumeration => values\n - Other types are let through without conversion,\n such as, int, boolean, dict, other classes.\n \"\"\"\n try:\n return self._cattrs_converter.unstructure(value)\n except (ValueError, TypeError) as e:\n raise SerializationException(str(e))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef detect_encoding(fp, default=None):\n init_pos = fp.tell()\n try:\n sample = fp.read(\n current_app.config.get('PREVIEWER_CHARDET_BYTES', 1024))\n # Result contains 'confidence' and 'encoding'\n result = cchardet.detect(sample)\n threshold = current_app.config.get('PREVIEWER_CHARDET_CONFIDENCE', 0.9)\n if result.get('confidence', 0) > threshold:\n return result.get('encoding', default)\n else:\n return default\n except Exception:\n current_app.logger.warning('Encoding detection failed.', exc_info=True)\n return default\n finally:\n fp.seek(init_pos)", "response": "Detect the cahracter encoding of a file pointer."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef is_type(value):\n if isinstance(value, type):\n return issubclass(value, Type)\n return isinstance(value, Type)", "response": "Determine if value is an instance or subclass of the class Type."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef serialize(self, value, **kwargs):\n return [self.item_type.serialize(val, **kwargs) for val in value]", "response": "Serialize every item of the list."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef amount_object_to_dict(self, amount):\n currency, amount = (\n amount.as_quantized(digits=2).as_tuple()\n if not isinstance(amount, dict)\n else (amount[\"currency\"], amount[\"amount\"])\n )\n if currency not in self.currencies:\n raise ValueError(self.err_unknown_currency.format(currency=currency))\n return {\n \"amount\": str(amount),\n \"currency\": str(currency),\n }", "response": "Return the dictionary representation of an Amount object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef load(raw_bytes):\n try:\n if not isinstance(raw_bytes, string_type):\n raw_bytes = raw_bytes.decode()\n return json.loads(raw_bytes)\n except ValueError as e:\n raise SerializationException(str(e))", "response": "Given a bytes object should return a base python data\n structure that represents the object."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\npreviews file for given record.", "response": "def preview(pid, record, template=None, **kwargs):\n \"\"\"Preview file for given record.\n\n Plug this method into your ``RECORDS_UI_ENDPOINTS`` configuration:\n\n .. code-block:: python\n\n RECORDS_UI_ENDPOINTS = dict(\n recid=dict(\n # ...\n route='/records/',\n view_imp='invenio_previewer.views.preview',\n record_class='invenio_records_files.api:Record',\n )\n )\n \"\"\"\n # Get file from record\n fileobj = current_previewer.record_file_factory(\n pid, record, request.view_args.get(\n 'filename', request.args.get('filename', type=str))\n )\n if not fileobj:\n abort(404)\n\n # Try to see if specific previewer is requested?\n try:\n file_previewer = fileobj['previewer']\n except KeyError:\n file_previewer = None\n\n # Find a suitable previewer\n fileobj = PreviewFile(pid, record, fileobj)\n for plugin in current_previewer.iter_previewers(\n previewers=[file_previewer] if file_previewer else None):\n if plugin.can_preview(fileobj):\n try:\n return plugin.preview(fileobj)\n except Exception:\n current_app.logger.warning(\n ('Preview failed for {key}, in {pid_type}:{pid_value}'\n .format(key=fileobj.file.key,\n pid_type=fileobj.pid.pid_type,\n pid_value=fileobj.pid.pid_value)),\n exc_info=True)\n return default.preview(fileobj)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfunction for creating a new template script or tool.", "response": "def new(template, target=None, name=None):\n \"\"\"\n Function for creating a template script or tool.\n \n :param template: template to be used ; one of TEMPLATES\n :param target: type of script/tool to be created\n :param name: name of the new script/tool\n \"\"\"\n if template not in TEMPLATES:\n raise ValueError(\"Template argument must be one of the followings: {}\"\n .format(\", \".join(TEMPLATES)))\n if target is not None and target not in TARGETS.keys():\n raise ValueError(\"Target argument must be one of the followings: {}\"\n .format(TARGETS.keys()))\n name = name or template\n if NAME_REGEX.match(name) is None:\n raise ValueError(\"Invalid {} name\".format(template))\n with open(\"{}.py\".format(name), 'w') as f:\n target_imp = \"\" if target is None else \"from {} import {}\\n\" \\\n .format(*target.split('.'))\n main = MAIN.format(base=TARGETS.get(target) or \"\")\n if template == \"script\":\n f.write(SHEBANG + IMPORTS.format(target=target_imp) + \"\\n\\n\" + main)\n elif template == \"tool\":\n f.write(SHEBANG + TOOL_METADATA + TOOL_SECTIONS \\\n .format(imports=IMPORTS.format(target=target_imp)) + main)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding a swagger. json route and a swagger. html route", "response": "def add_swagger(app, json_route, html_route, **kwargs):\n \"\"\"\n a convenience method for both adding a swagger.json route,\n as well as adding a page showing the html documentation\n \"\"\"\n app.route(json_route)(create_swagger_json_handler(app, **kwargs))\n add_swagger_api_route(app, html_route, json_route)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nadd a route to the app that will serve the swagger api.", "response": "def add_swagger_api_route(app, target_route, swagger_json_route):\n \"\"\"\n mount a swagger statics page.\n\n app: the flask app object\n target_route: the path to mount the statics page.\n swagger_json_route: the path where the swagger json definitions is\n expected to be.\n \"\"\"\n static_root = get_swagger_static_root()\n swagger_body = generate_swagger_html(\n STATIC_ROOT, swagger_json_route\n ).encode(\"utf-8\")\n\n def swagger_ui():\n return Response(swagger_body, content_type=\"text/html\")\n\n blueprint = Blueprint('swagger', __name__,\n static_url_path=STATIC_ROOT,\n static_folder=static_root)\n app.route(target_route)(swagger_ui)\n app.register_blueprint(blueprint)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a handler that returns the swagger definition for an application.", "response": "def create_swagger_json_handler(app, **kwargs):\n \"\"\"\n Create a handler that returns the swagger definition\n for an application.\n\n This method assumes the application is using the\n TransmuteUrlDispatcher as the router.\n \"\"\"\n\n spec = getattr(app, SWAGGER_ATTR_NAME, SwaggerSpec())\n _add_blueprint_specs(app, spec)\n spec_dict = spec.swagger_definition(**kwargs)\n encoded_spec = json.dumps(spec_dict).encode(\"UTF-8\")\n\n def swagger():\n return Response(\n encoded_spec,\n # we allow CORS, so this can be requested at swagger.io\n headers={\n \"Access-Control-Allow-Origin\": \"*\"\n },\n content_type=\"application/json\",\n )\n\n return swagger"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncommands for working with test data.", "response": "def fixtures():\n \"\"\"Command for working with test data.\"\"\"\n temp_path = os.path.join(os.path.dirname(__file__), 'temp')\n demo_files_path = os.path.join(os.path.dirname(__file__), 'demo_files')\n\n # Create location\n loc = Location(name='local', uri=temp_path, default=True)\n db.session.add(loc)\n db.session.commit()\n\n # Example files from the data folder\n demo_files = (\n 'markdown.md',\n 'csvfile.csv',\n 'zipfile.zip',\n 'jsonfile.json',\n 'xmlfile.xml',\n 'notebook.ipynb',\n 'jpgfile.jpg',\n 'pngfile.png',\n )\n\n rec_uuid = uuid4()\n provider = RecordIdProvider.create(object_type='rec', object_uuid=rec_uuid)\n data = {\n 'pid_value': provider.pid.pid_value,\n }\n\n record = Record.create(data, id_=rec_uuid)\n bucket = Bucket.create()\n RecordsBuckets.create(record=record.model, bucket=bucket)\n\n # Add files to the record\n for f in demo_files:\n with open(os.path.join(demo_files_path, f), 'rb') as fp:\n record.files[f] = fp\n\n record.files.flush()\n record.commit()\n db.session.commit()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrender HTML from Markdown file content.", "response": "def render(file):\n \"\"\"Render HTML from Markdown file content.\"\"\"\n with file.open() as fp:\n encoding = detect_encoding(fp, default='utf-8')\n result = mistune.markdown(fp.read().decode(encoding))\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nvalidate the length of a list.", "response": "def validate(self, value):\n \"\"\"Validate the length of a list.\n\n :param value: List of values.\n\n :raises: :class:`halogen.exception.ValidationError` exception when length of the list is less than\n minimum or greater than maximum.\n \"\"\"\n try:\n length = len(value)\n except TypeError:\n length = 0\n\n if self.min_length is not None:\n min_length = self.min_length() if callable(self.min_length) else self.min_length\n if length < min_length:\n raise exceptions.ValidationError(self.min_err.format(min_length))\n\n if self.max_length is not None:\n max_length = self.max_length() if callable(self.max_length) else self.max_length\n if length > max_length:\n raise exceptions.ValidationError(self.max_err.format(max_length))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef validate(self, value):\n if self.min is not None:\n min_value = self.min() if callable(self.min) else self.min\n if value < min_value:\n raise exceptions.ValidationError(self.min_err.format(val=value, min=min_value))\n\n if self.max is not None:\n max_value = self.max() if callable(self.max) else self.max\n if value > max_value:\n raise exceptions.ValidationError(self.max_err.format(val=value, max=max_value))", "response": "Validate value.\n\n :param value: Value which should be validated.\n\n :raises: :class:`halogen.exception.ValidationError` exception when either if value less than min in case when\n min is not None or if value greater than max in case when max is not None."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef __get_calls_from_parser(proxy_parser, real_parser):\n __parsers[proxy_parser] = real_parser\n for method, safe, args, kwargs, proxy_subparser in proxy_parser.calls:\n args = (__proxy_to_real_parser(v) for v in args)\n kwargs = {k: __proxy_to_real_parser(v) for k, v in kwargs.items()}\n real_subparser = getattr(real_parser, method)(*args, **kwargs)\n if real_subparser is not None:\n __get_calls_from_parser(proxy_subparser, real_subparser)", "response": "This actually executes the calls registered in the ProxyArgumentParser."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef __proxy_to_real_parser(value):\n if isinstance(value, ProxyArgumentParser):\n return __parsers[value]\n elif any(isinstance(value, t) for t in [list, tuple]):\n new_value = []\n for subvalue in iter(value):\n new_value.append(__proxy_to_real_parser(subvalue))\n return new_value\n return value", "response": "This recursively converts ProxyArgumentParser instances to actual parsers."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef initialize(glob,\n sudo=False,\n multi_level_debug=False,\n add_config=False,\n add_demo=False,\n add_interact=False,\n add_step=False,\n add_time=False,\n add_version=False,\n add_wizard=False,\n ext_logging=False,\n noargs_action=None,\n report_func=None):\n \"\"\"\n Initialization function ; sets up the arguments for the parser and creates a\n logger to be inserted in the input dictionary of global variables from the\n calling script.\n\n :param glob: globals() instance from the calling script\n :param sudo: if True, require sudo credentials and re-run\n script with sudo\n :param multi_level_debug: allow to use -v, -vv, -vvv (adjust logging level)\n instead of just -v (only debug on/off)\n :param relative_time: use relative time for log messages\n :param add_config: add an option to input an INI configuration file\n :param add_demo: add an option to re-run the process using a random\n entry from the __examples__ (only works if this\n variable is populated)\n :param add_interact: add an interaction option\n :param add_step: add an execution stepping option\n :param add_time: add an execution timing option\n :param add_version: add a version option\n :param add_wizard: add an option to run a wizard, asking for each\n input argument\n :param ext_logging: extended logging options\n :param noargs_action: action to be performed when no argument is input\n :param report_func: report generation function\n \"\"\"\n global parser, __parsers\n \n add = {'config': add_config, 'demo': add_demo, 'interact': add_interact,\n 'step': add_step, 'time': add_time, 'version': add_version,\n 'wizard': add_wizard, 'help': True}\n p = ArgumentParser(glob)\n # 1) handle action when no input argument is given\n add['demo'] = add['demo'] and p.examples\n noarg = len(sys.argv) == 1\n if noarg and noargs_action:\n if noargs_action not in add.keys():\n raise ValueError(\"Bad action when no args (should be one of: {})\"\n .format('|'.join(add.keys())))\n add[noargs_action] = True # ensure this action is enabled, even if it\n # is not given the passed arguments\n # 2) populate the real parser and add information arguments\n __parsers = {}\n __get_calls_from_parser(parser, p)\n # config handling feature, for reading/writing an INI config file with the\n # input arguments, e.g. for future reuse\n if add['config']:\n c = p.add_argument_group(gt(\"config arguments\"))\n opt = c.add_argument(\"-r\", \"--read-config\", action='config',\n help=gt(\"read args from a config file\"),\n note=gt(\"this overrides other arguments\"))\n c.add_argument(\"-w\", \"--write-config\", metavar=\"INI\",\n help=gt(\"write args to a config file\"))\n if noarg and noargs_action == \"config\":\n sys.argv[1:] = [opt, \"config.ini\"]\n i = p.add_argument_group(gt(\"extra arguments\"))\n # demonstration feature, for executing an example amongst these defined in\n # __examples__, useful for observing what the tool does\n if add['demo']:\n opt = i.add_argument(\"--demo\", action='demo', prefix=\"play\",\n help=gt(\"demonstrate a random example\"))\n if noarg and noargs_action == \"demo\":\n sys.argv[1:] = [opt]\n # help feature, for displaying classical or extended help about the tool\n if add['help']:\n if glob.get('__details__'):\n opt = i.add_argument(\"-h\", dest=\"help\", default=0, action=\"count\",\n help=gt(\"show extended help message and exit\"),\n note=gt(\"-hhh is the highest help detail level\"))\n else:\n opt = i.add_argument(\"-h\", \"--help\", action='help', prefix=\"show\",\n help=gt(\"show this help message and exit\"))\n if noarg and noargs_action == \"help\":\n sys.argv[1:] = [opt]\n # interaction mode feature, for interacting with the tool during its\n # execution, useful for debugging when it is complex\n if add['interact']:\n j = p.add_argument_group(gt(\"interaction arguments\"))\n opt = j.add_argument(\"--interact\", action=\"store_true\",\n suffix=\"mode\", help=gt(\"interaction mode\"))\n if opt:\n j.add_argument(\"--host\", default=\"127.0.0.1\", type=ip_address,\n prefix=\"remote\", help=gt(\"remote interacting host\"))\n j.add_argument(\"--port\", default=12345, type=port_number,\n prefix=\"remote\", help=gt(\"remote interacting port\"))\n if noarg and noargs_action == \"interact\":\n sys.argv[1:] = [opt]\n # stepping mode feature, for stepping within the tool during its execution,\n # especially useful for debugging when it is complex\n if add['step']:\n opt = i.add_argument(\"--step\", action=\"store_true\", last=True,\n suffix=\"mode\", help=gt(\"stepping mode\"))\n if noarg and noargs_action == \"step\":\n sys.argv[1:] = [opt]\n # timing mode feature, for measuring time along the execution of the tool\n if add['time']:\n b = p.add_argument_group(gt(\"timing arguments\"))\n opt = b.add_argument(\"--stats\", action='store_true', last=True,\n prefix=\"time\",\n help=gt(\"display execution time stats at exit\"))\n b.add_argument(\"--timings\", action='store_true', last=True,\n suffix=\"mode\",\n help=gt(\"display time stats during execution\"))\n if noarg and noargs_action == \"time\":\n sys.argv[1:] = [opt]\n # version feature, for displaying the version from __version__\n if add['version']:\n version = glob['__version__'] if '__version__' in glob else None\n if version is not None:\n opt = i.add_argument(\"--version\", action='version', prefix=\"show\",\n version=version, help=gt(\"show program's \"\n \"version number and exit\"))\n if noarg and noargs_action == \"version\":\n sys.argv[1:] = [opt]\n # verbosity feature, for displaying debugging messages, with the\n # possibility to handle multi-level verbosity\n if multi_level_debug:\n i.add_argument(\"-v\", dest=\"verbose\", default=0, action=\"count\",\n suffix=\"mode\", cancel=True, last=True,\n help=gt(\"verbose level\"),\n note=gt(\"-vvv is the highest verbosity level\"))\n else:\n i.add_argument(\"-v\", \"--verbose\", action=\"store_true\", last=True,\n suffix=\"mode\", help=gt(\"verbose mode\"))\n # wizard feature, for asking argument values to the user\n if add['wizard']:\n opt = i.add_argument(\"-w\", \"--wizard\", action='wizard',\n prefix=\"start\", help=gt(\"start a wizard\"))\n if noarg and noargs_action == \"wizard\":\n sys.argv[1:] = [opt]\n # reporting feature, for making a reporting with the results of the tool\n # at the end of its execution\n if report_func is not None and PYTHON3:\n if not isfunction(report_func):\n report_func = None\n glob['logger'].error(\"Bad report generation function\")\n return\n # lazily import report features\n # -> reason: they rely on pandas and weasyprint, which take time to be\n # imported ; so, when report features are not used in a\n # script, report classes won't be loaded\n all_list = __import__(\"tinyscript.report\", fromlist=['__all__']).__all__\n report = __import__(\"tinyscript.report\", fromlist=all_list)\n for f in all_list:\n glob[f] = globals()[f] = getattr(report, f)\n # now populate the parser with report-related arguments\n r = p.add_argument_group(gt(\"report arguments\"))\n output_func = list(filter(lambda x: not x[0].startswith('_'),\n getmembers(Report, predicate=isfunction)))\n choices = list(map(lambda x: x[0], output_func))\n if r.add_argument(\"--output\", choices=choices, default=\"pdf\", last=True,\n prefix=\"report\", help=gt(\"report output format\")):\n r.add_argument(\"--title\", last=True, prefix=\"report\",\n help=gt(\"report title\"))\n r.add_argument(\"--css\", last=True, prefix=\"report\",\n help=gt(\"report stylesheet file\"))\n r.add_argument(\"--theme\", default=\"default\", last=True,\n prefix=\"report\", help=gt(\"report stylesheet theme\"),\n note=gt(\"--css overrides this setting\"))\n r.add_argument(\"--filename\", last=True, prefix=\"report\",\n help=gt(\"report filename\"))\n elif report_func is not None and not PYTHON3:\n glob['logger'].warning(\"Report generation is only for Python 3\")\n # extended logging features\n if ext_logging:\n i.add_argument(\"-f\", \"--logfile\", last=True,\n help=gt(\"destination log file\"))\n i.add_argument(\"-r\", \"--relative\", action=\"store_true\", last=True,\n suffix=\"time\", help=gt(\"display relative time\"))\n if LINUX:\n i.add_argument(\"-s\", \"--syslog\", action=\"store_true\", last=True,\n suffix=\"mode\", help=gt(\"log to /var/log/syslog\"))\n glob['args'], glob['parser'] = p.parse_args(), p\n # 3) if sudo required, restart the script\n if sudo:\n # if not root, restart the script in another process and jump to this\n if os.geteuid() != 0:\n python = [] if glob['__file__'].startswith(\"./\") else [\"python\"]\n os.execvp(\"sudo\", [\"sudo\"] + python + sys.argv)\n # 4) configure logging and get the main logger\n configure_logger(glob, multi_level_debug,\n glob['args']._collisions.get(\"relative\"),\n glob['args']._collisions.get(\"logfile\"),\n glob['args']._collisions.get(\"syslog\"))\n # 5) append modes items\n set_interact_items(glob)\n set_step_items(glob)\n set_time_items(glob)\n # 6) finally, bind the global exit handler\n def __at_exit():\n # first, dump the config if required\n if add['config']:\n _save_config(glob)\n # then handle the state\n if _hooks.state == \"INTERRUPTED\":\n glob['at_interrupt']()\n elif _hooks.state == \"TERMINATED\":\n glob['at_terminate']()\n else:\n if report_func is not None and PYTHON3:\n # generate the report only when exiting gracefully, just before\n # the user-defined function at_graceful_exit\n _ = glob['args']\n r = Report(*report_func(), title=_.title, filename=_.filename,\n logger=glob['logger'], css=_.css)\n getattr(r, _.output)(False)\n t = glob['time_manager']\n if add['time'] and t._stats:\n t.stats()\n glob['at_graceful_exit']()\n glob['at_exit']()\n logging.shutdown()\n atexit.register(__at_exit)", "response": "Initializes the global parser and the global arguments for the a\n logger."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef validate(glob, *arg_checks):\n locals().update(glob) # allows to import user-defined objects from glob\n # into the local scope\n if glob['args'] is None or glob['logger'] is None:\n return\n exit_app = False\n for check in arg_checks:\n check = check + (None, ) * (4 - len(check))\n param, condition, message, default = check\n if re.match(r'^_?[a-zA-Z][a-zA-Z0-9_]*$', param) is None:\n raise ValueError(\"Illegal argument name\")\n try:\n result = eval(condition.replace(\" ? \", \" glob['args'].{} \"\n .format(param)))\n except Exception as e:\n result = True\n message = str(e)\n if result:\n if default is None:\n glob['logger'].error(message or \"Validation failed\")\n exit_app = True\n else:\n glob['logger'].warning(message or \"Validation failed\")\n setattr(glob['args'], param, default)\n if exit_app:\n sys.exit(2)", "response": "Function for validating group of arguments"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef mse(x, xhat):\n buf_ = x - xhat\n np.square(buf_, out=buf_) # square in-place\n sum_ = np.sum(buf_)\n sum_ /= x.size # divide in-place\n return sum_", "response": "Calcualte mse between vector or matrix x and xhat"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef low_rank_approximation(A, r):\n try:\n u, s, v = np.linalg.svd(A, full_matrices=False)\n except np.linalg.LinAlgError as e:\n print('Matrix:', A)\n print('Matrix rank:', np.linalg.matrix_rank(A))\n raise\n\n Ar = np.zeros((len(u), len(v)), dtype=u.dtype)\n buf_ = np.empty_like(Ar)\n sc_vec_ = np.empty((v.shape[1],), dtype=v.dtype)\n for i in range(r):\n np.multiply(v[i], s[i], out=sc_vec_)\n np.outer(u[:, i], sc_vec_, out=buf_)\n Ar += buf_\n return Ar", "response": "Returns an approximation of A of rank r in least - squares sense."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef eigendecomp(G, d):\n N = G.shape[0]\n lamda, u = np.linalg.eig(G)\n # test decomposition of G.\n #G_hat = np.dot(np.dot(u, np.diag(lamda)), u.T)\n #assert np.linalg.norm(G_hat - G) < 1e-10, 'decomposition not exact: err {}'.format(np.linalg.norm(G_hat - G))\n\n # sort the eigenvalues in decreasing order\n lamda = np.real(lamda)\n indices = np.argsort(lamda)[::-1]\n lamda_sorted = lamda[indices]\n assert (lamda_sorted[\n :d] > -1e-10).all(), \"{} not all positive!\".format(lamda_sorted[:d])\n\n u = u[:, indices]\n factor = np.empty((N,), dtype=lamda.dtype)\n np.sqrt(lamda_sorted[:d], out=factor[0:d])\n factor[d:] = 0.0\n return factor, np.real(u)", "response": "Computes the sorted eigendecomposition of G."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the vector xhat closest to x in 2 - norm satisfying A. xhat = b.", "response": "def projection(x, A, b):\n \"\"\" Returns the vector xhat closest to x in 2-norm, satisfying A.xhat =b.\n\n :param x: vector\n :param A, b: matrix and array characterizing the constraints on x (A.x = b)\n\n :return x_hat: optimum angle vector, minimizing cost.\n :return cost: least square error of xhat, x\n :return constraints_error: mse of constraint.\n :rtype: (numpy.ndarray, float, float)\n \"\"\"\n A_pseudoinv = pseudo_inverse(A)\n tmp_ = A.dot(x)\n tmp_ -= b\n x_hat = A_pseudoinv.dot(tmp_)\n np.subtract(x, x_hat, out=x_hat)\n\n cost = mse(x_hat, x)\n A.dot(x_hat, out=tmp_)\n constraints_error = mse(tmp_, b)\n return x_hat, cost, constraints_error"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nbuilds a list of all of the comparators that can be used for arguments.", "response": "def build_comparators(*values_or_types):\n '''\n All of the comparators that can be used for arguments.\n '''\n comparators = []\n for item in values_or_types:\n if isinstance(item, Comparator):\n comparators.append(item)\n elif isinstance(item, type):\n # If you are passing around a type you will have to build a Equals\n # comparator\n comparators.append(Any(IsA(item), Is(item)))\n else:\n comparators.append(Equals(item))\n return comparators"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef args(self, *args, **kwargs):\n self._any_args = False\n self._arguments_rule.set_args(*args, **kwargs)\n return self", "response": "Adds a new ArgumentsExpectationRule to the expectation"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the value for this expectation or raises the proper exception.", "response": "def return_value(self):\n \"\"\"\n Returns the value for this expectation or raises the proper exception.\n \"\"\"\n if self._raises:\n # Handle exceptions\n if inspect.isclass(self._raises):\n raise self._raises()\n else:\n raise self._raises\n else:\n if isinstance(self._returns, tuple):\n return tuple([x.value if isinstance(x, Variable)\n else x for x in self._returns])\n return self._returns.value if isinstance(self._returns, Variable) \\\n else self._returns"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nchecks if these args match this expectation.", "response": "def match(self, *args, **kwargs):\n \"\"\"\n Check the if these args match this expectation.\n \"\"\"\n return self._any_args or \\\n self._arguments_rule.validate(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef record_file_factory(self):\n try:\n get_distribution('invenio-records-files')\n from invenio_records_files.utils import record_file_factory\n default = record_file_factory\n except DistributionNotFound:\n def default(pid, record, filename):\n return None\n\n return load_or_import_from_config(\n 'PREVIEWER_RECORD_FILE_FACOTRY',\n app=self.app,\n default=default,\n )", "response": "Load default record file factory."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nregistering a previewer in the system.", "response": "def register_previewer(self, name, previewer):\n \"\"\"Register a previewer in the system.\"\"\"\n if name in self.previewers:\n assert name not in self.previewers, \\\n \"Previewer with same name already registered\"\n self.previewers[name] = previewer\n if hasattr(previewer, 'previewable_extensions'):\n self._previewable_extensions |= set(\n previewer.previewable_extensions)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\niterates over the previewers.", "response": "def iter_previewers(self, previewers=None):\n \"\"\"Get previewers ordered by PREVIEWER_PREVIEWERS_ORDER.\"\"\"\n if self.entry_point_group is not None:\n self.load_entry_point_group(self.entry_point_group)\n self.entry_point_group = None\n\n previewers = previewers or \\\n self.app.config.get('PREVIEWER_PREFERENCE', [])\n\n for item in previewers:\n if item in self.previewers:\n yield self.previewers[item]"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ninitializes the Flask application.", "response": "def init_app(self, app, entry_point_group='invenio_previewer.previewers'):\n \"\"\"Flask application initialization.\"\"\"\n self.init_config(app)\n app.register_blueprint(blueprint)\n state = _InvenioPreviewerState(\n app,\n entry_point_group=entry_point_group)\n app.extensions['invenio-previewer'] = state\n return state"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nparses a 9 bytes packet in the Humidity format and return a dictionary containing the data extracted.", "response": "def parse(self, data):\n \"\"\"Parse a 9 bytes packet in the Humidity format and return a\n dictionary containing the data extracted. An example of a return value\n would be:\n\n .. code-block:: python\n\n {\n 'id': \"0x2EB2\",\n 'packet_length': 8,\n 'packet_type': 81,\n 'packet_type_name': 'Humidity sensors',\n 'sequence_number': 0,\n 'packet_subtype': 1,\n 'packet_subtype_name': \"LaCrosse TX3\",\n 'humidity': 91,\n 'humidity_status': \"Wet\"\n 'signal_level': 9,\n 'battery_level': 6,\n }\n\n :param data: bytearray to be parsed\n :type data: bytearray\n\n :return: Data dictionary containing the parsed values\n :rtype: dict\n \"\"\"\n\n self.validate_packet(data)\n\n id_ = self.dump_hex(data[4:6])\n # channel = data[5] TBC\n\n humidity = data[6]\n humidity_status = self._extract_humidity_status(data[7])\n\n sensor_specific = {\n 'id': id_,\n # 'channel': channel, TBC\n 'humidity': humidity,\n 'humidity_status': humidity_status\n }\n\n results = self.parse_header_part(data)\n results.update(RfxPacketUtils.parse_signal_and_battery(data[8]))\n results.update(sensor_specific)\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a pipeline component from a factory.", "response": "def create_tokenizer(self, name, config=dict()):\n \"\"\"Create a pipeline component from a factory.\n\n name (unicode): Factory name to look up in `Language.factories`.\n config (dict): Configuration parameters to initialise component.\n RETURNS (callable): Pipeline component.\n \"\"\"\n if name not in self.factories:\n raise KeyError(Errors.E002.format(name=name))\n factory = self.factories[name]\n return factory(self, **config)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nadding error codes to string messages via class attribute names.", "response": "def add_codes(err_cls):\n \"\"\"Add error codes to string messages via class attribute names.\"\"\"\n class ErrorsWithCodes(object):\n def __getattribute__(self, code):\n msg = getattr(err_cls, code)\n return '[{code}] {msg}'.format(code=code, msg=msg)\n return ErrorsWithCodes()"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndisplays a warning message.", "response": "def _warn(message, warn_type='user'):\n \"\"\"\n message (unicode): The message to display.\n category (Warning): The Warning to show.\n \"\"\"\n w_id = message.split('[', 1)[1].split(']', 1)[0] # get ID from string\n if warn_type in SPACY_WARNING_TYPES and w_id not in SPACY_WARNING_IGNORE:\n category = WARNINGS[warn_type]\n stack = inspect.stack()[-1]\n with warnings.catch_warnings():\n if SPACY_WARNING_FILTER:\n warnings.simplefilter(SPACY_WARNING_FILTER, category)\n warnings.warn_explicit(message, category, stack[1], stack[2])"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef download(model=None, direct=False, *pip_args):\n if model is None:\n model = about.__default_corpus__\n\n if direct:\n dl = download_model('{m}/{m}.tar.gz#egg={m}'.format(m=model), pip_args)\n else:\n shortcuts = get_json(about.__shortcuts__, \"available shortcuts\")\n model_name = shortcuts.get(model, model)\n compatibility = get_compatibility()\n version = get_version(model_name, compatibility)\n dl = download_model('{m}-{v}/{m}-{v}.tar.gz#egg={m}=={v}'\n .format(m=model_name, v=version), pip_args)\n if dl != 0: # if download subprocess doesn't return 0, exit\n sys.exit(dl)\n try:\n # Get package path here because link uses\n # pip.get_installed_distributions() to check if model is a\n # package, which fails if model was just installed via\n # subprocess\n package_path = get_package_path(model_name)\n link(model_name, model, force=True, model_path=package_path)\n except:\n # Dirty, but since spacy.download and the auto-linking is\n # mostly a convenience wrapper, it's best to show a success\n # message and loading instructions, even if linking fails.\n prints(Messages.M001.format(name=model_name), title=Messages.M002)", "response": "Download compatible model from default download path using pip."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a symlink for models within the spacy data directory.", "response": "def link(origin, link_name, force=False, model_path=None):\n \"\"\"\n Create a symlink for models within the spacy/data directory. Accepts\n either the name of a pip package, or the local path to the model data\n directory. Linking models allows loading them via spacy.load(link_name).\n \"\"\"\n if util.is_package(origin):\n model_path = util.get_package_path(origin)\n else:\n model_path = Path(origin) if model_path is None else Path(model_path)\n if not model_path.exists():\n prints(Messages.M009.format(path=path2str(model_path)),\n title=Messages.M008, exits=1)\n data_path = util.get_data_path()\n if not data_path or not data_path.exists():\n spacy_loc = Path(__file__).parent.parent\n prints(Messages.M011, spacy_loc, title=Messages.M010, exits=1)\n link_path = util.get_data_path() / link_name\n if link_path.is_symlink() and not force:\n prints(Messages.M013, title=Messages.M012.format(name=link_name),\n exits=1)\n elif link_path.is_symlink(): # does a symlink exist?\n # NB: It's important to check for is_symlink here and not for exists,\n # because invalid/outdated symlinks would return False otherwise.\n link_path.unlink()\n elif link_path.exists(): # does it exist otherwise?\n # NB: Check this last because valid symlinks also \"exist\".\n prints(Messages.M015, link_path,\n title=Messages.M014.format(name=link_name), exits=1)\n msg = \"%s --> %s\" % (path2str(model_path), path2str(link_path))\n try:\n symlink_to(link_path, model_path)\n except:\n # This is quite dirty, but just making sure other errors are caught.\n prints(Messages.M017, msg, title=Messages.M016.format(name=link_name))\n raise\n prints(msg, Messages.M019.format(name=link_name), title=Messages.M018)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load_model_from_path(model_path, meta=False, **overrides):\n from .tokenizer_loader import TokenizerLoader\n\n if not meta:\n meta = get_model_meta(model_path)\n tokenizer_loader = TokenizerLoader(meta=meta, **overrides)\n tokenizers = meta.get('tokenizers', [])\n disable = overrides.get('disable', [])\n if tokenizers is True:\n tokenizers = TokenizerLoader.Defaults.tokenizers\n elif tokenizers in (False, None):\n tokenizers = []\n for tokenizer_name in tokenizers:\n if tokenizer_name not in disable:\n config = meta.get('tokenizer_args', {}).get(tokenizer_name, {})\n component = tokenizer_loader.create_tokenizer(tokenizer_name, config=config)\n tokenizer_loader.add_tokenizer(component, name=tokenizer_name)\n return tokenizer_loader.from_disk(model_path)", "response": "Loads a language model from a data directory path. Creates a pipeline from meta. json and then calls from_disk() with path."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef is_package(name):\n name = name.lower() # compare package name against lowercase name\n packages = pkg_resources.working_set.by_key.keys()\n for package in packages:\n if package.lower().replace('-', '_') == name:\n return True\n return False", "response": "Check if string maps to a package installed via pip."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef decaying(start, stop, decay):\n def clip(value):\n return max(value, stop) if (start > stop) else min(value, stop)\n nr_upd = 1.\n while True:\n yield clip(start * 1./(1. + decay * nr_upd))\n nr_upd += 1", "response": "Yields an infinite series of linearly decaying values."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nopen and load JSON from file.", "response": "def read_json(location):\n \"\"\"Open and load JSON from file.\n\n location (Path): Path to JSON file.\n RETURNS (dict): Loaded JSON content.\n \"\"\"\n location = ensure_path(location)\n with location.open('r', encoding='utf8') as f:\n return ujson.load(f)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting user input from command line via raw_input.", "response": "def get_raw_input(description, default=False):\n \"\"\"Get user input from the command line via raw_input / input.\n\n description (unicode): Text to display before prompt.\n default (unicode or False/None): Default value to display with prompt.\n RETURNS (unicode): User input.\n \"\"\"\n additional = ' (default: %s)' % default if default else ''\n prompt = ' %s%s: ' % (description, additional)\n user_input = input_(prompt)\n return user_input"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nprints data in table format.", "response": "def print_table(data, title=None):\n \"\"\"Print data in table format.\n\n data (dict or list of tuples): Label/value pairs.\n title (unicode or None): Title, will be printed above.\n \"\"\"\n if isinstance(data, dict):\n data = list(data.items())\n tpl_row = ' {:<15}' * len(data[0])\n table = '\\n'.join([tpl_row.format(l, unicode_(v)) for l, v in data])\n if title:\n print('\\n \\033[93m{}\\033[0m'.format(title))\n print('\\n{}\\n'.format(table))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nprinting data in GitHub - flavoured Markdown format for issues etc.", "response": "def print_markdown(data, title=None):\n \"\"\"Print data in GitHub-flavoured Markdown format for issues etc.\n\n data (dict or list of tuples): Label/value pairs.\n title (unicode or None): Title, will be rendered as headline 2.\n \"\"\"\n def excl_value(value):\n # contains path, i.e. personal info\n return isinstance(value, basestring_) and Path(value).exists()\n\n if isinstance(data, dict):\n data = list(data.items())\n markdown = [\"* **{}:** {}\".format(l, unicode_(v))\n for l, v in data if not excl_value(v)]\n if title:\n print(\"\\n## {}\".format(title))\n print('\\n{}\\n'.format('\\n'.join(markdown)))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nprinting formatted message (manual ANSI escape sequences to avoid dependency) *texts (unicode): Texts to print. Each argument is rendered as paragraph. **kwargs: 'title' becomes coloured headline. exits=True performs sys exit.", "response": "def prints(*texts, **kwargs):\n \"\"\"Print formatted message (manual ANSI escape sequences to avoid\n dependency)\n\n *texts (unicode): Texts to print. Each argument is rendered as paragraph.\n **kwargs: 'title' becomes coloured headline. exits=True performs sys exit.\n \"\"\"\n exits = kwargs.get('exits', None)\n title = kwargs.get('title', None)\n title = '\\033[93m{}\\033[0m\\n'.format(_wrap(title)) if title else ''\n message = '\\n\\n'.join([_wrap(text) for text in texts])\n print('\\n{}{}\\n'.format(title, message))\n if exits is not None:\n sys.exit(exits)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nwraps text at given width using textwrap module.", "response": "def _wrap(text, wrap_max=80, indent=4):\n \"\"\"Wrap text at given width using textwrap module.\n\n text (unicode): Text to wrap. If it's a Path, it's converted to string.\n wrap_max (int): Maximum line length (indent is deducted).\n indent (int): Number of spaces for indentation.\n RETURNS (unicode): Wrapped text.\n \"\"\"\n indent = indent * ' '\n wrap_width = wrap_max - len(indent)\n if isinstance(text, Path):\n text = path2str(text)\n return textwrap.fill(text, width=wrap_width, initial_indent=indent,\n subsequent_indent=indent, break_long_words=False,\n break_on_hyphens=False)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef escape_html(text):\n text = text.replace('&', '&')\n text = text.replace('<', '<')\n text = text.replace('>', '>')\n text = text.replace('\"', '"')\n return text", "response": "Escapes HTML in a displaCy text."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngive a dictionary make sure keys are unicode strings not bytes.", "response": "def normalize_string_keys(old):\n \"\"\"Given a dictionary, make sure keys are unicode strings, not bytes.\"\"\"\n new = {}\n for key, value in old.items():\n if isinstance(key, bytes_):\n new[key.decode('utf8')] = value\n else:\n new[key] = value\n return new"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef locale_escape(string, errors='replace'):\n '''\n Mangle non-supported characters, for savages with ascii terminals.\n '''\n encoding = locale.getpreferredencoding()\n string = string.encode(encoding, errors).decode('utf8')\n return string", "response": "Escape a string using the locale."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse_cmu(cmufh):\n pronunciations = list()\n for line in cmufh:\n line = line.strip().decode('utf-8')\n if line.startswith(';'):\n continue\n word, phones = line.split(\" \", 1)\n pronunciations.append((word.split('(', 1)[0].lower(), phones))\n return pronunciations", "response": "Parses an incoming CMU pronouncing dictionary file."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ninitializes the module s pronunciation data.", "response": "def init_cmu(filehandle=None):\n \"\"\"Initialize the module's pronunciation data.\n\n This function is called automatically the first time you attempt to use\n another function in the library that requires loading the pronunciation\n data from disk. You can call this function manually to control when and\n how the pronunciation data is loaded (e.g., you're using this module in\n a web application and want to load the data asynchronously).\n\n :param filehandle: a filehandle with CMUdict-formatted data\n :returns: None\n \"\"\"\n global pronunciations, lookup, rhyme_lookup\n if pronunciations is None:\n if filehandle is None:\n filehandle = cmudict.dict_stream()\n pronunciations = parse_cmu(filehandle)\n filehandle.close()\n lookup = collections.defaultdict(list)\n for word, phones in pronunciations:\n lookup[word].append(phones)\n rhyme_lookup = collections.defaultdict(list)\n for word, phones in pronunciations:\n rp = rhyming_part(phones)\n if rp is not None:\n rhyme_lookup[rp].append(word)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef rhyming_part(phones):\n phones_list = phones.split()\n for i in range(len(phones_list) - 1, 0, -1):\n if phones_list[i][-1] in '12':\n return ' '.join(phones_list[i:])\n return phones", "response": "Get the rhyming part of a string with CMUdict phones."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef search(pattern):\n init_cmu()\n regexp = re.compile(r\"\\b\" + pattern + r\"\\b\")\n return [word\n for word, phones in pronunciations\n if regexp.search(phones)]", "response": "Search the CMU dictionary for pronunciations matching a regular expression."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsearching the words whose stress pattern matches a regular expression.", "response": "def search_stresses(pattern):\n \"\"\"Get words whose stress pattern matches a regular expression.\n\n This function is a special case of :func:`search` that searches only the\n stress patterns of each pronunciation in the dictionary. You can get\n stress patterns for a word using the :func:`stresses_for_word` function.\n\n .. doctest::\n\n >>> import pronouncing\n >>> pronouncing.search_stresses('020120')\n ['gubernatorial']\n\n :param pattern: a string containing a regular expression\n :returns: a list of matching words\n \"\"\"\n init_cmu()\n regexp = re.compile(pattern)\n return [word\n for word, phones in pronunciations\n if regexp.search(stresses(phones))]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a list of words rhyming with a given word.", "response": "def rhymes(word):\n \"\"\"Get words rhyming with a given word.\n\n This function may return an empty list if no rhyming words are found in\n the dictionary, or if the word you pass to the function is itself not\n found in the dictionary.\n\n .. doctest::\n\n >>> import pronouncing\n >>> pronouncing.rhymes(\"conditioner\")\n ['commissioner', 'parishioner', 'petitioner', 'practitioner']\n\n :param word: a word\n :returns: a list of rhyming words\n \"\"\"\n phones = phones_for_word(word)\n combined_rhymes = []\n if phones:\n for element in phones:\n combined_rhymes.append([w for w in rhyme_lookup.get(rhyming_part(\n element), []) if w != word])\n combined_rhymes = list(chain.from_iterable(combined_rhymes))\n unique_combined_rhymes = sorted(set(combined_rhymes))\n return unique_combined_rhymes\n else:\n return []"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef set_ds9(self, level=\"PREF\"):\n self.set_zoom()\n ds9_settings = config.read(\"DS9.\"+level)\n for key in ds9_settings.keys():\n value = ds9_settings[key]\n cmd = key.replace(\"_\", \" \")\n self.ds9.set(\"{} {}\".format(cmd, value))", "response": "Set the default values on the ds9 display."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndoes a query of the CADC Megacam table.", "response": "def cfht_megacam_tap_query(ra_deg=180.0, dec_deg=0.0, width=1, height=1, date=None):\n \"\"\"Do a query of the CADC Megacam table.\n\n Get all observations inside the box (right now it turns width/height into a radius, should not do this).\n\n @rtype : Table\n @param ra_deg: center of search region, in degrees\n @param dec_deg: center of search region in degrees\n @param width: width of search region in degrees\n @param height: height of search region in degrees\n @param date: ISO format date string. Query will be +/- 0.5 days from date given.\n \"\"\"\n\n radius = min(90, max(width, height) / 2.0)\n\n query = (\"SELECT \"\n \"COORD1(CENTROID(Plane.position_bounds)) AS RAJ2000,\"\n \"COORD2(CENTROID(Plane.position_bounds)) AS DEJ2000,\"\n \"target_name \"\n \"FROM \"\n \"caom2.Observation as o \"\n \"JOIN caom2.Plane as Plane on o.obsID=Plane.obsID \"\n \"WHERE o.collection = 'CFHT' \"\n \"AND o.instrument_name = 'MegaPrime' \"\n \"AND INTERSECTS( CIRCLE('ICRS', %f, %f, %f), Plane.position_bounds ) = 1\")\n\n query = query % (ra_deg, dec_deg, radius)\n\n if date is not None:\n mjd = Time(date, scale='utc').mjd\n query += \" AND Plane.time_bounds_lower <= {} AND {} <= Plane.time_bounds_upper \".format(mjd+0.5, mjd-0.5)\n\n data = {\"QUERY\": query,\n \"REQUEST\": \"doQuery\",\n \"LANG\": \"ADQL\",\n \"FORMAT\": \"votable\"}\n\n url = \"http://www.cadc.hia.nrc.gc.ca/tap/sync\"\n\n warnings.simplefilter('ignore')\n ff = StringIO(requests.get(url, params=data).content)\n ff.seek(0)\n table = votable.parse(ff).get_first_table().to_table()\n assert isinstance(table, Table)\n return table"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef validate(self, size):\n msg = 'scale and array size must match, ' \\\n 'but were scale: {self.scale.n_bands}, array size: {size}'\n\n if size != len(self.scale):\n raise ValueError(msg.format(**locals()))", "response": "Raises a ValueError if the size of the dimension doesn t match the number of bands in the array"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a basic processing pipeline that can resample all incoming audio to a normalized sampling rate for downstream processing.", "response": "def resampled(\n chunksize_bytes=DEFAULT_CHUNK_SIZE,\n resample_to=SR44100(),\n store_resampled=False):\n \"\"\"\n Create a basic processing pipeline that can resample all incoming audio\n to a normalized sampling rate for downstream processing, and store a\n convenient, compressed version for playback\n\n :param chunksize_bytes: The number of bytes from the raw stream to process\n at once\n :param resample_to: The new, normalized sampling rate\n :return: A simple processing pipeline\n \"\"\"\n\n class Resampled(BaseModel):\n meta = JSONFeature(\n MetaData,\n store=True,\n encoder=AudioMetaDataEncoder)\n\n raw = ByteStreamFeature(\n ByteStream,\n chunksize=chunksize_bytes,\n needs=meta,\n store=False)\n\n ogg = OggVorbisFeature(\n OggVorbis,\n needs=raw,\n store=True)\n\n pcm = AudioSamplesFeature(\n AudioStream,\n needs=raw,\n store=False)\n\n resampled = AudioSamplesFeature(\n Resampler,\n needs=pcm,\n samplerate=resample_to,\n store=store_resampled)\n\n return Resampled"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef audio_graph(\n chunksize_bytes=DEFAULT_CHUNK_SIZE,\n resample_to=SR44100(),\n store_fft=False):\n \"\"\"\n Produce a base class suitable as a starting point for many audio processing\n pipelines. This class resamples all audio to a common sampling rate, and\n produces a bark band spectrogram from overlapping short-time fourier\n transform frames. It also compresses the audio into ogg vorbis format for\n compact storage.\n \"\"\"\n\n band = FrequencyBand(20, resample_to.nyquist)\n\n class AudioGraph(BaseModel):\n meta = JSONFeature(\n MetaData,\n store=True,\n encoder=AudioMetaDataEncoder)\n\n raw = ByteStreamFeature(\n ByteStream,\n chunksize=chunksize_bytes,\n needs=meta,\n store=False)\n\n ogg = OggVorbisFeature(\n OggVorbis,\n needs=raw,\n store=True)\n\n pcm = AudioSamplesFeature(\n AudioStream,\n needs=raw,\n store=False)\n\n resampled = AudioSamplesFeature(\n Resampler,\n needs=pcm,\n samplerate=resample_to,\n store=False)\n\n windowed = ArrayWithUnitsFeature(\n SlidingWindow,\n needs=resampled,\n wscheme=HalfLapped(),\n wfunc=OggVorbisWindowingFunc(),\n store=False)\n\n dct = ArrayWithUnitsFeature(\n DCT,\n needs=windowed,\n store=True)\n\n fft = ArrayWithUnitsFeature(\n FFT,\n needs=windowed,\n store=store_fft)\n\n bark = ArrayWithUnitsFeature(\n BarkBands,\n needs=fft,\n frequency_band=band,\n store=True)\n\n centroid = ArrayWithUnitsFeature(\n SpectralCentroid,\n needs=bark,\n store=True)\n\n chroma = ArrayWithUnitsFeature(\n Chroma,\n needs=fft,\n frequency_band=band,\n store=True)\n\n bfcc = ArrayWithUnitsFeature(\n BFCC,\n needs=fft,\n store=True)\n\n return AudioGraph", "response": "Generates a base class suitable as a starting point for many audio processing of a single object."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nproduces a mixin class that extracts onsets from a short - time fourier transform feature", "response": "def with_onsets(fft_feature):\n \"\"\"\n Produce a mixin class that extracts onsets\n :param fft_feature: The short-time fourier transform feature\n :return: A mixin class that extracts onsets\n \"\"\"\n\n class Onsets(BaseModel):\n onset_prep = ArrayWithUnitsFeature(\n SlidingWindow,\n needs=fft_feature,\n wscheme=HalfLapped() * Stride(frequency=1, duration=3),\n store=False)\n\n complex_domain = ArrayWithUnitsFeature(\n ComplexDomain,\n needs=onset_prep,\n store=False)\n\n sliding_detection = ArrayWithUnitsFeature(\n SlidingWindow,\n needs=complex_domain,\n wscheme=HalfLapped() * Stride(frequency=1, duration=11),\n padwith=5,\n store=False)\n\n slices = TimeSliceFeature(\n MovingAveragePeakPicker,\n needs=sliding_detection,\n aggregate=np.median,\n store=True)\n\n return Onsets"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef keplerian_sheared_field_locations(ax, kbos, date, ras, decs, names, elongation=False, plot=False):\n seps = {'dra': 0., 'ddec': 0.}\n for kbo in kbos:\n ra = kbo.ra\n dec = kbo.dec\n kbo.compute(date)\n seps['dra'] += kbo.ra - ra\n seps['ddec'] += kbo.dec - dec\n\n seps['dra'] /= float(len(kbos))\n seps['ddec'] /= float(len(kbos))\n\n print date, seps, len(kbos)\n\n for idx in range(len(ras)):\n name = names[idx]\n ra = ras[idx] + seps['dra']\n dec = decs[idx] + seps['ddec']\n if plot:\n ax.add_artist(Rectangle(xy=(math.degrees(ra) - camera_dimen / 2.0, math.degrees(dec) - camera_dimen / 2.0),\n height=camera_dimen,\n width=camera_dimen,\n edgecolor='b',\n facecolor='b',\n lw=0.5, fill=True, alpha=0.2))\n if elongation:\n # For each field centre, plot the elongation onto the field at that date.\n elong = field_elongation(ephem.degrees(ra), ephem.degrees(dec), date)\n ax.annotate(name, (math.degrees(ra) + camera_dimen / 2., math.degrees(dec)), size=3)\n ax.annotate(\"%0.1f\" % elong, (math.degrees(ra) + camera_dimen / 4., math.degrees(dec) - camera_dimen / 4.),\n size=5)\n\n return ax", "response": "Plot the Keplerian sheared field locations."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef field_elongation(ra, dec, date):\n sun = ephem.Sun()\n sun.compute(date)\n sep = ephem.separation((ra, dec), sun)\n retval = 180. - math.degrees(sep)\n\n return retval", "response": "Calculates the solar elongation at the given date."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting string value to timedelta instance according to given format.", "response": "def str_to_timedelta(value: str,\n fmt: str = None) -> Optional[datetime.timedelta]:\n \"\"\"\n Convert string value to timedelta instance according to the given format.\n\n If format not set function tries to load timedelta using default\n ``TIMEDELTA_FORMAT`` and then both of magic \"full\" formats.\n\n You should also specify list of formats and function tries to convert\n to timedelta using each of formats in list. First matched format would\n return the converted timedelta instance.\n\n If user specified format, but function cannot convert string to\n new timedelta instance - ``ValueError`` would be raised. But if user did\n not specify the format, function would be fail silently and return ``None``\n as result.\n\n :param value: String representation of timedelta.\n :param fmt: Format to use for conversion.\n \"\"\"\n def timedelta_kwargs(data: DictStrInt) -> DictStrInt:\n \"\"\"\n Convert day_hours, hour_minutes, minute_seconds, week_days and weeks to\n timedelta seconds.\n \"\"\"\n seconds = data.get('seconds', 0)\n seconds += data.get('day_hours', 0) * 3600\n seconds += data.pop('hour_minutes', 0) * 60\n seconds += data.pop('minute_seconds', 0)\n seconds += data.pop('week_days', 0) * SECONDS_PER_DAY\n seconds += data.pop('weeks', 0) * SECONDS_PER_WEEK\n data.update({'seconds': seconds})\n return data\n\n if not isinstance(value, str):\n raise ValueError(\n 'Value should be a \"str\" instance. You use {0}.'\n .format(type(value)))\n\n user_fmt = fmt\n\n if isinstance(fmt, (list, tuple)):\n formats = list(fmt)\n elif fmt is None:\n formats = [TIMEDELTA_FORMAT, 'F', 'f']\n else:\n formats = [fmt]\n\n locale_data = {\n 'days_label': '({0}|{1})'.format('day', 'days'),\n 'short_days_label': 'd',\n 'short_week_days_label': 'd',\n 'short_weeks_label': 'w',\n 'week_days_label': '({0}|{1})'.format('day', 'days'),\n 'weeks_label': '({0}|{1})'.format('week', 'weeks'),\n }\n regexps = []\n\n for item in formats:\n processed = r'^'\n\n for part in item:\n if part in TIMEDELTA_FORMATS:\n part = TIMEDELTA_FORMATS[part][1] % locale_data\n else:\n part = re.escape(part)\n processed += part\n\n processed += r'$'\n regexps.append(processed)\n\n for regexp in regexps:\n timedelta_re = re.compile(regexp)\n matched = timedelta_re.match(value)\n\n if matched:\n data = {\n key: to_int(value) or 0\n for key, value in matched.groupdict().items()}\n return datetime.timedelta(**timedelta_kwargs(data))\n\n if user_fmt:\n raise ValueError(\n 'Cannot convert {0!r} to timedelta instance, using {1!r} format.'\n .format(value, user_fmt))\n\n return None"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef timedelta_div(first: datetime.timedelta,\n second: datetime.timedelta) -> Optional[float]:\n \"\"\"Implement divison for timedelta instances.\n\n :param first: First timedelta instance.\n :param second: Second timedelta instance.\n \"\"\"\n first_seconds = timedelta_seconds(first)\n second_seconds = timedelta_seconds(second)\n\n if not second_seconds:\n return None\n\n return first_seconds / second_seconds", "response": "Implement divison for timedelta instances."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the number of seconds from a timedelta instance.", "response": "def timedelta_seconds(value: datetime.timedelta) -> int:\n \"\"\"Return full number of seconds from timedelta.\n\n By default, Python returns only one day seconds, not all timedelta seconds.\n\n :param value: Timedelta instance.\n \"\"\"\n return SECONDS_PER_DAY * value.days + value.seconds"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef timedelta_to_str(value: datetime.timedelta, fmt: str = None) -> str:\n # Only ``datetime.timedelta`` instances allowed for this function\n if not isinstance(value, datetime.timedelta):\n raise ValueError(\n 'Value should be a \"datetime.timedelta\" instance. You use {0}.'\n .format(type(value)))\n\n # Generate total data\n days = value.days\n microseconds = value.microseconds\n seconds = timedelta_seconds(value)\n\n hours = seconds // 3600\n minutes = seconds // 60\n weeks = days // 7\n\n # Generate collapsed data\n day_hours = hours - days * 24\n hour_minutes = minutes - hours * 60\n minute_seconds = seconds - minutes * 60\n week_days = days - weeks * 7\n\n days_label = 'day' if days % 10 == 1 else 'days'\n short_days_label = 'd'\n short_week_days_label = 'd'\n short_weeks_label = 'w'\n week_days_label = 'day' if week_days % 10 == 1 else 'days'\n weeks_label = 'week' if weeks % 10 == 1 else 'weeks'\n\n # Collect data\n data = locals()\n\n fmt = fmt or TIMEDELTA_FORMAT\n processed = ''\n\n for part in fmt:\n if part in TIMEDELTA_FORMATS:\n is_full_part = part in ('f', 'F')\n is_repr_part = part in ('r', 'R')\n\n part = TIMEDELTA_FORMATS[part][0]\n\n if is_full_part or is_repr_part:\n if is_repr_part and not days:\n part = part.replace('%(days)d', '')\n part = part.replace('%(days_label)s,', '')\n part = part.replace('%(short_days_label)s', '')\n\n if is_full_part and not minute_seconds:\n part = part.replace(':%(minute_seconds)02d', '')\n\n if is_full_part and not weeks:\n part = part.replace('%(weeks)d', '')\n part = part.replace('%(short_weeks_label)s', '')\n part = part.replace('%(weeks_label)s,', '')\n\n if is_full_part and not week_days:\n part = part.replace('%(week_days)d', '')\n part = part.replace('%(short_week_days_label)s', '')\n part = part.replace('%(week_days_label)s,', '')\n\n part = part.strip()\n part = ' '.join(part.split())\n\n processed += part\n\n return processed % data", "response": "Display the timedelta formatted according to the given string."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a shifts file that transforms the space and flux time scale of all images to the first image.", "response": "def align(expnums, ccd, version='s', dry_run=False):\n \"\"\"Create a 'shifts' file that transforms the space/flux/time scale of all images to the first image.\n\n This function relies on the .fwhm, .trans.jmp, .phot and .zeropoint.used files for inputs.\n The scaling we are computing here is for use in planting sources into the image at the same sky/flux locations\n while accounting for motions of sources with time.\n\n :param expnums: list of MegaPrime exposure numbers to add artificial KBOs to,\n the first frame in the list is the reference.\n :param ccd: which ccd to work on.\n :param version: Add sources to the 'o', 'p' or 's' images\n :param dry_run: don't push results to VOSpace.\n \"\"\"\n\n # Get the images and supporting files that we need from the VOSpace area\n # get_image and get_file check if the image/file is already on disk.\n # re-computed fluxes from the PSF stars and then recompute x/y/flux scaling.\n\n # some dictionaries to hold the various scale\n pos = {}\n apcor = {}\n mags = {}\n zmag = {}\n mjdates = {}\n\n for expnum in expnums:\n filename = storage.get_image(expnum, ccd=ccd, version=version)\n zmag[expnum] = storage.get_zeropoint(expnum, ccd, prefix=None, version=version)\n mjdates[expnum] = float(fits.open(filename)[0].header.get('MJD-OBS'))\n apcor[expnum] = [float(x) for x in open(storage.get_file(expnum,\n ccd=ccd,\n version=version,\n ext=storage.APCOR_EXT)).read().split()]\n keys = ['crval1', 'cd1_1', 'cd1_2', 'crval2', 'cd2_1', 'cd2_2']\n # load the .trans.jmp values into a 'wcs' like dictionary.\n # .trans.jmp maps current frame to reference frame in pixel coordinates.\n # the reference frame of all the frames supplied must be the same.\n shifts = dict(zip(keys, [float(x) for x in open(storage.get_file(expnum,\n ccd=ccd,\n version=version,\n ext='trans.jmp')).read().split()]))\n shifts['crpix1'] = 0.0\n shifts['crpix2'] = 0.0\n # now create a wcs object based on those transforms, this wcs links the current frame's\n # pixel coordinates to the reference frame's pixel coordinates.\n w = get_wcs(shifts)\n\n # get the PHOT file that was produced by the mkpsf routine\n logging.debug(\"Reading .phot file {}\".format(expnum))\n phot = ascii.read(storage.get_file(expnum, ccd=ccd, version=version, ext='phot'), format='daophot')\n\n # compute the small-aperture magnitudes of the stars used in the PSF\n import daophot\n logging.debug(\"Running phot on {}\".format(filename))\n mags[expnum] = daophot.phot(filename,\n phot['XCENTER'],\n phot['YCENTER'],\n aperture=apcor[expnum][0],\n sky=apcor[expnum][1] + 1,\n swidth=apcor[expnum][0],\n zmag=zmag[expnum])\n\n # covert the x/y positions to positions in Frame 1 based on the trans.jmp values.\n logging.debug(\"Doing the XY translation to refrence frame: {}\".format(w))\n (x, y) = w.wcs_pix2world(mags[expnum][\"XCENTER\"], mags[expnum][\"YCENTER\"], 1)\n pos[expnum] = numpy.transpose([x, y])\n # match this exposures PSF stars position against those in the first image of the set.\n logging.debug(\"Matching lists\")\n idx1, idx2 = util.match_lists(pos[expnums[0]], pos[expnum])\n\n # compute the magnitdue offset between the current frame and the reference.\n dmags = numpy.ma.array(mags[expnums[0]][\"MAG\"] - apcor[expnums[0]][2] -\n (mags[expnum][\"MAG\"][idx1] - apcor[expnum][2]),\n mask=idx1.mask)\n dmags.sort()\n logging.debug(\"Computed dmags between input and reference: {}\".format(dmags))\n error_count = 0\n\n error_count += 1\n logging.debug(\"{}\".format(error_count))\n\n # compute the median and determine if that shift is small compared to the scatter.\n try:\n midx = int(numpy.sum(numpy.any([~dmags.mask], axis=0)) / 2.0)\n dmag = float(dmags[midx])\n logging.debug(\"Computed a mag delta of: {}\".format(dmag))\n except Exception as e:\n logging.error(str(e))\n logging.error(\"Failed to compute mag offset between plant and found using: {}\".format(dmags))\n dmag = 99.99\n\n error_count += 1\n logging.debug(\"{}\".format(error_count))\n\n try:\n if math.fabs(dmag) > 3 * (dmags.std() + 0.01):\n logging.warning(\"Magnitude shift {} between {} and {} is large: {}\".format(dmag,\n expnums[0],\n expnum,\n shifts))\n except Exception as e:\n logging.error(str(e))\n\n error_count += 1\n logging.debug(\"{}\".format(error_count))\n\n shifts['dmag'] = dmag\n shifts['emag'] = dmags.std()\n shifts['nmag'] = len(dmags.mask) - dmags.mask.sum()\n shifts['dmjd'] = mjdates[expnums[0]] - mjdates[expnum]\n shift_file = os.path.basename(storage.get_uri(expnum, ccd, version, '.shifts'))\n\n error_count += 1\n logging.debug(\"{}\".format(error_count))\n\n try:\n fh = open(shift_file, 'w')\n fh.write(json.dumps(shifts, sort_keys=True, indent=4, separators=(',', ': ')))\n fh.write('\\n')\n fh.close()\n except Exception as e:\n logging.error(\"Creation of SHIFTS file failed while trying to write: {}\".format(shifts))\n raise e\n\n error_count += 1\n logging.debug(\"{}\".format(error_count))\n\n if not dry_run:\n storage.copy(shift_file, storage.get_uri(expnum, ccd, version, '.shifts'))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the distance between the values in range uniformly or via some function.", "response": "def dist(self):\n \"\"\"\n How should the values in range be sampled, uniformly or via some function.\n :return:\n \"\"\"\n if self.func is None:\n return random.uniform(self.min, self.max)\n if self._dist is None:\n x = numpy.arange(self.min, self.max, (self.max-self.min)/1000.0)\n p = self.func(x).cumsum()\n p -= p.min()\n p /= p.max()\n self._dist = interpolate.interp1d(p, x)\n return self._dist(random.random())"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the next set of values that can be used for an planted object builder.", "response": "def next(self):\n \"\"\"\n :return: a set of values that can be used for an planted object builder.\n \"\"\"\n # x y mag pix rate angle ''/h rate id\n # 912.48 991.06 22.01 57.32 -45.23 10.60 0\n self._n -= 1\n if self._n < 0:\n raise StopIteration()\n return {'x': self.x(), 'y': self.y(), 'mag': self.mag(), 'sky_rate': self.rate(), 'angle': self.angle(),\n 'id': self.id}"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef send_command(self, cmd, params=None, raw=False):\n '''\n Send command to foscam.\n '''\n paramstr = ''\n if params:\n paramstr = urlencode(params)\n paramstr = '&' + paramstr if paramstr else ''\n cmdurl = 'http://%s/cgi-bin/CGIProxy.fcgi?usr=%s&pwd=%s&cmd=%s%s' % (\n self.url,\n self.usr,\n self.pwd,\n cmd,\n paramstr,\n )\n if self.ssl and ssl_enabled:\n cmdurl = cmdurl.replace('http:','https:')\n\n # Parse parameters from response string.\n if self.verbose:\n print ('Send Foscam command: %s' % cmdurl)\n try:\n raw_string = ''\n if self.ssl and ssl_enabled:\n gcontext = ssl.SSLContext(ssl.PROTOCOL_TLSv1) # disable cert\n raw_string = urlopen(cmdurl,context=gcontext, timeout=5).read()\n else:\n raw_string = urlopen(cmdurl,timeout=5).read()\n if raw:\n if self.verbose:\n print ('Returning raw Foscam response: len=%d' % len(raw_string))\n return FOSCAM_SUCCESS, raw_string\n root = ET.fromstring(raw_string)\n except:\n if self.verbose:\n print ('Foscam exception: ' + raw_string)\n return ERROR_FOSCAM_UNAVAILABLE, None\n code = ERROR_FOSCAM_UNKNOWN\n params = OrderedDict()\n for child in root.iter():\n if child.tag == 'result':\n code = int(child.text)\n\n elif child.tag != 'CGI_Result':\n params[child.tag] = unquote(child.text)\n\n if self.verbose:\n print ('Received Foscam response: %s, %s' % (code, params))\n return code, params", "response": "Send a command to foscam."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nexecutes a command and return a parsed response.", "response": "def execute_command(self, cmd, params=None, callback=None, raw=False):\n '''\n Execute a command and return a parsed response.\n '''\n def execute_with_callbacks(cmd, params=None, callback=None, raw=False):\n code, params = self.send_command(cmd, params, raw)\n if callback:\n callback(code, params)\n return code, params\n\n if self.daemon:\n t = Thread(target=execute_with_callbacks,\n args=(cmd, ), kwargs={'params':params, 'callback':callback, 'raw':raw})\n t.daemon = True\n t.start()\n else:\n return execute_with_callbacks(cmd, params, callback, raw)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_ip_info(self, is_dhcp, ip='', gate='', mask='',\n dns1='', dns2='', callback=None):\n '''\n isDHCP: 0(False), 1(True)\n System will reboot automatically to take effect after call this CGI command.\n '''\n params = {'isDHCP': is_dhcp,\n 'ip': ip,\n 'gate': gate,\n 'mask': mask,\n 'dns1': dns1,\n 'dns2': dns2,\n }\n\n return self.execute_command('setIpInfo', params, callback=callback)", "response": "Set the IP information of the current system."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_port_info(self, webport, mediaport, httpsport,\n onvifport, callback=None):\n '''\n Set http port and media port of camera.\n '''\n params = {'webPort' : webport,\n 'mediaPort' : mediaport,\n 'httpsPort' : httpsport,\n 'onvifPort' : onvifport,\n }\n return self.execute_command('setPortInfo', params, callback=callback)", "response": "Set http port and media port of camera."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the wifi list around after refreshWifiList.", "response": "def get_wifi_list(self, startno, callback=None):\n '''\n Get the aps around after refreshWifiList.\n Note: Only 10 aps will be returned one time.\n '''\n params = {'startNo': startno}\n return self.execute_command('getWifiList', params, callback=callback)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsets wifi config. Camera will not connect to AP unless you enject your cable.", "response": "def set_wifi_setting(self, ssid, psk, isenable, isusewifi, nettype,\n encryptype, authmode, keyformat, defaultkey,\n key1='', key2='', key3='', key4='',\n key1len=64, key2len=64, key3len=64, key4len=64,\n callback=None):\n '''\n Set wifi config.\n Camera will not connect to AP unless you enject your cable.\n '''\n params = {'isEnable' : isenable,\n 'isUseWifi' : isusewifi,\n 'ssid' : ssid,\n 'netType' : nettype,\n 'encryptType': encryptype,\n 'psk' : psk,\n 'authMode' : authmode,\n 'keyFormat' : keyformat,\n 'defaultKey' : defaultkey,\n 'key1' : key1,\n 'key2' : key2,\n 'key3' : key3,\n 'key4' : key4,\n 'key1Len' : key1len,\n 'key2Len' : key2len,\n 'key3Len' : key3len,\n 'key4Len' : key4len,\n }\n return self.execute_command('setWifiSetting', params, callback=callback)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef set_upnp_config(self, isenable, callback=None):\n '''\n Set UPnP config\n '''\n params = {'isEnable': isenable}\n return self.execute_command('setUPnPConfig', params, callback=callback)", "response": "Set UPnP config for the current set of UPnP entries."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef set_sub_video_stream_type(self, format, callback=None):\n '''\n Set the stream fromat of sub stream.\n Supported format: (1) H264 : 0\n (2) MotionJpeg 1\n '''\n params = {'format': format}\n return self.execute_command('setSubVideoStreamType',\n params, callback=callback)", "response": "Set the stream fromat of sub stream."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsets the format of the sub stream", "response": "def set_sub_stream_format(self, format, callback=None):\n '''\n Set the stream fromat of sub stream????\n '''\n params = {'format': format}\n return self.execute_command('setSubStreamFormat',\n params, callback=callback)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsets the stream type of main stream", "response": "def set_main_video_stream_type(self, streamtype, callback=None):\n '''\n Set the stream type of main stream\n '''\n params = {'streamType': streamtype}\n return self.execute_command('setMainVideoStreamType',\n params, callback=callback)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set_video_stream_param(self, streamtype, resolution, bitrate,\n framerate, gop, isvbr, callback=None):\n '''\n Set the video stream param of stream N\n streamtype(0~3): Stream N.\n resolution(0~4): 0 720P,\n 1 VGA(640*480),\n 2 VGA(640*360),\n 3 QVGA(320*240),\n 4 QVGA(320*180).\n bitrate: Bit rate of stream type N(20480~2097152).\n framerate: Frame rate of stream type N.\n GOP: P frames between 1 frame of stream type N.\n The suggest value is: X * framerate.\n isvbr: 0(Not in use currently), 1(In use).\n '''\n params = {'streamType': streamtype,\n 'resolution': resolution,\n 'bitRate' : bitrate,\n 'frameRate' : framerate,\n 'GOP' : gop,\n 'isVBR' : isvbr\n }\n return self.execute_command('setVideoStreamParam',\n params, callback=callback)", "response": "Set the video stream param of the current session."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef mirror_video(self, is_mirror, callback=None):\n '''\n Mirror video\n ``is_mirror``: 0 not mirror, 1 mirror\n '''\n params = {'isMirror': is_mirror}\n return self.execute_command('mirrorVideo', params, callback=callback)", "response": "Mirror the video in the current node."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nflip video ``is_flip``: 0 Not flip, 1 Flip", "response": "def flip_video(self, is_flip, callback=None):\n '''\n Flip video\n ``is_flip``: 0 Not flip, 1 Flip\n '''\n params = {'isFlip': is_flip }\n return self.execute_command('flipVideo', params, callback=callback)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef change_password(self, usrname, oldpwd, newpwd, callback=None):\n '''\n Change password.\n '''\n params = {'usrName': usrname,\n 'oldPwd' : oldpwd,\n 'newPwd' : newpwd,\n }\n return self.execute_command('changePassword',\n params, callback=callback)", "response": "Change password of a resource."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset the state of the Infrared LED to the specified mode.", "response": "def set_infra_led_config(self, mode, callback=None):\n '''\n Set Infrared LED configuration\n cmd: setInfraLedConfig\n mode(0,1): 0=Auto mode, 1=Manual mode\n '''\n params = {'mode': mode}\n return self.execute_command('setInfraLedConfig', params, callback=callback)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nset the motion detection on or off of the current config.", "response": "def set_motion_detection(self, enabled=1):\n '''\n Get the current config and set the motion detection on or off\n '''\n result, current_config = self.get_motion_detect_config()\n if result != FOSCAM_SUCCESS:\n return result\n current_config['isEnable'] = enabled\n self.set_motion_detect_config(current_config)\n return FOSCAM_SUCCESS"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset the motion detection on or off for one of the available resources.", "response": "def set_motion_detection1(self, enabled=1):\n '''\n Get the current config and set the motion detection on or off\n '''\n result, current_config = self.get_motion_detect_config1()\n if result != FOSCAM_SUCCESS:\n return result\n current_config['isEnable'] = enabled\n self.set_motion_detect_config1(current_config)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset alarm record config", "response": "def set_alarm_record_config(self, is_enable_prerecord=1,\n prerecord_secs=5, alarm_record_secs=300, callback=None):\n '''\n Set alarm record config\n Return: set result(0-success, -1-error)\n '''\n params = {'isEnablePreRecord': is_enable_prerecord,\n 'preRecordSecs' : prerecord_secs,\n 'alarmRecordSecs' : alarm_record_secs\n }\n return self.execute_command('setAlarmRecordConfig', params, callback=callback)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set_local_alarm_record_config(self, is_enable_local_alarm_record = 1,\n local_alarm_record_secs = 30, callback=None):\n '''\n Set local alarm-record config\n `is_enable_local_alarm_record`: 0 disable, 1 enable\n '''\n params = {'isEnableLocalAlarmRecord': is_enable_local_alarm_record,\n 'localAlarmRecordSecs' : local_alarm_record_secs}\n return self.execute_command('setLocalAlarmRecordConfig', params, callback=callback)", "response": "Set local alarm - record configuration"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsets the frame shipping reference mode of H264 encode stream.", "response": "def set_h264_frm_ref_mode(self, mode=1, callback=None):\n '''\n Set frame shipping reference mode of H264 encode stream.\n params:\n `mode`: see docstr of meth::get_h264_frm_ref_mode\n '''\n params = {'mode': mode}\n return self.execute_command('setH264FrmRefMode', params, callback)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting schedule record config. cmd: setScheduleRecordConfig args: See docstring of meth::get_schedule_record_config", "response": "def set_schedule_record_config(self, is_enable, record_level,\n space_full_mode, is_enable_audio,\n schedule0 = 0, schedule1 = 0, schedule2 = 0,\n schedule3 = 0, schedule4 = 0, schedule5 = 0,\n schedule6 = 0, callback=None):\n '''\n Set schedule record config.\n cmd: setScheduleRecordConfig\n args: See docstring of meth::get_schedule_record_config\n '''\n\n params = {'isEnable' : is_enable,\n 'isEnableAudio': is_enable_audio,\n 'recordLevel' : record_level,\n 'spaceFullMode': space_full_mode,\n 'schedule0' : schedule0,\n 'schedule1' : schedule1,\n 'schedule2' : schedule2,\n 'schedule3' : schedule3,\n 'schedule4' : schedule4,\n 'schedule5' : schedule5,\n 'schedule6' : schedule6,\n }\n return self.execute_command('setScheduleRecordConfig', params, callback=callback)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_record_path(self, path, callback=None):\n '''\n Set Record path: sd/ftp\n cmd: setRecordPath\n param:\n path: (0,SD), (2, FTP)\n '''\n params = {'Path': path}\n return self.execute_command('setRecordPath', params, callback=callback)", "response": "Set the path of the record"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nretrieve log records from camera.", "response": "def get_log(self, offset, count=10, callback=None):\n '''\n Retrieve log records from camera.\n cmd: getLog\n param:\n offset: log offset for first record\n count: number of records to return\n '''\n params = {'offset': offset, 'count': count}\n return self.execute_command('getLog', params, callback=callback)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_apcor(expnum, ccd, version='p', prefix=None):\n uri = get_uri(expnum, ccd, ext=APCOR_EXT, version=version, prefix=prefix)\n apcor_file_name = tempfile.NamedTemporaryFile()\n client.copy(uri, apcor_file_name.name)\n apcor_file_name.seek(0)\n return [float(x) for x in apcor_file_name.readline().split()]", "response": "retrieve the aperture correction for this exposure"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndo a QUERY on the TAP service for all observations that are part of OSSOS (*P05/*P016) where taken after mjd and have calibration 'observable'. @param ra: RA center of search cont @type ra: Quantity @param dec: float degrees @type dec: Quantity @param dra: float degrees @type dra: Quantity @param ddec: float degrees @type ddec: Quantity @param calibration_level: What calibration level must the found image have, @param mjdate: what data must the observation be to @param collection: name of the data collection to be searched. @param use_ssos: USE the SSOIS server to find comparison? False -> Use CAOM2 TAP query. ke on. this is a CAOM2 parameter of CADC, 2 means calibrated data.", "response": "def cone_search(ra, dec, dra=0.01, ddec=0.01, mjdate=None, calibration_level=2, use_ssos=True,\n collection='CFHTMEGAPIPE'):\n \"\"\"Do a QUERY on the TAP service for all observations that are part of OSSOS (*P05/*P016)\n where taken after mjd and have calibration 'observable'.\n\n @param ra: RA center of search cont\n @type ra: Quantity\n @param dec: float degrees\n @type dec: Quantity\n @param dra: float degrees\n @type dra: Quantity\n @param ddec: float degrees\n @type ddec: Quantity\n @param calibration_level: What calibration level must the found image have,\n @param mjdate: what data must the observation be to\n @param collection: name of the data collection to be searched.\n @param use_ssos: USE the SSOIS server to find comparison? False -> Use CAOM2 TAP query.\n ke on.\n this is a CAOM2 parameter of CADC, 2 means calibrated data.\n \"\"\"\n\n if use_ssos:\n ssois_server = \"http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/cadcbin/ssos/fixedssos.pl\"\n params = dict(pos=\"{0:f},{1:f}\".format(ra.to(units.degree).value, dec.to(units.degree).value))\n result = requests.get(ssois_server, params=params)\n table = ascii.read(result.text, format='tab')\n table = table[table['Telescope/Instrument'] == 'CFHT/MegaCam']\n column_name_mapping = {'Image': 'collectionID',\n 'Filter': 'filter',\n 'Exptime': 'exptime'}\n # rename the columns\n for key in column_name_mapping:\n table[key].name = column_name_mapping[key]\n table['collectionID'] = [x[:-1] for x in table['collectionID']]\n # compute the mjdate from the time string.\n table['mjdate'] = Time(table['Date/Time']).mjd\n return table\n\n data = dict(QUERY=(\" SELECT Observation.observationID as collectionID, \"\n \" Plane.time_bounds_lower AS mjdate, \"\n \" Plane.time_exposure AS exptime, \"\n \" Plane.energy_bandpassName as filter\"\n \" FROM caom2.Observation AS Observation \"\n \" JOIN caom2.Plane AS Plane \"\n \" ON Observation.obsID = Plane.obsID \"\n \" WHERE ( Observation.collection = '{}' ) \"\n \" AND Plane.calibrationLevel > {} \"\n \" AND ( Plane.energy_bandpassName LIKE 'r.%' OR Plane.energy_bandpassName LIKE 'gri.%' ) \"\n \" AND ( Observation.proposal_id LIKE '%P05' or Observation.proposal_id LIKE '%P06' )\"\n \" AND Observation.target_name NOT LIKE 'WP%'\"),\n REQUEST=\"doQuery\",\n LANG=\"ADQL\",\n FORMAT=\"tsv\")\n\n data[\"QUERY\"] = data[\"QUERY\"].format(calibration_level, collection)\n data[\"QUERY\"] += (\" AND \"\n \" CONTAINS( BOX('ICRS', {}, {}, {}, {}), \"\n \" Plane.position_bounds ) = 1 \").format(ra.to(units.degree).value, dec.to(units.degree).value,\n dra.to(units.degree).value, ddec.to(units.degree).value)\n if mjdate is not None:\n data[\"QUERY\"] += \" AND Plane.time_bounds_lower < {} AND Plane.time_bounds_cval2 > {} \".format(\n mjdate + 1.0 / 24.0,\n mjdate - 1 / 24.0)\n\n result = requests.get(TAP_WEB_SERVICE, params=data, verify=False)\n\n logger.debug(\"Doing TAP Query using url: %s\" % (str(result.url)))\n\n table_reader = ascii.get_reader(Reader=ascii.Basic)\n table_reader.header.splitter.delimiter = '\\t'\n table_reader.data.splitter.delimiter = '\\t'\n table = table_reader.read(result.text)\n\n logger.debug(str(table))\n return table"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef populate(dataset_name, data_web_service_url=DATA_WEB_SERVICE + \"CFHT\"):\n\n data_dest = get_uri(dataset_name, version='o', ext=FITS_EXT)\n data_source = \"%s/%so.{}\" % (data_web_service_url, dataset_name, FITS_EXT)\n\n mkdir(os.path.dirname(data_dest))\n\n try:\n client.link(data_source, data_dest)\n except IOError as e:\n if e.errno == errno.EEXIST:\n pass\n else:\n raise e\n\n header_dest = get_uri(dataset_name, version='o', ext='head')\n header_source = \"%s/%so.fits.fz?cutout=[0]\" % (\n data_web_service_url, dataset_name)\n try:\n client.link(header_source, header_dest)\n except IOError as e:\n if e.errno == errno.EEXIST:\n pass\n else:\n raise e\n\n header_dest = get_uri(dataset_name, version='p', ext='head')\n header_source = \"%s/%s/%sp.head\" % (\n 'http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/pub', 'CFHTSG', dataset_name)\n try:\n client.link(header_source, header_dest)\n except IOError as e:\n if e.errno == errno.EEXIST:\n pass\n else:\n raise e\n\n return True", "response": "Populate the DBIMAGES and HEADERS of a CFHT dataset."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_cands_uri(field, ccd, version='p', ext='measure3.cands.astrom', prefix=None,\n block=None):\n \"\"\"\n return the nominal URI for a candidate file.\n\n @param field: the OSSOS field name\n @param ccd: which CCD are the candidates on\n @param version: either the 'p', or 's' (scrambled) candidates.\n @param ext: Perhaps we'll change this one day.\n @param prefix: if this is a 'fake' dataset then add 'fk'\n @param block: Which BLOCK of the field are we looking at? eg. 15BS+1+1\n @return:\n \"\"\"\n\n if prefix is None:\n prefix = \"\"\n if len(prefix) > 0:\n prefix += \"_\"\n if len(field) > 0:\n field += \"_\"\n\n if ext is None:\n ext = \"\"\n if len(ext) > 0 and ext[0] != \".\":\n ext = \".{}\".format(ext)\n\n measure3_dir = MEASURE3\n if block is not None:\n measure3_dir + \"/{}\".format(block)\n return \"{}/{}{}{}{}{}\".format(measure3_dir, prefix, field, version, ccd, ext)", "response": "Returns the nominal URI for a candidate file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_uri(expnum, ccd=None,\n version='p', ext=FITS_EXT,\n subdir=None, prefix=None):\n \"\"\"\n Build the uri for an OSSOS image stored in the dbimages\n containerNode.\n\n :rtype : basestring\n expnum: CFHT exposure number\n ccd: CCD in the mosaic [0-35]\n version: one of p,s,o etc.\n dbimages: dbimages containerNode.\n @type subdir: str\n @param expnum: int\n @param ccd:\n @param version:\n @param ext:\n @param subdir:\n @param prefix:\n \"\"\"\n if subdir is None:\n subdir = str(expnum)\n if prefix is None:\n prefix = ''\n uri = os.path.join(DBIMAGES, subdir)\n\n if ext is None:\n ext = ''\n elif len(ext) > 0 and ext[0] != '.':\n ext = '.' + ext\n\n if version is None:\n version = ''\n\n if ccd is None:\n uri = os.path.join(uri,\n '%s%s%s%s' % (prefix, str(expnum),\n version,\n ext))\n else:\n ccd = str(ccd).zfill(2)\n uri = os.path.join(uri,\n 'ccd{}'.format(ccd),\n '%s%s%s%s%s' % (prefix, str(expnum),\n version,\n ccd,\n ext))\n\n return uri", "response": "Build the URI for an OSSOS image stored in the dbimages file."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_tag(expnum, key):\n\n uri = tag_uri(key)\n force = uri not in get_tags(expnum)\n value = get_tags(expnum, force=force).get(uri, None)\n return value", "response": "given a key return the vospace tag value."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a string that represents the process tag for a given CCD number and version.", "response": "def get_process_tag(program, ccd, version='p'):\n \"\"\"\n make a process tag have a suffix indicating which ccd its for.\n @param program: Name of the process that a tag is built for.\n @param ccd: the CCD number that this process ran on.\n @param version: The version of the exposure (s, p, o) that the process ran on.\n @return: The string that represents the processing tag.\n \"\"\"\n return \"%s_%s%s\" % (program, str(version), str(ccd).zfill(2))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_status(task, prefix, expnum, version, ccd, return_message=False):\n key = get_process_tag(prefix+task, ccd, version)\n status = get_tag(expnum, key)\n logger.debug('%s: %s' % (key, status))\n if return_message:\n return status\n else:\n return status == SUCCESS", "response": "Report back status of a given program."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting the processing status of the given program.", "response": "def set_status(task, prefix, expnum, version, ccd, status):\n \"\"\"\n set the processing status of the given program.\n @param task: name of the processing task\n @param prefix: was there a prefix on the exposure number processed?\n @param expnum: exposure number processed.\n @param version: which version of the exposure? (p, s, o)\n @param ccd: the number of the CCD processing.\n @param status: What status to record: \"SUCCESS\" we hope.\n @return: Success?\n \"\"\"\n return set_tag(expnum, get_process_tag(prefix+task, ccd, version), status)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _cutout_expnum(observation, sky_coord, radius):\n\n uri = observation.get_image_uri()\n cutout_filehandle = tempfile.NamedTemporaryFile()\n disposition_filename = client.copy(uri + \"({},{},{})\".format(sky_coord.ra.to('degree').value,\n sky_coord.dec.to('degree').value,\n radius.to('degree').value),\n cutout_filehandle.name,\n disposition=True)\n cutouts = decompose_content_decomposition(disposition_filename)\n\n cutout_filehandle.seek(0)\n hdulist = fits.open(cutout_filehandle)\n hdulist.verify('silentfix+ignore')\n logger.debug(\"Initial Length of HDUList: {}\".format(len(hdulist)))\n\n # Make sure here is a primaryHDU\n if len(hdulist) == 1:\n phdu = fits.PrimaryHDU()\n phdu.header['ORIGIN'] = \"OSSOS\"\n hdulist.insert(0, phdu)\n\n logger.debug(\"Final Length of HDUList: {}\".format(len(hdulist)))\n\n if len(cutouts) != len(hdulist) - 1:\n raise ValueError(\"Wrong number of cutout structures found in Content-Disposition response.\")\n\n for hdu in hdulist[1:]:\n cutout = cutouts.pop(0)\n if 'ASTLEVEL' not in hdu.header:\n print(\"WARNING: ******* NO ASTLEVEL KEYWORD ********** for {0} ********\".format(observation.get_image_uri))\n hdu.header['ASTLEVEL'] = 0\n hdu.header['EXTNO'] = cutout[0]\n naxis1 = hdu.header['NAXIS1']\n naxis2 = hdu.header['NAXIS2']\n default_datasec = \"[{}:{},{}:{}]\".format(1, naxis1, 1, naxis2)\n datasec = hdu.header.get('DATASEC', default_datasec)\n datasec = datasec_to_list(datasec)\n corners = datasec\n for idx in range(len(corners)):\n try:\n corners[idx] = int(cutout[idx+1])\n except Exception:\n pass\n\n hdu.header['DATASEC'] = reset_datasec(\"[{}:{},{}:{}]\".format(corners[0],\n corners[1],\n corners[2],\n corners[3]),\n hdu.header.get('DATASEC', default_datasec),\n hdu.header['NAXIS1'],\n hdu.header['NAXIS2'])\n hdu.header['XOFFSET'] = int(corners[0]) - 1\n hdu.header['YOFFSET'] = int(corners[2]) - 1\n hdu.converter = CoordinateConverter(hdu.header['XOFFSET'], hdu.header['YOFFSET'])\n try:\n hdu.wcs = WCS(hdu.header)\n except Exception as ex:\n logger.error(\"Failed trying to initialize the WCS for {}\".format(uri))\n raise ex\n logger.debug(\"Sending back {}\".format(hdulist))\n return hdulist", "response": "Get a cutout from an exposure based on the RA and DEC location."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef frame2expnum(frameid):\n result = {}\n parts = re.search('(?P\\d{7})(?P\\S)(?P\\d\\d)', frameid)\n assert parts is not None\n result['expnum'] = parts.group('expnum')\n result['ccd'] = parts.group('ccd')\n result['version'] = parts.group('type')\n return result", "response": "Given an OSSOS frameid return the expnum version and ccdnum as a dictionary."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting a FITS file for this exposure number and CCD.", "response": "def get_image(expnum, ccd=None, version='p', ext=FITS_EXT,\n subdir=None, prefix=None, cutout=None, return_file=True, flip_image=True):\n \"\"\"Get a FITS file for this expnum/ccd from VOSpace.\n\n\n @param cutout: (str)\n @param return_file: return an filename (True) or HDUList (False)\n @param expnum: CFHT exposure number (int)\n @param ccd: @param ccd:\n @param version: [p, s, o] (char)\n @param ext:\n @param subdir:\n @param prefix:\n @param flip_image: Should the image be x/y flipped after being retrieved?\n @return: astropy.io.fits.PrimaryHDU\n \"\"\"\n\n filename = os.path.basename(get_uri(expnum, ccd, version, ext=ext, subdir=subdir, prefix=prefix))\n if os.access(filename, os.F_OK) and return_file and cutout is None:\n return filename\n\n cutout_string = cutout\n try:\n if os.access(filename, os.F_OK) and cutout:\n cutout = datasec_to_list(cutout)\n hdulist = fits.open(filename)\n if len(hdulist) > 1:\n raise ValueError(\"Local cutout access not designed to work on MEFs yet.\")\n header = hdulist[0].header\n cutout[0] = cutout[0] < 0 and header['NAXIS1'] - cutout[0] + 1 or cutout[0]\n cutout[1] = cutout[1] < 0 and header['NAXIS1'] - cutout[1] + 1 or cutout[1]\n cutout[2] = cutout[2] < 0 and header['NAXIS2'] - cutout[2] + 1 or cutout[2]\n cutout[3] = cutout[3] < 0 and header['NAXIS2'] - cutout[3] + 1 or cutout[3]\n logger.debug(\"DATA array shape: {}\".format(hdulist[0].data.shape))\n logger.debug(\"CUTOUT array: {} {} {} {}\".format(cutout[0],\n cutout[1],\n cutout[2],\n cutout[3]))\n flip = cutout[0] < cutout[1] and 1 or -1\n flop = cutout[2] < cutout[3] and 1 or -1\n header['CRPIX1'] = (header.get(\"CRPIX1\", 1) - cutout[0]) * flip + 1\n header['CRPIX2'] = (header.get(\"CRPIX2\", 1) - cutout[2]) * flop + 1\n header['CD1_1'] = header.get(\"CD1_1\", 1) * flip\n header['CD2_1'] = header.get(\"CD2_1\", 1) * flip\n header['CD2_2'] = header.get(\"CD2_2\", 1) * flop\n header['CD1_2'] = header.get(\"CD1_2\", 1) * flop\n\n data = hdulist[0].data[cutout[2]-1:cutout[3], cutout[0]-1:cutout[1]]\n hdulist[0].data = data\n header['DATASEC'] = reset_datasec(cutout_string,\n header['DATASEC'],\n header['NAXIS1'],\n header['NAXIS2'])\n if return_file:\n cutout_filename = os.path.splitext(filename)[0]+\"_{}_{}_{}_{}.fits\".format(cutout[0],\n cutout[1],\n cutout[2],\n cutout[3])\n hdulist.writeto(cutout_filename)\n return cutout_filename\n else:\n return hdulist\n except Exception as e:\n logger.debug(str(e))\n logger.debug(\"Failed trying to access local copy: {} with cutout [{}:{}, {}:{}], using VOSpace\".format(\n filename, cutout[2]-1, cutout[3], cutout[0]-1, cutout[1]))\n\n cutout = cutout_string\n\n if not subdir:\n subdir = str(expnum)\n\n logger.debug(\"Building list of possible uri locations\")\n # # here is the list of places we will look, in order\n if version != 'p':\n locations = [(get_uri(expnum, ccd, version, ext=ext, subdir=subdir, prefix=prefix),\n cutout)]\n else:\n locations = []\n logger.debug(str(locations))\n if ccd is not None:\n try:\n for this_ext in [ext]:\n ext_no = int(ccd) + 1\n # extension 1 -> 18 + 37,36 should be flipped.\n flip_these_extensions = range(1, 19)\n flip_these_extensions.append(37)\n flip_these_extensions.append(38)\n flip = (cutout is None and \"fits\" in ext and (\n (ext_no in flip_these_extensions and flip_image) and \"[-*,-*]\" or \"[*,*]\")) or cutout\n locations.append((get_uri(expnum, version=version, ext=this_ext, subdir=subdir),\n \"[{}]{}\".format(ext_no, flip)))\n except Exception as e:\n logger.error(str(e))\n pass\n else:\n uri = get_uri(expnum, ccd, version, ext=ext, subdir=subdir, prefix=prefix)\n locations.append((uri, cutout))\n # uri = get_uri(expnum, ccd, version, ext=ext + \".fz\", subdir=subdir, prefix=prefix)\n # locations.append((uri, cutout))\n\n err = errno.EFAULT\n while len(locations) > 0:\n (uri, cutout) = locations.pop(0)\n try:\n hdu_list = get_hdu(uri, cutout)\n if return_file:\n hdu_list.writeto(filename)\n del hdu_list\n return filename\n else:\n return hdu_list\n except Exception as e:\n err = getattr(e, 'errno', errno.EAGAIN)\n logger.debug(\"{}\".format(type(e)))\n logger.debug(\"Failed to open {} cutout:{}\".format(uri, cutout))\n logger.debug(\"vos sent back error: {} code: {}\".format(str(e), getattr(e, 'errno', 0)))\n\n if err == errno.EAGAIN:\n time.sleep(5)\n return get_image(expnum, ccd=ccd, version=version, ext=ext,\n subdir=subdir, prefix=prefix, cutout=cutout, return_file=return_file, flip_image=flip_image)\n raise IOError(err, \"Failed to get image at uri: {} using {} {} {} {}.\".format(uri, expnum, version, ccd, cutout))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef reset_datasec(cutout, datasec, naxis1, naxis2):\n\n if cutout is None or cutout == \"[*,*]\":\n return datasec\n try:\n datasec = datasec_to_list(datasec)\n except:\n return datasec\n\n # check for '-*' in the cutout string and replace is naxis:1\n cutout = cutout.replace(\" \", \"\")\n cutout = cutout.replace(\"[-*,\", \"{}:1,\".format(naxis1))\n cutout = cutout.replace(\",-*]\", \",{}:1]\".format(naxis2))\n cutout = cutout.replace(\"[*,\", \"[1:{},\".format(naxis1))\n cutout = cutout.replace(\",*]\", \",1:{}]\".format(naxis1))\n\n try:\n cutout = [int(x) for x in re.findall(r\"([-+]?[*\\d]+?)[:,\\]]+\", cutout)]\n except:\n logger.debug(\"Failed to processes the cutout pattern: {}\".format(cutout))\n return datasec\n\n if len(cutout) == 5:\n # cutout likely starts with extension, remove\n cutout = cutout[1:]\n\n # -ve integer offsets indicate offset from the end of array.\n for idx in [0, 1]:\n if cutout[idx] < 0:\n cutout[idx] = naxis1 - cutout[idx] + 1\n for idx in [2, 3]:\n if cutout[idx] < 0:\n cutout[idx] = naxis2 - cutout[idx] + 1\n\n flip = cutout[0] > cutout[1]\n flop = cutout[2] > cutout[3]\n\n logger.debug(\"Working with cutout: {}\".format(cutout))\n\n if flip:\n cutout = [naxis1 - cutout[0] + 1, naxis1 - cutout[1] + 1, cutout[2], cutout[3]]\n datasec = [naxis1 - datasec[1] + 1, naxis1 - datasec[0] + 1, datasec[2], datasec[3]]\n\n if flop:\n cutout = [cutout[0], cutout[1], naxis2 - cutout[2] + 1, naxis2 - cutout[3] + 1]\n datasec = [datasec[0], datasec[1], naxis2 - datasec[3] + 1, naxis2 - datasec[2] + 1]\n\n datasec = [max(datasec[0] - cutout[0] + 1, 1),\n min(datasec[1] - cutout[0] + 1, naxis1),\n max(datasec[2] - cutout[2] + 1, 1),\n min(datasec[3] - cutout[2] + 1, naxis2)]\n\n return \"[{}:{},{}:{}]\".format(datasec[0], datasec[1], datasec[2], datasec[3])", "response": "reset the datasec to account for a possible cutout"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_hdu(uri, cutout=None):\n try:\n # the filename is based on the Simple FITS images file.\n filename = os.path.basename(uri)\n if os.access(filename, os.F_OK) and cutout is None:\n logger.debug(\"File already on disk: {}\".format(filename))\n hdu_list = fits.open(filename, scale_back=True)\n hdu_list.verify('silentfix+ignore')\n\n else:\n logger.debug(\"Pulling: {}{} from VOSpace\".format(uri, cutout))\n fpt = tempfile.NamedTemporaryFile(suffix='.fits')\n cutout = cutout is not None and cutout or \"\"\n copy(uri+cutout, fpt.name)\n fpt.seek(0, 2)\n fpt.seek(0)\n logger.debug(\"Read from vospace completed. Building fits object.\")\n hdu_list = fits.open(fpt, scale_back=False)\n hdu_list.verify('silentfix+ignore')\n\n logger.debug(\"Got image from vospace\")\n try:\n hdu_list[0].header['DATASEC'] = reset_datasec(cutout, hdu_list[0].header['DATASEC'],\n hdu_list[0].header['NAXIS1'],\n hdu_list[0].header['NAXIS2'])\n except Exception as e:\n logging.debug(\"error converting datasec: {}\".format(str(e)))\n\n for hdu in hdu_list:\n logging.debug(\"Adding converter to {}\".format(hdu))\n hdu.converter = CoordinateConverter(0, 0)\n try:\n hdu.wcs = WCS(hdu.header)\n except Exception as ex:\n logger.error(\"Failed trying to initialize the WCS: {}\".format(ex))\n except Exception as ex:\n raise ex\n return hdu_list", "response": "Get a HDU from the given URI from the VOSpace."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_fwhm_tag(expnum, ccd, prefix=None, version='p'):\n uri = get_uri(expnum, ccd, version, ext='fwhm', prefix=prefix)\n if uri not in fwhm:\n key = \"fwhm_{:1s}{:02d}\".format(version, int(ccd))\n fwhm[uri] = get_tag(expnum, key)\n return fwhm[uri]", "response": "Get the FWHM from the VOSpace annotation."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_fwhm(expnum, ccd, prefix=None, version='p'):\n\n uri = get_uri(expnum, ccd, version, ext='fwhm', prefix=prefix)\n filename = os.path.basename(uri)\n\n try:\n return fwhm[uri]\n except:\n pass\n\n try:\n fwhm[uri] = float(open(filename, 'r').read())\n return fwhm[uri]\n except:\n pass\n\n try:\n fwhm[uri] = float(open_vos_or_local(uri).read())\n return fwhm[uri]\n except Exception as ex:\n logger.error(str(ex))\n fwhm[uri] = 4.0\n return fwhm[uri]", "response": "Get the FWHM computed for the given expnum and ccd."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nretrieve the zeropoint stored in the tags associated with this image.", "response": "def _get_zeropoint(expnum, ccd, prefix=None, version='p'):\n \"\"\"\n Retrieve the zeropoint stored in the tags associated with this image.\n @param expnum: Exposure number\n @param ccd: ccd of the exposure\n @param prefix: possible prefix (such as 'fk')\n @param version: which version: p, s, or o ?\n @return: zeropoint\n \"\"\"\n if prefix is not None:\n DeprecationWarning(\"Prefix is no longer used here as the 'fk' and 's' have the same zeropoint.\")\n\n key = \"zeropoint_{:1s}{:02d}\".format(version, int(ccd))\n return get_tag(expnum, key)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_zeropoint(expnum, ccd, prefix=None, version='p'):\n uri = get_uri(expnum, ccd, version, ext='zeropoint.used', prefix=prefix)\n try:\n return zmag[uri]\n except:\n pass\n\n try:\n zmag[uri] = float(open_vos_or_local(uri).read())\n return zmag[uri]\n except:\n pass\n\n zmag[uri] = 0.0\n return zmag[uri]", "response": "Get the zeropoint for this exposure using the zeropoint. used file created during source planting.."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef mkdir(dirname):\n dir_list = []\n\n while not client.isdir(dirname):\n dir_list.append(dirname)\n dirname = os.path.dirname(dirname)\n while len(dir_list) > 0:\n logging.info(\"Creating directory: %s\" % (dir_list[-1]))\n try:\n client.mkdir(dir_list.pop())\n except IOError as e:\n if e.errno == errno.EEXIST:\n pass\n else:\n raise e", "response": "make directory tree in vospace.\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef vofile(filename, **kwargs):\n basename = os.path.basename(filename)\n if os.access(basename, os.R_OK):\n return open(basename, 'r')\n kwargs['view'] = kwargs.get('view', 'data')\n return client.open(filename, **kwargs)", "response": "Open and return a handle on a VOSpace data connection"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nopen a file which can either be in VOSpace or the local filesystem.", "response": "def open_vos_or_local(path, mode=\"rb\"):\n \"\"\"\n Opens a file which can either be in VOSpace or the local filesystem.\n\n @param path:\n @param mode:\n @return:\n \"\"\"\n filename = os.path.basename(path)\n if os.access(filename, os.F_OK):\n return open(filename, mode)\n if path.startswith(\"vos:\"):\n primary_mode = mode[0]\n if primary_mode == \"r\":\n vofile_mode = os.O_RDONLY\n elif primary_mode == \"w\":\n vofile_mode = os.O_WRONLY\n elif primary_mode == \"a\":\n vofile_mode = os.O_APPEND\n else:\n raise ValueError(\"Can't open with mode %s\" % mode)\n\n return vofile(path, mode=vofile_mode)\n else:\n return open(path, mode)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncopy a file from source to dest", "response": "def copy(source, dest):\n \"\"\"\n use the vospace service to get a file.\n\n @param source:\n @param dest:\n @return:\n \"\"\"\n logger.info(\"copying {} -> {}\".format(source, dest))\n\n return client.copy(source, dest)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nmake a link between two versions of a file", "response": "def vlink(s_expnum, s_ccd, s_version, s_ext,\n l_expnum, l_ccd, l_version, l_ext, s_prefix=None, l_prefix=None):\n \"\"\"make a link between two version of a file.\n\n @param s_expnum:\n @param s_ccd:\n @param s_version:\n @param s_ext:\n @param l_expnum:\n @param l_ccd:\n @param l_version:\n @param l_ext:\n @param s_prefix:\n @param l_prefix:\n @return:\n \"\"\"\n source_uri = get_uri(s_expnum, ccd=s_ccd, version=s_version, ext=s_ext, prefix=s_prefix)\n link_uri = get_uri(l_expnum, ccd=l_ccd, version=l_version, ext=l_ext, prefix=l_prefix)\n\n return client.link(source_uri, link_uri)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndelete a file, no error on does not exist @param expnum: @param ccd: @param version: @param ext: @param prefix: @return:", "response": "def delete(expnum, ccd, version, ext, prefix=None):\n \"\"\"\n delete a file, no error on does not exist\n\n @param expnum:\n @param ccd:\n @param version:\n @param ext:\n @param prefix:\n @return:\n \"\"\"\n uri = get_uri(expnum, ccd=ccd, version=version, ext=ext, prefix=prefix)\n remove(uri)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget a listing matching pattern @param pattern: @return:", "response": "def my_glob(pattern):\n \"\"\"\n get a listing matching pattern\n\n @param pattern:\n @return:\n \"\"\"\n result = []\n if pattern[0:4] == 'vos:':\n dirname = os.path.dirname(pattern)\n flist = listdir(dirname)\n for fname in flist:\n fname = '/'.join([dirname, fname])\n if fnmatch.fnmatch(fname, pattern):\n result.append(fname)\n else:\n result = glob(pattern)\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck if a node in VOSpace has the specified property.", "response": "def has_property(node_uri, property_name, ossos_base=True):\n \"\"\"\n Checks if a node in VOSpace has the specified property.\n\n @param node_uri:\n @param property_name:\n @param ossos_base:\n @return:\n \"\"\"\n if get_property(node_uri, property_name, ossos_base) is None:\n return False\n else:\n return True"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nretrieve the value associated with a property on a node in VOSpace.", "response": "def get_property(node_uri, property_name, ossos_base=True):\n \"\"\"\n Retrieves the value associated with a property on a node in VOSpace.\n\n @param node_uri:\n @param property_name:\n @param ossos_base:\n @return:\n \"\"\"\n # Must use force or we could have a cached copy of the node from before\n # properties of interest were set/updated.\n node = client.get_node(node_uri, force=True)\n property_uri = tag_uri(property_name) if ossos_base else property_name\n\n if property_uri not in node.props:\n return None\n\n return node.props[property_uri]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_property(node_uri, property_name, property_value, ossos_base=True):\n node = client.get_node(node_uri)\n property_uri = tag_uri(property_name) if ossos_base else property_name\n\n # If there is an existing value, clear it first\n if property_uri in node.props:\n node.props[property_uri] = None\n client.add_props(node)\n\n node.props[property_uri] = property_value\n client.add_props(node)", "response": "Sets the value of a property on a node in VOSpace."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef build_counter_tag(epoch_field, dry_run=False):\n logger.info(\"Epoch Field: {}, OBJECT_COUNT {}\".format(str(epoch_field), str(OBJECT_COUNT)))\n tag = epoch_field[1] + \"-\" + OBJECT_COUNT\n\n if dry_run:\n tag += \"-DRYRUN\"\n\n return tag", "response": "Builds the tag for the counter of a given epoch field."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef read_object_counter(node_uri, epoch_field, dry_run=False):\n return get_property(node_uri, build_counter_tag(epoch_field, dry_run),\n ossos_base=True)", "response": "Reads the object counter for the given epoch and field."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef increment_object_counter(node_uri, epoch_field, dry_run=False):\n current_count = read_object_counter(node_uri, epoch_field, dry_run=dry_run)\n\n if current_count is None:\n new_count = \"01\"\n else:\n new_count = coding.base36encode(coding.base36decode(current_count) + 1,\n pad_length=2)\n\n set_property(node_uri,\n build_counter_tag(epoch_field, dry_run=dry_run),\n new_count,\n ossos_base=True)\n\n return new_count", "response": "Increment the object counter for a node."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nretrieves the mopheader from cache or local file.", "response": "def get_mopheader(expnum, ccd, version='p', prefix=None):\n \"\"\"\n Retrieve the mopheader, either from cache or from vospace\n\n @param expnum:\n @param ccd:\n @param version:\n @param prefix:\n @return: Header\n \"\"\"\n prefix = prefix is None and \"\" or prefix\n mopheader_uri = dbimages_uri(expnum=expnum,\n ccd=ccd,\n version=version,\n prefix=prefix,\n ext='.mopheader')\n if mopheader_uri in mopheaders:\n return mopheaders[mopheader_uri]\n\n filename = os.path.basename(mopheader_uri)\n\n if os.access(filename, os.F_OK):\n logger.debug(\"File already on disk: {}\".format(filename))\n mopheader_fpt = StringIO(open(filename, 'r').read())\n else:\n mopheader_fpt = StringIO(open_vos_or_local(mopheader_uri).read())\n\n with warnings.catch_warnings():\n warnings.simplefilter('ignore', AstropyUserWarning)\n mopheader = fits.open(mopheader_fpt)\n\n # add some values to the mopheader so it can be an astrom header too.\n header = mopheader[0].header\n try:\n header['FWHM'] = get_fwhm(expnum, ccd)\n except IOError:\n header['FWHM'] = 10\n header['SCALE'] = mopheader[0].header['PIXSCALE']\n header['NAX1'] = header['NAXIS1']\n header['NAX2'] = header['NAXIS2']\n header['MOPversion'] = header['MOP_VER']\n header['MJD_OBS_CENTER'] = str(Time(header['MJD-OBSC'],\n format='mjd',\n scale='utc', precision=5).replicate(format='mpc'))\n header['MAXCOUNT'] = MAXCOUNT\n mopheaders[mopheader_uri] = header\n mopheader.close()\n return mopheaders[mopheader_uri]"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets the astrometric header for a given CFHT exposure number.", "response": "def _get_sghead(expnum):\n \"\"\"\n Use the data web service to retrieve the stephen's astrometric header.\n\n :param expnum: CFHT exposure number you want the header for\n :rtype : list of astropy.io.fits.Header objects.\n \"\"\"\n version = 'p'\n\n key = \"{}{}\".format(expnum, version)\n if key in sgheaders:\n return sgheaders[key]\n\n url = \"http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/pub/CFHTSG/{}{}.head\".format(expnum, version)\n logging.getLogger(\"requests\").setLevel(logging.ERROR)\n logging.debug(\"Attempting to retrieve {}\".format(url))\n resp = requests.get(url)\n if resp.status_code != 200:\n raise IOError(errno.ENOENT, \"Could not get {}\".format(url))\n\n header_str_list = re.split('END \\n', resp.content)\n\n # # make the first entry in the list a Null\n headers = [None]\n for header_str in header_str_list:\n headers.append(fits.Header.fromstring(header_str, sep='\\n'))\n logging.debug(headers[-1].get('EXTVER', -1))\n sgheaders[key] = headers\n return sgheaders[key]"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets a FITS header from the given URI.", "response": "def get_header(uri):\n \"\"\"\n Pull a FITS header from observation at the given URI\n\n @param uri: The URI of the image in VOSpace.\n \"\"\"\n if uri not in astheaders:\n astheaders[uri] = get_hdu(uri, cutout=\"[1:1,1:1]\")[0].header\n return astheaders[uri]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_astheader(expnum, ccd, version='p', prefix=None):\n logger.debug(\"Getting ast header for {}\".format(expnum))\n if version == 'p':\n try:\n # Get the SG header if possible.\n sg_key = \"{}{}\".format(expnum, version)\n if sg_key not in sgheaders:\n _get_sghead(expnum)\n if sg_key in sgheaders:\n for header in sgheaders[sg_key]:\n if header.get('EXTVER', -1) == int(ccd):\n return header\n except:\n pass\n\n try:\n ast_uri = dbimages_uri(expnum, ccd, version=version, ext='.fits')\n if ast_uri not in astheaders:\n hdulist = get_image(expnum, ccd=ccd, version=version, prefix=prefix,\n cutout=\"[1:1,1:1]\", return_file=False, ext='.fits')\n assert isinstance(hdulist, fits.HDUList)\n astheaders[ast_uri] = hdulist[0].header\n except:\n ast_uri = dbimages_uri(expnum, ccd, version=version, ext='.fits.fz')\n if ast_uri not in astheaders:\n hdulist = get_image(expnum, ccd=ccd, version=version, prefix=prefix,\n cutout=\"[1:1,1:1]\", return_file=False, ext='.fits.fz')\n assert isinstance(hdulist, fits.HDUList)\n astheaders[ast_uri] = hdulist[0].header\n return astheaders[ast_uri]", "response": "Retrieve the header for a given expnum and ccd."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the string representation of the tag used to annotate the status in VOSpace.", "response": "def tag(self):\n \"\"\"\n Get the string representation of the tag used to annotate the status in VOSpace.\n @return: str\n \"\"\"\n return \"{}{}_{}{:02d}\".format(self.target.prefix,\n self,\n self.target.version,\n self.target.ccd)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nscrambling the image files for the given set of exposures", "response": "def scramble(expnums, ccd, version='p', dry_run=False):\n \"\"\"run the plant script on this combination of exposures\"\"\"\n\n mjds = []\n fobjs = []\n for expnum in expnums:\n filename = storage.get_image(expnum, ccd=ccd, version=version)\n fobjs.append(fits.open(filename))\n # Pull out values to replace in headers.. must pull them\n # as otherwise we get pointers...\n mjds.append(fobjs[-1][0].header['MJD-OBS'])\n\n order = [0, 2, 1]\n for idx in range(len(fobjs)):\n logging.info(\"Flipping %d to %d\" % (fobjs[idx][0].header['EXPNUM'],\n expnums[order[idx]]))\n fobjs[idx][0].header['EXPNUM'] = expnums[order[idx]]\n fobjs[idx][0].header['MJD-OBS'] = mjds[order[idx]]\n uri = storage.get_uri(expnums[order[idx]],\n ccd=ccd,\n version='s',\n ext='fits')\n fname = os.path.basename(uri)\n if os.access(fname, os.F_OK):\n os.unlink(fname)\n fobjs[idx].writeto(fname)\n if dry_run:\n continue\n storage.copy(fname, uri)\n\n return"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate an OSSOS mopheader file for the specified image.", "response": "def mk_mopheader(expnum, ccd, version, dry_run=False, prefix=\"\"):\n \"\"\"Run the OSSOS mopheader script.\n\n \"\"\"\n ## confirm destination directory exists.\n destdir = os.path.dirname(\n storage.dbimages_uri(expnum, ccd, prefix=prefix, version=version, ext='fits'))\n if not dry_run:\n storage.mkdir(destdir)\n\n ## get image from the vospace storage area\n filename = storage.get_image(expnum, ccd, version=version, prefix=prefix)\n logging.info(\"Running mopheader on %s %d\" % (expnum, ccd))\n ## launch the mopheader script\n ## launch the makepsf script\n expname = os.path.basename(filename).strip('.fits')\n logging.info(util.exec_prog(['stepZjmp',\n '-f',\n expname]))\n\n mopheader_filename = expname+\".mopheader\"\n\n # mopheader_filename = mopheader.main(filename)\n\n if dry_run:\n return\n\n destination = storage.dbimages_uri(expnum, ccd, prefix=prefix, version=version, ext='mopheader')\n source = mopheader_filename\n storage.copy(source, destination)\n\n return"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef read_cands(filename):\n import sre\n \n lines=file(filename).readlines()\n exps=[]\n cands=[]\n coo=[]\n for line in lines:\n if ( line[0:2]==\"##\" ) :\n break\n exps.append(line[2:].strip())\n\n for line in lines:\n if ( line[0]==\"#\" ) :\n continue\n if len(line.strip())==0:\n if len(coo)!=0:\n cands.append(coo)\n coo=[]\n continue\n vals=line.split()\n cols=['x','y','x_0','y_0','flux','size','max_int','elon']\n values={}\n for j in range(len(cols)):\n col=cols.pop().strip()\n val=vals.pop().strip()\n values[col]=float(val)\n coo.append(values)\n cands.append(coo)\n return {'fileId': exps, 'cands': cands}", "response": "Read in the contents of a cands comb file"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndisplaying the candidates contained in a candidate record list", "response": "def discands(record):\n \"\"\"Display the candidates contained in a candidate record list\"\"\"\n import pyraf, pyfits\n\n pyraf.iraf.tv()\n display = pyraf.iraf.tv.display\n \n width=128\n \n cands = record['cands']\n exps= record['fileId']\n\n ### load some header info from the mophead file\n headers={}\n for exp in exps:\n f = pyfits.open(exp+\".fits\")\n headers[exp]={}\n for key in ['MJDATE', 'NAXIS1', 'NAXIS2', 'EXPTIME', 'FILTER']:\n headers[exp][key]=f[0].header[key]\n headers[exp]['MJD-OBSC']=headers[exp]['MJDATE']+headers[exp]['EXPTIME']/2.0/3600.0/24.0\n f.close()\n\n import math,os\n for cand in cands:\n for i in range(len(exps)):\n x2=[]\n y2=[]\n y1=[]\n x1=[]\n fileId=exps[i]\n x2.append(int(min(math.floor(cand[i]['x'])+width,headers[fileId]['NAXIS1'])))\n y2.append(int(min(math.floor(cand[i]['y'])+width,headers[fileId]['NAXIS2'])))\n x1.append(int(max(math.floor(cand[i]['x'])-width,1)))\n y1.append(int(max(math.floor(cand[i]['y'])-width,1)))\n x_1 = min(x1)\n y_1 = min(y1)\n x_2 = max(x2)\n y_2 = max(y2)\n for i in range(len(exps)):\n tvmark=open('tv.coo','w')\n xshift=cand[i]['x']-cand[i]['x_0']\n yshift=cand[i]['y']-cand[i]['y_0']\n tvmark.write('%f %f\\n' % ( cand[i]['x'], cand[i]['y']))\n x1=x_1 + xshift\n y1=y_1 + yshift\n x2=x_2 + xshift\n y2=y_2 + yshift\n cutout = \"[%d:%d,%d:%d]\" % (x1,x2,y1,y2)\n fileId=exps[i]\n display(fileId+cutout,i+1)\n tvmark.close()\n pyraf.iraf.tv.tvmark(i+1,'tv.coo',mark='circle',radii=15)\n os.unlink('tv.coo')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef query_for_observations(mjd, observable, runid_list):\n\n data = {\"QUERY\": (\"SELECT Observation.target_name as TargetName, \"\n \"COORD1(CENTROID(Plane.position_bounds)) AS RA,\"\n \"COORD2(CENTROID(Plane.position_bounds)) AS DEC, \"\n \"Plane.time_bounds_lower AS StartDate, \"\n \"Plane.time_exposure AS ExposureTime, \"\n \"Observation.instrument_name AS Instrument, \"\n \"Plane.energy_bandpassName AS Filter, \"\n \"Observation.observationID AS dataset_name, \"\n \"Observation.proposal_id AS ProposalID, \"\n \"Observation.proposal_pi AS PI \"\n \"FROM caom2.Observation AS Observation \"\n \"JOIN caom2.Plane AS Plane ON \"\n \"Observation.obsID = Plane.obsID \"\n \"WHERE ( Observation.collection = 'CFHT' ) \"\n \"AND Plane.time_bounds_lower > %d \"\n \"AND Plane.calibrationLevel=%s \"\n \"AND Observation.proposal_id IN %s \") % (mjd, observable, str(runid_list)),\n \"REQUEST\": \"doQuery\",\n \"LANG\": \"ADQL\",\n \"FORMAT\": \"votable\"}\n\n result = requests.get(storage.TAP_WEB_SERVICE, params=data, verify=False)\n assert isinstance(result, requests.Response)\n logging.debug(\"Doing TAP Query using url: %s\" % (str(result.url)))\n temp_file = tempfile.NamedTemporaryFile()\n with open(temp_file.name, 'w') as outfile:\n outfile.write(result.text)\n try:\n vot = parse(temp_file.name).get_first_table()\n except Exception as ex:\n logging.error(str(ex))\n logging.error(result.text)\n raise ex\n\n vot.array.sort(order='StartDate')\n t = vot.array\n temp_file.close()\n\n logging.debug(\"Got {} lines from tap query\".format(len(t)))\n\n return t", "response": "Query the TAP service for all observations that are part of runid and have calibration observable."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_ascii_table(observation_table, outfile):\n\n logging.info(\"writing text log to %s\" % outfile)\n\n stamp = \"#\\n# Last Updated: \" + time.asctime() + \"\\n#\\n\"\n header = \"| %20s | %20s | %20s | %20s | %20s | %20s | %20s |\\n\" % (\n \"EXPNUM\", \"OBS-DATE\", \"FIELD\", \"EXPTIME(s)\", \"RA\", \"DEC\", \"RUNID\")\n bar = \"=\" * (len(header) - 1) + \"\\n\"\n\n if outfile[0:4] == \"vos:\":\n temp_file = tempfile.NamedTemporaryFile(suffix='.txt')\n fout = temp_file\n else:\n fout = open(outfile, 'w')\n\n t2 = None\n fout.write(bar + stamp + bar + header)\n\n populated = storage.list_dbimages()\n for i in range(len(observation_table) - 1, -1, -1):\n row = observation_table.data[i]\n if row['dataset_name'] not in populated:\n storage.populate(row['dataset_name'])\n str_date = str(ephem.date(row.StartDate +\n 2400000.5 -\n ephem.julian_date(ephem.date(0))))[:20]\n t1 = time.strptime(str_date, \"%Y/%m/%d %H:%M:%S\")\n if t2 is None or math.fabs(time.mktime(t2) - time.mktime(t1)) > 3 * 3600.0:\n fout.write(bar)\n t2 = t1\n ra = str(ephem.hours(math.radians(row.RA)))\n dec = str(ephem.degrees(math.radians(row.DEC)))\n line = \"| %20s | %20s | %20s | %20.1f | %20s | %20s | %20s |\\n\" % (\n str(row.dataset_name),\n str(ephem.date(row.StartDate + 2400000.5 -\n ephem.julian_date(ephem.date(0))))[:20],\n row.TargetName[:20],\n row.ExposureTime, ra[:20], dec[:20], row.ProposalID[:20])\n fout.write(line)\n\n fout.write(bar)\n\n if outfile[0:4] == \"vos:\":\n fout.flush()\n storage.copy(fout.name, outfile)\n fout.close()\n\n return", "response": "Create an ascii log file for easy parsing."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_sky_plot(observation_table, outfile):\n\n # camera dimensions\n width = 0.98\n height = 0.98\n ax = None\n\n if outfile[0:4] == 'vos:':\n temp_file = tempfile.NamedTemporaryFile(suffix='.pdf')\n pdf = PdfPages(temp_file.name)\n else:\n pdf = PdfPages(outfile)\n\n saturn = ephem.Saturn()\n uranus = ephem.Uranus()\n\n t2 = None\n fig = None\n proposal_id = None\n limits = {'13A': (245, 200, -20, 0),\n '13B': (0, 45, 0, 20)}\n for row in reversed(observation_table.data):\n date = ephem.date(row.StartDate + 2400000.5 - ephem.julian_date(ephem.date(0)))\n str_date = str(date)\n # Saturn only a problem in 2013A fields\n saturn.compute(date)\n sra = math.degrees(saturn.ra)\n sdec = math.degrees(saturn.dec)\n uranus.compute(date)\n ura = math.degrees(uranus.ra)\n udec = math.degrees(uranus.dec)\n t1 = time.strptime(str_date, \"%Y/%m/%d %H:%M:%S\")\n if t2 is None or (math.fabs(time.mktime(t2) - time.mktime(\n t1)) > 3 * 3600.0 and opt.stack) or proposal_id is None or proposal_id != row.ProposalID:\n if fig is not None:\n pdf.savefig()\n close()\n proposal_id = row.ProposalID\n fig = figure(figsize=(7, 2))\n ax = fig.add_subplot(111, aspect='equal')\n ax.set_title(\"Data taken on %s-%s-%s\" % (t1.tm_year, t1.tm_mon, t1.tm_mday), fontdict={'fontsize': 8})\n ax.axis(limits.get(row.ProposalID[0:3], (0, 20, 0, 20))) # appropriate only for 2013A fields\n ax.grid()\n ax.set_xlabel(\"RA (deg)\", fontdict={'fontsize': 8})\n ax.set_ylabel(\"DEC (deg)\", fontdict={'fontsize': 8})\n t2 = t1\n ra = row.RA - width / 2.0\n dec = row.DEC - height / 2.0\n color = 'b'\n if 'W' in row['TargetName']:\n color = 'g'\n ax.add_artist(Rectangle(xy=(ra, dec), height=height, width=width,\n edgecolor=color, facecolor=color,\n lw=0.5, fill='g', alpha=0.33))\n ax.add_artist(Rectangle(xy=(sra, sdec), height=0.3, width=0.3,\n edgecolor='r',\n facecolor='r',\n lw=0.5, fill='k', alpha=0.33))\n ax.add_artist(Rectangle(xy=(ura, udec), height=0.3, width=0.3,\n edgecolor='b',\n facecolor='b',\n lw=0.5, fill='b', alpha=0.33))\n\n if ax is not None:\n ax.axis((270, 215, -20, 0))\n pdf.savefig()\n close()\n pdf.close()\n if outfile[0:4] == \"vos:\":\n storage.copy(pdf.name, outfile)\n pdf.close()\n\n return", "response": "Create a sky coverge plot for the given observation table."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load_observations((observations, regex, rename), path, filenames):\n\n for filename in filenames:\n if re.search(regex, filename) is None:\n logging.warning(\"Skipping {}\".format(filename))\n continue\n print os.path.join(path,filename)\n obs = mpc.MPCReader().read(os.path.join(path,filename))\n for ob in obs:\n if rename:\n new_provisional_name = os.path.basename(filename)\n new_provisional_name = new_provisional_name[0:new_provisional_name.find(\".\")]\n ob.provisional_name = new_provisional_name\n # ob.comment.name = new_provisional_name\n key1 = \"{:.5f}\".format(ob.date.mjd)\n key2 = ob.provisional_name \n if key1 not in observations:\n observations[key1] = {}\n if key2 in observations[key1]:\n if observations[key1][key2]:\n continue\n if not observation.null_observation:\n logger.error(filename)\n logger.error(line)\n logger.error(str(observations[key1][key2]))\n raise ValueError(\"conflicting observations for {} in {}\".format(key2, key1))\n observations[key1][key2] = ob", "response": "Load the MPC file and store the observations into the observations dictionary."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nwrites the header to a FITS file.", "response": "def writeto(self, filename, **kwargs):\n \"\"\"\n Write the header to a fits file.\n :param filename:\n :return:\n \"\"\"\n fits.PrimaryHDU(header=self).writeto(filename, output_verify='ignore', **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef crpix(self):\n try:\n return self.wcs.crpix1, self.wcs.crpix2\n except Exception as ex:\n logging.debug(\"Couldn't get CRPIX from WCS: {}\".format(ex))\n logging.debug(\"Switching to use DATASEC for CRPIX value computation.\")\n\n try:\n (x1, x2), (y1, y2) = util.get_pixel_bounds_from_datasec_keyword(self['DETSEC'])\n dx = float(self['NAXIS1'])\n dy = float(self['NAXIS2'])\n except KeyError as ke:\n raise KeyError(\"Header missing keyword: {}, required for CRPIX[12] computation\".format(ke.args[0]))\n\n crpix1 = self._DET_X_CEN - (x1 + x2) / 2. + dx / 2.\n crpix2 = self._DET_Y_CEN - (y1 + y2) / 2. + dy / 2.\n\n return crpix1, crpix2", "response": "Return the location of the reference coordinate in the pixel frame."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef mjd_obsc(self):\n # TODO Check if this exposure was taken afer or before correction needed.\n\n try:\n utc_end = self['UTCEND']\n exposure_time = float(self['EXPTIME'])\n date_obs = self['DATE-OBS']\n except KeyError as ke:\n raise KeyError(\"Header missing keyword: {}, required for MJD-OBSC computation\".format(ke.args[0]))\n\n utc_end = Time(date_obs + \"T\" + utc_end)\n utc_cen = utc_end - TimeDelta(0.73, format='sec') - TimeDelta(exposure_time / 2.0, format='sec')\n return round(utc_cen.mjd, 7)", "response": "Compute the MJD of the Megaprime image header."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets the world coordinate of the reference pixel.", "response": "def crval(self):\n \"\"\"\n Get the world coordinate of the reference pixel.\n\n @rtype: float, float\n \"\"\"\n try:\n return self.wcs.crval1, self.wcs.crval2\n except Exception as ex:\n logging.debug(\"Couldn't get CRVAL from WCS: {}\".format(ex))\n logging.debug(\"Trying RA/DEC values\")\n\n try:\n return (float(self['RA-DEG']),\n float(self['DEC-DEG']))\n except KeyError as ke:\n KeyError(\"Can't build CRVAL1/2 missing keyword: {}\".format(ke.args[0]))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the pixel scale of the detector in arcseconds.", "response": "def pixscale(self):\n \"\"\"Return the pixel scale of the detector, in arcseconds.\n\n This routine first attempts to compute the size of a pixel using the WCS. Then looks for PIXSCAL in header.\n\n @return: pixscal\n @rtype: float\n \"\"\"\n try:\n (x, y) = self['NAXIS1'] / 2.0, self['NAXIS2'] / 2.0\n p1 = SkyCoord(*self.wcs.xy2sky(x, y) * units.degree)\n p2 = SkyCoord(*self.wcs.xy2sky(x + 1, y + 1) * units.degree)\n return round(p1.separation(p2).to(units.arcsecond).value / math.sqrt(2), 3)\n except Exception as ex:\n logging.debug(\"Failed to compute PIXSCALE using WCS: {}\".format(ex))\n\n return float(self['PIXSCAL'])"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncomputing the magnitude of the star at location x y", "response": "def iraf_phot(f,x,y,zmag=26.5,apin=10,skyin=15,skywidth=10):\n \"\"\"Compute the magnitude of the star at location x/y\"\"\"\n import pyfits\n import re\n infits=pyfits.open(f,'update')\n f=re.sub(r'.fits$','',f)\n \n ### Get my python routines\n from pyraf import iraf\n from pyraf.irafpar import IrafParList\n\n ### keep all the parameters locally cached.\n iraf.set(uparm=\"./\")\n iraf.set(imtype=\"fits\")\n\n\n ### Load the required IRAF packages\n iraf.digiphot()\n iraf.apphot()\n iraf.daophot()\n \n ### temp file name hash.\n tfile={}\n \n iraf.datapars.datamax=60000\n iraf.datapars.datamin=-1000\n iraf.datapars.airmass='AIRMASS'\n iraf.datapars.filter='FILTER'\n iraf.datapars.obstime='TIME-OBS'\n iraf.datapars.exposure='EXPTIME'\n iraf.datapars.gain='GAIN'\n iraf.datapars.ccdread='RDNOISE'\n iraf.datapars.fwhmpsf=5.0\n \n iraf.centerpars.calgorithm='centroid'\n iraf.photpars.zmag=zmag\n iraf.photpars.apertures=apin\n iraf.fitskypars.annulus=skyin\n iraf.fitskypars.dannulus=skywidth\n iraf.daophot.verbose=iraf.no\n iraf.daophot.verify=iraf.no\n iraf.daophot.update=iraf.no\n \n iraf.psf.interactive=iraf.no\n iraf.pstselect.interactive=iraf.no\n \n iraf.datapars.saveParList()\n iraf.fitskypars.saveParList()\n iraf.centerpars.saveParList()\n iraf.findpars.saveParList()\n iraf.photpars.saveParList()\n \n tfiles = ['coo','mag']\n \n\n \n for file in tfiles:\n extname=f\n tfile[file]=extname+\".\"+file\n if ( os.access(tfile[file],os.F_OK) ):\n os.unlink(tfile[file])\n \n this_image=f\n \n fd = open(tfile['coo'],'w')\n fd.write('%f %f\\n' % ( x, y) )\n fd.close()\n \n print \"Measuring photometry psf star in \"+tfile['coo']\n iraf.daophot.phot(image=this_image,\n coords=tfile['coo'],\n output=tfile['mag'])\n\n import string\n a=iraf.txdump(tfile['mag'],\"MAG,XCEN,YCEN\",iraf.yes,Stdout=1)\n (mag,x,y)=string.split(a[0])\n\n inhdu=infits[0].header\n \n inhdu.update(\"PSFMAG\",float(mag),comment=\"PSF Magnitude\")\n inhdu.update(\"PSF_X\",float(x),comment=\"PSF Magnitude\")\n inhdu.update(\"PSF_Y\",float(y),comment=\"PSF Magnitude\")\n inhdu.update(\"ZMAG\",zmag,comment=\"ZMAG of PSF \")\n\n ## now measure using a smaller aperture to get aper correction\n iraf.photpars.apertures=apin*3.0\n os.unlink(tfile['mag'])\n iraf.daophot.phot(image=this_image,\n coords=tfile['coo'],\n output=tfile['mag'])\n a=iraf.txdump(tfile['mag'],\"MAG,XCEN,YCEN\",iraf.yes,Stdout=1)\n (magout,x,y)=string.split(a[0])\n inhdu.update(\"APCOR\",float(magout)-float(mag),comment=\"AP_OUT - AP_IN\")\n inhdu.update(\"AP_IN\",apin*3.0,comment=\"Small aperature\")\n inhdu.update(\"AP_OUT\",apin,comment=\"Large aperture\")\n\n \n # ### append this psf to the output images....\n \n infits.close()\n \n ### remove the temp file we used for this computation.\n #for tf in tfile.keys():\n # if os.access(tfile[tf],os.F_OK):\n # os.unlink(tfile[tf])\n return 0"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nuse the rates program to determine the minimum and maximum bounds for planting", "response": "def get_rates(file,au_min=25,au_max=150):\n \"\"\" Use the rates program to determine the minimum and maximum bounds for planting\"\"\"\n import os, string\n\n rate_command='rate.pl --file %s %d ' % ( file, au_min)\n rate=os.popen(rate_command)\n line=rate.readline()\n print line\n rate.close()\n (min_rate, min_ang, min_aw, min_rmin, min_rmax)=string.split(line)\n rate_command='rate.pl --file %s %d ' % ( file, au_min)\n rate=os.popen(rate_command)\n line=rate.readline()\n rate.close()\n (max_rate, max_ang, max_aw, max_rmin, max_rmax)=string.split(line)\n rmin=float(min(max_rmin, min_rmin))\n rmax=float(max(max_rmax, min_rmax))\n aw=float(max(max_aw,min_aw))\n angle=(float(max_ang)+float(min_ang))/2.0\n rates={'rmin': rmin, 'rmax': rmax, 'angle': angle, 'aw': aw}\n return rates"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef kbo_gen(file,outfile='objects.list',mmin=22.5,mmax=24.5):\n\n header=get_rates(file)\n print header\n import pyfits\n hdulist=pyfits.open(file)\n header['xmin']=1\n header['xmax']=hdulist[0].header.get('NAXIS1',2048)\n header['ymin']=1\n header['aw']=round(header['aw'],2)\n header['angle']=round(header['angle'],2)\n header['ymax']=hdulist[0].header.get('NAXIS2',4096)\n header['pixscale']=hdulist[0].header.get('PIXSCALE',0.185)\n header['rmax']=round(float(header['rmax'])/float(header['pixscale']),2)\n header['rmin']=round(float(header['rmin'])/float(header['pixscale']),2)\n header['mmin']=mmin\n header['mmax']=mmax\n header['expnum']=hdulist[0].header.get('EXPNUM',1000000)\n header['chipnum']=hdulist[0].header.get('CHIPNUM')\n \n import random\n number = 250\n cdata={'x': [], 'y': [], 'mag': [], 'pix_rate': [], 'angle': [], 'id':[]}\n order=['x','y','mag','pix_rate','angle','arc_rate','id']\n for i in range(number):\n cdata['x'].append( random.uniform(header['xmin'],header['xmax']))\n cdata['y'].append( random.uniform(header['ymin'],header['ymax']))\n cdata['pix_rate'].append( random.uniform(header['rmin'],header['rmax']))\n cdata['angle'].append(random.uniform(header['angle']-header['aw'],\n header['angle']+header['aw']))\n cdata['mag'].append(random.uniform(header['mmin'],header['mmax']))\n cdata['id'].append(i)\n\n hdu={'data': cdata, 'header': header}\n return hdu", "response": "Generate a file with object moving at a range of rates and angles"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncommanding line interface for the get_family_info function.", "response": "def main():\n \"\"\"\n Input asteroid family, filter type, and image type to query SSOIS\n \"\"\"\n\n parser = argparse.ArgumentParser(description='Run SSOIS and return the available images in a particular filter.')\n\n parser.add_argument(\"--filter\",\n action=\"store\",\n default='r',\n dest=\"filter\",\n choices=['r', 'u'],\n help=\"Passband: default is r.\")\n parser.add_argument(\"--family\", '-f',\n action=\"store\",\n default=None,\n help='List of objects to query.')\n parser.add_argument(\"--member\", '-m',\n action=\"store\",\n default=None,\n help='Member object of family to query.')\n\n args = parser.parse_args()\n\n if args.family != None and args.member == None:\n get_family_info(str(args.family), args.filter)\n elif args.family == None and args.member != None:\n get_member_info(str(args.member), args.filter)\n else:\n print \"Please input either a family or single member name\""} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nquerying the ssois table for objects of a given family.", "response": "def get_family_info(familyname, filtertype='r', imagetype='p'):\n \"\"\"\n Query the ssois table for images of objects in a given family. Then parse through for desired image type,\n filter, exposure time, and telescope instrument\n \"\"\"\n\n # establish input\n family_list = '{}/{}_family.txt'.format(_FAMILY_LISTS, familyname)\n\n if os.path.exists(family_list):\n with open(family_list) as infile:\n filestr = infile.read()\n object_list = filestr.split('\\n') # array of objects to query\n elif familyname == 'all':\n object_list = find_family.get_all_families_list()\n else:\n object_list = find_family.find_family_members(familyname)\n\n for obj in object_list[0:len(object_list) - 1]: # skip header lines\n get_member_info(obj, filtertype)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_member_info(object_name, filtertype='r', imagetype='p'):\n\n # From the given input, identify the desired filter and rename appropriately Replace this?\n if filtertype.lower().__contains__('r'):\n filtertype = 'r.MP9601' # this is the old (standard) r filter for MegaCam\n if filtertype.lower().__contains__('u'):\n filtertype = 'u.MP9301'\n\n # Define time period of image search, basically while MegaCam in operation\n search_start_date = Time('2013-01-01', scale='utc') # epoch1=2013+01+01\n search_end_date = Time('2017-01-01', scale='utc') # epoch2=2017+1+1\n\n print(\"----- Searching for images of object {}\".format(object_name))\n\n image_list = []\n expnum_list = []\n ra_list = []\n dec_list = []\n\n query = Query(object_name, search_start_date=search_start_date, search_end_date=search_end_date)\n result = query.get()\n print(result)\n try:\n objects = parse_ssois_return(query.get(), object_name, imagetype, camera_filter=filtertype)\n except IOError:\n print(\"Sleeping 30 seconds\")\n time.sleep(30)\n objects = parse_ssois_return(query.get(), object_name, imagetype, camera_filter=filtertype)\n\n print(objects)\n # Setup output, label columns\n if len(objects)>0:\n output = '{}/{}_object_images.txt'.format(_OUTPUT_DIR, object_name)\n with open(output, 'w') as outfile:\n outfile.write(\"{}\\t{}\\t{}\\t{}\\t{}\\t{}\\t{}\\t{}\\t{}\\n\".format(\n \"Object\", \"Image\", \"Exp_time\", \"RA (deg)\", \"Dec (deg)\", \"time\", \"filter\", \"RA rate (\\\"/hr)\", \"Dec rate (\\\"/hr)\"))\n\n for line in objects:\n with open(output, 'a') as outfile:\n image_list.append(object_name)\n expnum_list.append(line['Exptime'])\n t_start = Time(line['MJD'], format='mjd', scale='utc') - 1.0 * units.minute\n t_stop = t_start + 2 * units.minute\n hq = horizons.Query(object_name)\n hq.start_time = t_start\n hq.stop_time = t_stop\n hq.step_size = 1 * units.minute\n p_ra = hq.table[1]['R.A._(ICRF/J2000.0']\n p_dec = hq.table[1]['DEC_(ICRF/J2000.0']\n ra_dot = hq.table[1]['dRA*cosD']\n dec_dot = hq.table[1]['dDEC/dt']\n try:\n outfile.write(\"{}\\t{}\\t{}\\t{}\\t{}\\t{}\\t{}\\t{}\\t{}\\n\".format(\n object_name, line['Image'], line['Exptime'], p_ra, p_dec,\n Time(line['MJD'], format='mjd', scale='utc'), line['Filter'], ra_dot, dec_dot))\n except Exception as e:\n print(\"Error writing to outfile: {}\".format(e))\n\n return image_list, expnum_list, ra_list, dec_list", "response": "Query the ssois table for images of a given object and return a list of the image names and the exposure times and telescope instrument objects."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse_ssois_return(ssois_return, object_name, imagetype, camera_filter='r.MP9601',\n telescope_instrument='CFHT/MegaCam'):\n \"\"\"\n Parse through objects in ssois query and filter out images of desired filter, type, exposure time, and instrument\n \"\"\"\n\n assert camera_filter in ['r.MP9601', 'u.MP9301']\n\n ret_table = []\n good_table = 0\n\n table_reader = ascii.get_reader(Reader=ascii.Basic)\n table_reader.inconsistent_handler = _skip_missing_data\n table_reader.header.splitter.delimiter = '\\t'\n table_reader.data.splitter.delimiter = '\\t'\n table = table_reader.read(ssois_return)\n\n for row in table:\n # Excludes the OSSOS wallpaper.\n # note: 'Telescope_Insturment' is a typo in SSOIS's return format\n if not 'MegaCam' in row['Telescope_Insturment']:\n continue\n # Check if image of object exists in OSSOS observations\n if not storage.exists(storage.get_uri(row['Image'][:-1])):\n continue\n if not str(row['Image_target']).startswith('WP'):\n good_table += 1\n ret_table.append(row)\n\n if good_table > 0:\n print(\" %d images found\" % good_table)\n\n return ret_table", "response": "Parse the SSOIS query and return a list of dictionaries."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef match_mopfiles(mopfile1, mopfile2):\n pos1 = pos2 = numpy.array([])\n if len(mopfile1.data) > 0:\n X_COL = \"X_{}\".format(mopfile1.header.file_ids[0])\n Y_COL = \"Y_{}\".format(mopfile1.header.file_ids[0])\n pos1 = numpy.array([mopfile1.data[X_COL].data, mopfile1.data[Y_COL].data]).transpose()\n\n if len(mopfile2.data) > 0:\n X_COL = \"X_{}\".format(mopfile2.header.file_ids[0])\n Y_COL = \"Y_{}\".format(mopfile2.header.file_ids[0])\n pos2 = numpy.array([mopfile2.data[X_COL].data, mopfile2.data[Y_COL].data]).transpose()\n\n # match_idx is an order list. The list is in the order of the first list of positions and each entry\n # is the index of the matching position from the second list.\n match_idx1, match_idx2 = util.match_lists(pos1, pos2)\n mopfile1.data.add_column(Column(data=match_idx1.filled(-1), name=\"real\", length=len(mopfile1.data)))\n idx = 0\n for file_id in mopfile1.header.file_ids:\n idx += 1\n mopfile1.data.add_column(Column(data=[file_id]*len(mopfile1.data), name=\"ID_{}\".format(idx)))\n return mopfile1", "response": "Given two MOPFiles mopfile1 and mopfile2 provide a result file that contains\n the measured values from real detections and a flag indicating if they are real or false."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef measure_mags(measures):\n import daophot\n\n image_downloader = ImageDownloader()\n\n observations = {}\n for measure in measures:\n for reading in measure:\n if reading.obs not in observations:\n observations[reading.obs] = {'x': [],\n 'y': [],\n 'source': image_downloader.download(reading, needs_apcor=True)}\n assert isinstance(reading.obs, Observation)\n observations[reading.obs]['x'].append(reading.x)\n observations[reading.obs]['y'].append(reading.y)\n\n for observation in observations:\n source = observations[observation]['source']\n assert isinstance(source, SourceCutout)\n source.update_pixel_location(observations[observation]['x'],\n observations[observation]['y'])\n hdulist_index = source.get_hdulist_idx(observation.ccdnum)\n observations[observation]['mags'] = daophot.phot(source._hdu_on_disk(hdulist_index),\n observations[observation]['x'],\n observations[observation]['y'],\n aperture=source.apcor.aperture,\n sky=source.apcor.sky,\n swidth=source.apcor.swidth,\n apcor=source.apcor.apcor,\n zmag=source.zmag,\n maxcount=30000,\n extno=0)\n\n return observations", "response": "Given a list of readings compute the magnitudes for all sources in each reading."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nmatches the Object. planted file with the given fk_candidate_observations.", "response": "def match_planted(fk_candidate_observations, match_filename, bright_limit=BRIGHT_LIMIT, object_planted=OBJECT_PLANTED,\n minimum_bright_detections=MINIMUM_BRIGHT_DETECTIONS, bright_fraction=MINIMUM_BRIGHT_FRACTION):\n \"\"\"\n Using the fk_candidate_observations as input get the Object.planted file from VOSpace and match\n planted sources with found sources.\n\n The Object.planted list is pulled from VOSpace based on the standard file-layout and name of the\n first exposure as read from the .astrom file.\n\n :param fk_candidate_observations: name of the fk*reals.astrom file to check against Object.planted\n :param match_filename: a file that will contain a list of all planted sources and the matched found source\n @param minimum_bright_detections: if there are too few bright detections we raise an error.\n\n \"\"\"\n\n found_pos = []\n detections = fk_candidate_observations.get_sources()\n for detection in detections:\n reading = detection.get_reading(0)\n # create a list of positions, to be used later by match_lists\n found_pos.append([reading.x, reading.y])\n\n # Now get the Object.planted file, either from the local FS or from VOSpace.\n objects_planted_uri = object_planted\n if not os.access(objects_planted_uri, os.F_OK):\n objects_planted_uri = fk_candidate_observations.observations[0].get_object_planted_uri()\n lines = open(objects_planted_uri).read()\n\n # we are changing the format of the Object.planted header to be compatible with astropy.io.ascii but\n # there are some old Object.planted files out there so we do these string/replace calls to reset those.\n new_lines = lines.replace(\"pix rate\", \"pix_rate\")\n new_lines = new_lines.replace(\"\"\"''/h rate\"\"\", \"sky_rate\")\n planted_objects_table = ascii.read(new_lines, header_start=-1, data_start=0)\n planted_objects_table.meta = None\n # The match_list method expects a list that contains a position, not an x and a y vector, so we transpose.\n planted_pos = numpy.transpose([planted_objects_table['x'].data, planted_objects_table['y'].data])\n # match_idx is an order list. The list is in the order of the first list of positions and each entry\n # is the index of the matching position from the second list.\n (match_idx, match_fnd) = util.match_lists(numpy.array(planted_pos), numpy.array(found_pos))\n assert isinstance(match_idx, numpy.ma.MaskedArray)\n assert isinstance(match_fnd, numpy.ma.MaskedArray)\n false_positives_table = Table()\n # Once we've matched the two lists we'll need some new columns to store the information in.\n # these are masked columns so that object.planted entries that have no detected match are left 'blank'.\n new_columns = [MaskedColumn(name=\"measure_x\", length=len(planted_objects_table), mask=True),\n MaskedColumn(name=\"measure_y\", length=len(planted_objects_table), mask=True),\n MaskedColumn(name=\"measure_rate\", length=len(planted_objects_table), mask=True),\n MaskedColumn(name=\"measure_angle\", length=len(planted_objects_table), mask=True),\n MaskedColumn(name=\"measure_mag1\", length=len(planted_objects_table), mask=True),\n MaskedColumn(name=\"measure_merr1\", length=len(planted_objects_table), mask=True),\n MaskedColumn(name=\"measure_mag2\", length=len(planted_objects_table), mask=True),\n MaskedColumn(name=\"measure_merr2\", length=len(planted_objects_table), mask=True),\n MaskedColumn(name=\"measure_mag3\", length=len(planted_objects_table), mask=True),\n MaskedColumn(name=\"measure_merr3\", length=len(planted_objects_table), mask=True)]\n planted_objects_table.add_columns(new_columns)\n tlength = 0\n new_columns = [MaskedColumn(name=\"measure_x\", length=tlength, mask=True),\n MaskedColumn(name=\"measure_y\", length=tlength, mask=True),\n MaskedColumn(name=\"measure_rate\", length=0, mask=True),\n MaskedColumn(name=\"measure_angle\", length=0, mask=True),\n MaskedColumn(name=\"measure_mag1\", length=0, mask=True),\n MaskedColumn(name=\"measure_merr1\", length=0, mask=True),\n MaskedColumn(name=\"measure_mag2\", length=0, mask=True),\n MaskedColumn(name=\"measure_merr2\", length=0, mask=True),\n MaskedColumn(name=\"measure_mag3\", length=tlength, mask=True),\n MaskedColumn(name=\"measure_merr3\", length=tlength, mask=True)]\n false_positives_table.add_columns(new_columns)\n\n # We do some 'checks' on the Object.planted match to diagnose pipeline issues. Those checks are made using just\n # those planted sources we should have detected.\n bright = planted_objects_table['mag'] < bright_limit\n n_bright_planted = numpy.count_nonzero(planted_objects_table['mag'][bright])\n\n measures = []\n idxs = []\n for idx in range(len(match_idx)):\n # The match_idx value is False if nothing was found.\n if not match_idx.mask[idx]:\n # Each 'source' has multiple 'readings'\n measures.append(detections[match_idx[idx]].get_readings())\n idxs.append(idx)\n\n observations = measure_mags(measures)\n\n for oidx in range(len(measures)):\n idx = idxs[oidx]\n readings = measures[oidx]\n start_jd = util.Time(readings[0].obs.header['MJD_OBS_CENTER'], format='mpc', scale='utc').jd\n end_jd = util.Time(readings[-1].obs.header['MJD_OBS_CENTER'], format='mpc', scale='utc').jd\n rate = math.sqrt((readings[-1].x - readings[0].x) ** 2 + (readings[-1].y - readings[0].y) ** 2) / (\n 24 * (end_jd - start_jd))\n rate = int(rate * 100) / 100.0\n angle = math.degrees(math.atan2(readings[-1].y - readings[0].y, readings[-1].x - readings[0].x))\n angle = int(angle * 100) / 100.0\n planted_objects_table[idx]['measure_rate'] = rate\n planted_objects_table[idx]['measure_angle'] = angle\n planted_objects_table[idx]['measure_x'] = observations[readings[0].obs]['mags'][\"XCENTER\"][oidx]\n planted_objects_table[idx]['measure_y'] = observations[readings[0].obs]['mags'][\"YCENTER\"][oidx]\n for ridx in range(len(readings)):\n reading = readings[ridx]\n mags = observations[reading.obs]['mags']\n planted_objects_table[idx]['measure_mag{}'.format(ridx+1)] = mags[\"MAG\"][oidx]\n planted_objects_table[idx]['measure_merr{}'.format(ridx+1)] = mags[\"MERR\"][oidx]\n\n # for idx in range(len(match_fnd)):\n # if match_fnd.mask[idx]:\n # measures = detections[idx].get_readings()\n # false_positives_table.add_row()\n # false_positives_table[-1] = measure_mags(measures, false_positives_table[-1])\n\n # Count an object as detected if it has a measured magnitude in the first frame of the triplet.\n n_bright_found = numpy.count_nonzero(planted_objects_table['measure_mag1'][bright])\n # Also compute the offset and standard deviation of the measured magnitude from that planted ones.\n offset = numpy.mean(planted_objects_table['mag'][bright] - planted_objects_table['measure_mag1'][bright])\n try:\n offset = \"{:5.2f}\".format(offset)\n except:\n offset = \"indef\"\n\n std = numpy.std(planted_objects_table['mag'][bright] - planted_objects_table['measure_mag1'][bright])\n try:\n std = \"{:5.2f}\".format(std)\n except:\n std = \"indef\"\n\n if os.access(match_filename, os.R_OK):\n fout = open(match_filename, 'a')\n else:\n fout = open(match_filename, 'w')\n\n fout.write(\"#K {:10s} {:10s}\\n\".format(\"EXPNUM\", \"FWHM\"))\n for measure in detections[0].get_readings():\n fout.write('#V {:10s} {:10s}\\n'.format(measure.obs.header['EXPNUM'], measure.obs.header['FWHM']))\n\n fout.write(\"#K \")\n for keyword in [\"RMIN\", \"RMAX\", \"ANGLE\", \"AWIDTH\"]:\n fout.write(\"{:10s} \".format(keyword))\n fout.write(\"\\n\")\n\n fout.write(\"#V \")\n for keyword in [\"RMIN\", \"RMAX\", \"ANGLE\", \"AWIDTH\"]:\n fout.write(\"{:10s} \".format(fk_candidate_observations.sys_header[keyword]))\n fout.write(\"\\n\")\n\n fout.write(\"#K \")\n for keyword in [\"NBRIGHT\", \"NFOUND\", \"OFFSET\", \"STDEV\"]:\n fout.write(\"{:10s} \".format(keyword))\n fout.write(\"\\n\")\n fout.write(\"#V {:<10} {:<10} {:<10} {:<10}\\n\".format(n_bright_planted,\n n_bright_found,\n offset,\n std))\n\n try:\n writer = ascii.FixedWidth\n # add a hash to the start of line that will have header columns: for JMP\n fout.write(\"#\")\n fout.flush()\n ascii.write(planted_objects_table, output=fout, Writer=writer, delimiter=None)\n if len(false_positives_table) > 0:\n with open(match_filename+\".fp\", 'a') as fpout:\n fpout.write(\"#\")\n ascii.write(false_positives_table, output=fpout, Writer=writer, delimiter=None)\n except Exception as e:\n logging.error(str(e))\n raise e\n finally:\n fout.close()\n\n # Some simple checks to report a failure how we're doing.\n if n_bright_planted < minimum_bright_detections:\n raise RuntimeError(1, \"Too few bright objects planted.\")\n\n if n_bright_found / float(n_bright_planted) < bright_fraction:\n raise RuntimeError(2, \"Too few bright objects found.\")\n\n return \"{} {} {} {}\".format(n_bright_planted, n_bright_found, offset, std)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef append(self, item):\n if len(self) == 0:\n # Special case, we make this the current item\n self.index = 0\n\n self.items.append(item)", "response": "Adds a new item to the end of the collection."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating an error from a message.", "response": "def make_error(self,\n message: str,\n *,\n error: Exception = None,\n # ``error_class: Type[Exception]=None`` doesn't work on\n # Python 3.5.2, but that is exact version ran by Read the\n # Docs :( More info: http://stackoverflow.com/q/42942867\n error_class: Any = None) -> Exception:\n \"\"\"Return error instantiated from given message.\n\n :param message: Message to wrap.\n :param error: Validation error.\n :param error_class:\n Special class to wrap error message into. When omitted\n ``self.error_class`` will be used.\n \"\"\"\n if error_class is None:\n error_class = self.error_class if self.error_class else Error\n return error_class(message)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nvalidates response data and wrap it inside response factory.", "response": "def make_response(self,\n data: Any = None,\n **kwargs: Any) -> Any:\n r\"\"\"Validate response data and wrap it inside response factory.\n\n :param data: Response data. Could be ommited.\n :param \\*\\*kwargs: Keyword arguments to be passed to response factory.\n \"\"\"\n if not self._valid_request:\n logger.error('Request not validated, cannot make response')\n raise self.make_error('Request not validated before, cannot make '\n 'response')\n\n if data is None and self.response_factory is None:\n logger.error('Response data omit, but no response factory is used')\n raise self.make_error('Response data could be omitted only when '\n 'response factory is used')\n\n response_schema = getattr(self.module, 'response', None)\n if response_schema is not None:\n self._validate(data, response_schema)\n\n if self.response_factory is not None:\n return self.response_factory(\n *([data] if data is not None else []),\n **kwargs)\n return data"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef validate_request(self,\n data: Any,\n *additional: AnyMapping,\n merged_class: Type[dict] = dict) -> Any:\n r\"\"\"Validate request data against request schema from module.\n\n :param data: Request data.\n :param \\*additional:\n Additional data dicts to be merged with base request data.\n :param merged_class:\n When additional data dicts supplied method by default will return\n merged **dict** with all data, but you can customize things to\n use read-only dict or any other additional class or callable.\n \"\"\"\n request_schema = getattr(self.module, 'request', None)\n if request_schema is None:\n logger.error(\n 'Request schema should be defined',\n extra={'schema_module': self.module,\n 'schema_module_attrs': dir(self.module)})\n raise self.make_error('Request schema should be defined')\n\n # Merge base and additional data dicts, but only if additional data\n # dicts have been supplied\n if isinstance(data, dict) and additional:\n data = merged_class(self._merge_data(data, *additional))\n\n try:\n self._validate(data, request_schema)\n finally:\n self._valid_request = False\n\n self._valid_request = True\n\n processor = getattr(self.module, 'request_processor', None)\n return processor(data) if processor else data", "response": "r Validates the request data against the request schema from module."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts data to pure dict instance so that it can be used in jsonschema. validate func.", "response": "def _pure_data(self, data: Any) -> Any:\n \"\"\"\n If data is dict-like object, convert it to pure dict instance, so it\n will be possible to pass to default ``jsonschema.validate`` func.\n\n :param data: Request or response data.\n \"\"\"\n if not isinstance(data, dict) and not isinstance(data, list):\n try:\n return dict(data)\n except TypeError:\n ...\n return data"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _validate(self, data: Any, schema: AnyMapping) -> Any:\n try:\n return self.validate_func(schema, self._pure_data(data))\n except self.validation_error_class as err:\n logger.error(\n 'Schema validation error',\n exc_info=True,\n extra={'schema': schema, 'schema_module': self.module})\n if self.error_class is None:\n raise\n raise self.make_error('Validation Error', error=err) from err", "response": "Validate data against given schema."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\naccess the cadc certificate server", "response": "def getCert(certHost=vos.vos.SERVER, certfile=None,\n certQuery=\"/cred/proxyCert?daysValid=\",daysValid=2):\n \"\"\"Access the cadc certificate server\"\"\"\n\n if certfile is None:\n certfile = os.path.join(os.getenv(\"HOME\",\"/tmp\"),\".ssl/cadcproxy.pem\")\n\n dirname = os.path.dirname(certfile)\n try:\n os.makedirs(dirname)\n except OSError as e:\n if os.path.isdir(dirname):\n pass\n elif e.errno == 20 or e.errno == 17:\n sys.stderr.write(str(e)+\": %s \\n\" % dirname)\n sys.stderr.write(\"Expected %s to be a directory.\\n\" % ( dirname))\n sys.exit(e.errno)\n else:\n raise e\n \n # Example taken from voidspace.org.uk\n # create a password manager\n password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()\n\n (username, passwd) = getUserPassword(host=certHost)\n\n # Add the username and password.\n # If we knew the realm, we could use it instead of ``None``.\n top_level_url = \"http://\"+certHost\n password_mgr.add_password(None, top_level_url, username, passwd)\n \n handler = urllib2.HTTPBasicAuthHandler(password_mgr)\n \n # create \"opener\" (OpenerDirector instance)\n opener = urllib2.build_opener(handler)\n \n # Install the opener. \n urllib2.install_opener(opener)\n\n # Now all calls to urllib2.urlopen use our opener.\n url=\"http://\"+certHost+certQuery+str(daysValid)\n r= urllib2.urlopen(url)\n w= file(certfile,'w')\n while True:\n buf=r.read()\n if not buf:\n break\n w.write(buf)\n w.close()\n r.close()\n return"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets the username and password for host from. netrc filie", "response": "def getUserPassword(host='www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca'):\n \"\"\"\"Getting the username/password for host from .netrc filie \"\"\"\n if os.access(os.path.join(os.environ.get('HOME','/'),\".netrc\"),os.R_OK):\n auth=netrc.netrc().authenticators(host)\n else:\n auth=False\n if not auth:\n sys.stdout.write(\"CADC Username: \")\n username=sys.stdin.readline().strip('\\n')\n password=getpass.getpass().strip('\\n')\n else:\n username=auth[0]\n password=auth[2]\n return (username,password)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef clear_tags(my_expnum, ops_set, my_ccds, dry_run=True):\n\n for ops in ops_set:\n for fake in ops[0]:\n for my_program in ops[1]:\n for version in ops[2]:\n props = {}\n for ccd in my_ccds:\n # print my_expnum, fake, my_program, version, ccd\n key = storage.get_process_tag(fake + my_program, ccd, version)\n props[key] = None\n if not dry_run:\n storage.set_tags(my_expnum, props)\n else:\n sys.stdout.write(\" \".join(props)+\"\\n\")", "response": "Clear the tags for the given expnum and ccd set."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef to_bool(value: Any) -> bool:\n return bool(strtobool(value) if isinstance(value, str) else value)", "response": "Convert string or other Python object to boolean."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts given value to int.", "response": "def to_int(value: str, default: T = None) -> Union[int, Optional[T]]:\n \"\"\"Convert given value to int.\n\n If conversion failed, return default value without raising Exception.\n\n :param value: Value to convert to int.\n :param default: Default value to use in case of failed conversion.\n \"\"\"\n try:\n return int(value)\n except (TypeError, ValueError):\n return default"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef mk_dict(results,description):\n \n rows=[]\n for row in results:\n row_dict={}\n for idx in range(len(row)):\n col=description[idx][0]\n row_dict[col]=row[idx]\n rows.append(row_dict)\n return rows", "response": "Given a list of results and a list of description sequences return a list of dictionaries"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nqueries the orbit table for the object whose official desingation matches the specified parameter official.", "response": "def get_orbits(official='%'):\n \"\"\"Query the orbit table for the object whose official desingation\n matches parameter official. By default all entries are returned\n \"\"\"\n \n sql= \"SELECT * FROM orbits WHERE official LIKE '%s' \" % (official, ) \n cfeps.execute(sql)\n return mk_dict(cfeps.fetchall(),cfeps.description)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_astrom(official='%',provisional='%'):\n\n sql= \"SELECT m.* FROM measure m \"\n sql+=\"LEFT JOIN object o ON m.provisional LIKE o.provisional \"\n if not official:\n sql+=\"WHERE o.official IS NULL\"\n else:\n sql+=\"WHERE o.official LIKE '%s' \" % ( official, ) \n sql+=\" AND m.provisional LIKE '%s' \" % ( provisional, )\n\n cfeps.execute(sql)\n return mk_dict(cfeps.fetchall(), cfeps.description)", "response": "Query the measure table for all measurements of a particular object."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a link that connects to a getData URL", "response": "def getData(file_id,ra,dec):\n \"\"\"Create a link that connects to a getData URL\"\"\"\n DATA=\"www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca\"\n BASE=\"http://\"+DATA+\"/authProxy/getData\"\n archive=\"CFHT\"\n wcs=\"corrected\"\n import re\n groups=re.match('^(?P\\d{6}).*',file_id)\n\n if not groups:\n return None\n \n file_id=groups.group('file_id')\n file_id+=\"p\"\n #### THIS IS NOT WORKING YET....\n URL=BASE+\"?dataset_name=\"+file_id+\"&cutout=circle(\"+str(ra*57.3)+\",\"\n URL+=str(dec*57.3)+\",\"+str(5.0/60.0)+\")\"\n\n return URL"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_event(self, event):\n ev = api.Event.create(self.journal.collection, event)\n ev.save()", "response": "Create an event in the local cache"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nediting event in calendar with uid", "response": "def update_event(self, event, uid):\n \"\"\"Edit event\n\n Parameters\n ----------\n event : iCalendar file as a string\n (calendar containing one event to be updated)\n uid : uid of event to be updated\n \"\"\"\n ev_for_change = self.calendar.get(uid)\n ev_for_change.content = event\n ev_for_change.save()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndeleting event and sync calendar", "response": "def delete_event(self, uid):\n \"\"\"Delete event and sync calendar\n\n Parameters\n ----------\n uid : uid of event to be deleted\n \"\"\"\n ev_for_deletion = self.calendar.get(uid)\n ev_for_deletion.delete()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef simple_lmdb_settings(path, map_size=1e9, user_supplied_id=False):\n\n def decorator(cls):\n provider = \\\n ff.UserSpecifiedIdProvider(key='_id') \\\n if user_supplied_id else ff.UuidProvider()\n\n class Settings(ff.PersistenceSettings):\n id_provider = provider\n key_builder = ff.StringDelimitedKeyBuilder('|')\n database = ff.LmdbDatabase(\n path, key_builder=key_builder, map_size=map_size)\n\n class Model(cls, Settings):\n pass\n\n Model.__name__ = cls.__name__\n Model.__module__ = cls.__module__\n return Model\n\n return decorator", "response": "A class decorator that can be used to configure sane default LMDB persistence settings for a single model"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef simple_in_memory_settings(cls):\n\n class Settings(ff.PersistenceSettings):\n id_provider = ff.UuidProvider()\n key_builder = ff.StringDelimitedKeyBuilder()\n database = ff.InMemoryDatabase(key_builder=key_builder)\n\n class Model(cls, Settings):\n pass\n\n Model.__name__ = cls.__name__\n Model.__module__ = cls.__module__\n return Model", "response": "Decorator that returns a class that persists data in - memory."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef offset(self, index=0):\n eta = self._geometry[self.camera][index][\"ra\"]\n xi = self._geometry[self.camera][index][\"dec\"]\n ra = self.origin.ra - (eta/math.cos(self.dec.radian))*units.degree\n dec = self.origin.dec - xi * units.degree + 45 * units.arcsec\n self._coordinate = SkyCoord(ra, dec)", "response": "Offset the camera pointing to be centred on a particular CCD."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef coord(self):\n if self._coordinate is None:\n self._coordinate = SkyCoord(self.origin.ra, self.origin.dec + 45 * units.arcsec)\n return self._coordinate", "response": "The center of the camera pointing in sky coordinates"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns an array of rectangles that represent the ra dec corners of the FOV", "response": "def geometry(self):\n \"\"\"Return an array of rectangles that represent the 'ra,dec' corners of the FOV\n @rtype: list\n\n \"\"\"\n\n if self.coord is None:\n return None\n ra = self.coord.ra.degree\n dec = self.coord.dec.degree\n ccds = []\n\n for geo in self._geometry[self.camera]:\n ycen = geo[\"dec\"] + dec\n xcen = geo[\"ra\"] / math.cos(math.radians(ycen)) + ra\n try:\n dy = geo[\"ddec\"]\n dx = geo[\"dra\"] / math.cos(math.radians(ycen))\n ccds.append([xcen - dx / 2.0, ycen - dy / 2.0, xcen + dx / 2.0, ycen + dy / 2.0])\n except:\n rad = geo[\"rad\"]\n ccds.append([xcen, ycen, rad])\n return ccds"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncomputing the separation between self and ( ra dec )", "response": "def separation(self, ra, dec):\n \"\"\"Compute the separation between self and (ra,dec)\"\"\"\n if self.coord is None:\n return None\n return self.coord.separation(SkyCoord(ra, dec, unit=('degree', 'degree')))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef trim(hdu):\n datasec = re.findall(r'(\\d+)',\n hdu.header.get('DATASEC'))\n l=int(datasec[0])-1\n r=int(datasec[1])\n b=int(datasec[2])-1\n t=int(datasec[3])\n if opt.verbose:\n print \"Trimming [%d:%d,%d:%d]\" % ( l,r,b,t)\n hdu.data = hdu.data[b:t,l:r] \n hdu.header.update('DATASEC',\"[%d:%d,%d:%d]\" % (1,r-l+1,1,t-b+1), comment=\"Image was trimmed\")\n hdu.header.update('ODATASEC',\"[%d:%d,%d:%d]\" % (l+1,r,b+1,t), comment=\"previous DATASEC\")\n return", "response": "TRIM a CFHT MEGAPRIME frame using the DATASEC keyword"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef overscan(hdu):\n\n for amp in (['A','B']):\n AMPKW= 'ASEC'+amp\n BIASKW= 'BSEC'+amp\n dsec=re.findall(r'(\\d+)',\n hdu.header.get(AMPKW))\n bias=re.findall(r'(\\d+)',\n hdu.header.get(BIASKW))\n \n ## the X-boundaries set the amp section off from the bias section\n ## (ie. this is a column orriented device)\n al=int(dsec[0])-1\n ah=int(dsec[1])\n bl=int(bias[0])-1\n bh=int(bias[1])\n ### the Y directions must match or the array math fails\n ## b == Bottom\n ## t == Top\n b=max(int(bias[2]),int(dsec[2]))-1\n t=min(int(bias[3]),int(dsec[3]))\n\n bias = np.add.reduce(hdu.data[b:t,bl:bh],axis=1)/float(len(hdu.data[b:t,bl:bh][0]))\n mean = bias.mean()\n hdu.data[b:t,al:ah] -= bias[:,np.newaxis]\n\thdu.data = hdu.data.astype('int16')\n hdu.header.update(\"BIAS\",mean,comment=\"Mean bias level\")\n\tdel(bias)\n\n ### send back the mean bias level subtracted\n return mean", "response": "Overscan subtracts a CFHT MEGAPRIME frame from the current AMP section and BIAS section keywords."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef requires_lock(function):\n\n def new_lock_requiring_function(self, filename, *args, **kwargs):\n if self.owns_lock(filename):\n return function(self, filename, *args, **kwargs)\n else:\n raise RequiresLockException()\n\n return new_lock_requiring_function", "response": "Decorator to check if the user owns the required lock."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nremove all persistence - related files from the directory.", "response": "def clean(self, suffixes=None):\n \"\"\"\n Remove all persistence-related files from the directory.\n \"\"\"\n if suffixes is None:\n suffixes = [DONE_SUFFIX, LOCK_SUFFIX, PART_SUFFIX]\n\n for suffix in suffixes:\n listing = self.working_context.get_listing(suffix)\n for filename in listing:\n self.working_context.remove(filename)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef setFigForm():\n\n fig_width_pt = 245.26*2\n inches_per_pt = 1.0/72.27\n golden_mean = (math.sqrt(5.)-1.0)/2.0\n fig_width = fig_width_pt*inches_per_pt\n fig_height = fig_width*golden_mean\n fig_size = [1.5*fig_width, fig_height]\n \n params = {'backend': 'ps',\n 'axes.labelsize': 12,\n 'text.fontsize': 12,\n 'legend.fontsize': 7,\n 'xtick.labelsize': 11,\n 'ytick.labelsize': 11,\n 'text.usetex': True,\n 'font.family': 'serif',\n 'font.serif': 'Times',\n 'image.aspect': 'auto',\n 'figure.subplot.left': 0.1,\n 'figure.subplot.bottom': 0.1,\n 'figure.subplot.hspace': 0.25,\n 'figure.figsize': fig_size}\n \n rcParams.update(params)", "response": "set the rcparams to EmulateApJ columnwidth = 245. 26 pts"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\naccess the cadc certificate server.", "response": "def getCert(username, password,\n certHost=_SERVER,\n certfile=None,\n certQuery=_PROXY):\n \"\"\"Access the cadc certificate server.\"\"\"\n\n if certfile is None:\n certfile = tempfile.NamedTemporaryFile()\n\n # Add the username and password.\n # If we knew the realm, we could use it instead of ``None``.\n password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()\n top_level_url = \"http://\" + certHost\n logging.debug(top_level_url)\n password_mgr.add_password(None, top_level_url, username, password)\n handler = urllib2.HTTPBasicAuthHandler(password_mgr)\n logging.debug(str(handler))\n # create \"opener\" (OpenerDirector instance)\n opener = urllib2.build_opener(handler)\n\n # Install the opener. \n urllib2.install_opener(opener)\n\n # buuld the url that with 'GET' a certificat using user_id/password info\n url = \"http://\" + certHost + certQuery\n logging.debug(url)\n\n r = None\n try:\n r = opener.open(url)\n except urllib2.HTTPError as e:\n logging.debug(url)\n logging.debug(str(e))\n return False\n\n logging.debug(str(r))\n\n if r is not None:\n while True:\n buf = r.read()\n logging.debug(buf)\n if not buf:\n break\n certfile.write(buf)\n r.close()\n\n return certfile"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngive a certfile load a list of groups that user is a member of", "response": "def getGroupsURL(certfile, group):\n \"\"\"given a certfile load a list of groups that user is a member of\"\"\"\n\n GMS = \"https://\" + _SERVER + _GMS\n\n certfile.seek(0)\n buf = certfile.read()\n\n x509 = crypto.load_certificate(crypto.FILETYPE_PEM, buf)\n sep = \"\"\n dn = \"\"\n parts = []\n for i in x509.get_issuer().get_components():\n #print i\n if i[0] in parts:\n continue\n parts.append(i[0])\n dn = i[0] + \"=\" + i[1] + sep + dn\n sep = \",\"\n\n return GMS + \"/\" + group + \"/\" + urllib.quote(dn)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef isMember(userid, password, group):\n try:\n certfile = getCert(userid, password)\n\n group_url = getGroupsURL(certfile, group)\n logging.debug(\"group url: %s\" % ( group_url))\n\n con = httplib.HTTPSConnection(_SERVER,\n 443,\n key_file=certfile.name,\n cert_file=certfile.name,\n timeout=600)\n\n con.connect()\n con.request(\"GET\", group_url)\n resp = con.getresponse()\n if resp.status == 200:\n return True\n except Exception as e:\n logging.error(str(e))\n\n #logging.debug(str(resp.status))\n\n return False", "response": "Test to see if the given userid / password combo is an authenticated member of group."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ntransform from celestial coordinates to pixel coordinates to taking non-linear distortion into account with the World Coordinate System FITS keywords as used in MegaPipe. For the inverse operation see xy2sky. Reference material: http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/megapipe/docs/CD_PV_keywords.pdf Args: ra: float Right ascension dec: float Declination crpix1: float Tangent point x, pixels crpix2: float Tangent point y, pixels crval1: float Tangent point RA, degrees crval2: Tangent point Dec, degrees dc: 2d array Expresses the scale, the rotation and any possible skew of the image with respect to the sky. It is the inverse of the cd matrix in xy2sky. pv: 2d array nord: int order of the fit Returns: x, y: float Pixel coordinates", "response": "def sky2xypv(ra, dec, crpix1, crpix2, crval1, crval2, dc, pv, nord, maxiter=300):\n \"\"\"\n Transforms from celestial coordinates to pixel coordinates to taking\n non-linear distortion into account with the World Coordinate System\n FITS keywords as used in MegaPipe.\n\n For the inverse operation see xy2sky.\n\n Reference material:\n http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/megapipe/docs/CD_PV_keywords.pdf\n\n Args:\n ra: float\n Right ascension\n dec: float\n Declination\n crpix1: float\n Tangent point x, pixels\n crpix2: float\n Tangent point y, pixels\n crval1: float\n Tangent point RA, degrees\n crval2:\n Tangent point Dec, degrees\n dc: 2d array\n Expresses the scale, the rotation and any possible skew of the image\n with respect to the sky. It is the inverse of the cd matrix in xy2sky.\n pv: 2d array\n nord: int\n order of the fit\n\n Returns:\n x, y: float\n Pixel coordinates\n \"\"\"\n if math.fabs(ra - crval1) > 100:\n if crval1 < 180:\n ra -= 360\n else:\n ra += 360\n\n ra /= PI180\n dec /= PI180\n\n tdec = math.tan(dec)\n ra0 = crval1 / PI180\n dec0 = crval2 / PI180\n ctan = math.tan(dec0)\n ccos = math.cos(dec0)\n\n traoff = math.tan(ra - ra0)\n craoff = math.cos(ra - ra0)\n etar = (1 - ctan * craoff / tdec) / (ctan + craoff / tdec)\n xir = traoff * ccos * (1 - etar * ctan)\n xi = xir * PI180\n eta = etar * PI180\n\n if nord < 0:\n # The simple solution\n x = xi\n y = eta\n else:\n # Reverse by Newton's method\n tolerance = 0.001 / 3600\n\n # Initial guess\n x = xi\n y = eta\n iteration = 0\n\n converged = False\n while not converged:\n assert nord >= 0\n\n # Estimates\n f = pv[0][0]\n g = pv[1][0]\n\n # Derivatives\n fx = 0\n fy = 0\n gx = 0\n gy = 0\n\n if nord >= 1:\n r = math.sqrt(x ** 2 + y ** 2)\n f += pv[0][1] * x + pv[0][2] * y + pv[0][3] * r\n g += pv[1][1] * y + pv[1][2] * x + pv[1][3] * r\n fx += pv[0][1] + pv[0][3] * x / r\n fy += pv[0][2] + pv[0][3] * y / r\n gx += pv[1][2] + pv[1][3] * x / r\n gy += pv[1][1] + pv[1][3] * y / r\n\n if nord >= 2:\n x2 = x ** 2\n xy = x * y\n y2 = y ** 2\n\n f += pv[0][4] * x2 + pv[0][5] * xy + pv[0][6] * y2\n g += pv[1][4] * y2 + pv[1][5] * xy + pv[1][6] * x2\n fx += pv[0][4] * 2 * x + pv[0][5] * y\n fy += pv[0][5] * x + pv[0][6] * 2 * y\n gx += pv[1][5] * y + pv[1][6] * 2 * x\n gy += pv[1][4] * 2 * y + pv[1][5] * x\n\n if nord >= 3:\n x3 = x ** 3\n x2y = x2 * y\n xy2 = x * y2\n y3 = y ** 3\n\n f += pv[0][7] * x3 + pv[0][8] * x2y + pv[0][9] * xy2 + pv[0][10] * y3\n g += pv[1][7] * y3 + pv[1][8] * xy2 + pv[1][9] * x2y + pv[1][10] * x3\n fx += pv[0][7] * 3 * x2 + pv[0][8] * 2 * xy + pv[0][9] * y2\n fy += pv[0][8] * x2 + pv[0][9] * 2 * xy + pv[0][10] * 3 * y2\n gx += pv[0][8] * y2 + pv[1][9] * 2 * xy + pv[1][10] * 3 * x2\n gy += pv[1][7] * 3 * y2 + pv[0][8] * 2 * xy + pv[1][9] * x2\n\n f -= xi\n g -= eta\n dx = (-f * gy + g * fy) / (fx * gy - fy * gx)\n dy = (-g * fx + f * gx) / (fx * gy - fy * gx)\n x += dx\n y += dy\n\n if math.fabs(dx) < tolerance and math.fabs(dy) < tolerance:\n converged = True\n\n iteration += 1\n\n if iteration > maxiter:\n break\n\n xp = dc[0][0] * x + dc[0][1] * y\n yp = dc[1][0] * x + dc[1][1] * y\n\n x = xp + crpix1\n y = yp + crpix2\n\n return x, y"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef xy2skypv(x, y, crpix1, crpix2, crval1, crval2, cd, pv, nord):\n xp = x - crpix1\n yp = y - crpix2\n\n # IMPORTANT NOTE: 0-based indexing in Python means indexing for values\n # in cd and pv will be shifted from in the paper.\n x_deg = cd[0][0] * xp + cd[0][1] * yp\n y_deg = cd[1][0] * xp + cd[1][1] * yp\n\n if nord < 0:\n xi = x\n eta = y\n else:\n xi = pv[0][0]\n eta = pv[1][0]\n\n if nord >= 1:\n r = numpy.sqrt(x_deg ** 2 + y_deg ** 2)\n xi += pv[0][1] * x_deg + pv[0][2] * y_deg + pv[0][3] * r\n eta += pv[1][1] * y_deg + pv[1][2] * x_deg + pv[1][3] * r\n\n if nord >= 2:\n x2 = x_deg ** 2\n xy = x_deg * y_deg\n y2 = y_deg ** 2\n\n xi += pv[0][4] * x2 + pv[0][5] * xy + pv[0][6] * y2\n eta += pv[1][4] * y2 + pv[1][5] * xy + pv[1][6] * x2\n\n if nord >= 3:\n x3 = x_deg ** 3\n x2y = x2 * y_deg\n xy2 = x_deg * y2\n y3 = y_deg ** 3\n\n xi += pv[0][7] * x3 + pv[0][8] * x2y + pv[0][9] * xy2 + pv[0][10] * y3\n eta += pv[1][7] * y3 + pv[1][8] * xy2 + pv[1][9] * x2y + pv[1][10] * x3\n\n xir = xi / PI180\n etar = eta / PI180\n\n ra0 = crval1 / PI180\n dec0 = crval2 / PI180\n\n ctan = numpy.tan(dec0)\n ccos = numpy.cos(dec0)\n raoff = numpy.arctan2(xir / ccos, 1 - etar * ctan)\n ra = raoff + ra0\n dec = numpy.arctan(numpy.cos(raoff) / ((1 - (etar * ctan)) / (etar + ctan)))\n\n ra *= PI180\n ra = numpy.where(ra < 0 , ra + 360, ra)\n ra = numpy.where(ra > 360, ra - 360, ra)\n # f = numpy.int(ra < 0) + 360\n # print f\n # if ra < 0:\n # ra += 360\n # if ra >= 360:\n # ra -= 360\n\n dec *= PI180\n\n return ra * units.degree, dec * units.degree", "response": "This function takes from pixel coordinates to celestial coordinates taking a non - linear distortion into account with the World Coordinate System FITS keywords as used in MegaPipe."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nparse the PV array from an astropy. io. fits. header. Header object.", "response": "def parse_pv(header):\n \"\"\"\n Parses the PV array from an astropy FITS header.\n\n Args:\n header: astropy.io.fits.header.Header\n The header containing the PV values.\n\n Returns:\n cd: 2d array (list(list(float))\n [[PV1_0, PV1_1, ... PV1_N], [PV2_0, PV2_1, ... PV2_N]]\n Note that N depends on the order of the fit. For example, an\n order 3 fit goes up to PV?_10.\n \"\"\"\n order_fit = parse_order_fit(header)\n\n def parse_with_base(i):\n key_base = \"PV%d_\" % i\n\n pvi_x = [header[key_base + \"0\"]]\n\n def parse_range(lower, upper):\n for j in range(lower, upper + 1):\n pvi_x.append(header[key_base + str(j)])\n\n if order_fit >= 1:\n parse_range(1, 3)\n\n if order_fit >= 2:\n parse_range(4, 6)\n\n if order_fit >= 3:\n parse_range(7, 10)\n\n return pvi_x\n\n return [parse_with_base(1), parse_with_base(2)]"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef safe_unit_norm(a):\n if 1 == len(a.shape):\n n = np.linalg.norm(a)\n if n:\n return a / n\n return a\n\n norm = np.sum(np.abs(a) ** 2, axis=-1) ** (1. / 2)\n\n # Dividing by a norm of zero will cause a warning to be issued. Set those\n # values to another number. It doesn't matter what, since we'll be dividing\n # a vector of zeros by the number, and 0 / N always equals 0.\n norm[norm == 0] = -1e12\n return a / norm[:, np.newaxis]", "response": "Ensure that the vector or vectors have unit norm"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\npads an n - dimensional numpy array with zeros along the zero - th dimension with the desired length. Return the unchanged if the length of the array is greater than or equal to the desired length.", "response": "def pad(a, desiredlength):\n \"\"\"\n Pad an n-dimensional numpy array with zeros along the zero-th dimension\n so that it is the desired length. Return it unchanged if it is greater\n than or equal to the desired length\n \"\"\"\n\n if len(a) >= desiredlength:\n return a\n\n islist = isinstance(a, list)\n a = np.array(a)\n diff = desiredlength - len(a)\n shape = list(a.shape)\n shape[0] = diff\n\n padded = np.concatenate([a, np.zeros(shape, dtype=a.dtype)])\n return padded.tolist() if islist else padded"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _wpad(l, windowsize, stepsize):\n if l <= windowsize:\n return windowsize\n\n nsteps = ((l // stepsize) * stepsize)\n overlap = (windowsize - stepsize)\n if overlap:\n return nsteps + overlap\n\n diff = (l - nsteps)\n left = max(0, windowsize - diff)\n return l + left if diff else l", "response": "This function pads the array into a window of size windowsize with the given stepsize."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a new array with the elements of a that are within a given window.", "response": "def windowed(a, windowsize, stepsize=None, dopad=False):\n \"\"\"\n Parameters\n a - the input array to restructure into overlapping windows\n windowsize - the size of each window of samples\n stepsize - the number of samples to shift the window each step. If not\n specified, this defaults to windowsize\n dopad - If false (default), leftover samples are returned seperately.\n If true, the input array is padded with zeros so that all\n samples are used.\n \"\"\"\n\n if windowsize < 1:\n raise ValueError('windowsize must be greater than or equal to one')\n\n if stepsize is None:\n stepsize = windowsize\n\n if stepsize < 1:\n raise ValueError('stepsize must be greater than or equal to one')\n\n if not a.flags['C_CONTIGUOUS']:\n a = a.copy()\n\n if windowsize == 1 and stepsize == 1:\n # A windowsize and stepsize of one mean that no windowing is necessary.\n # Return the array unchanged.\n return np.zeros((0,) + a.shape[1:], dtype=a.dtype), a\n\n if windowsize == 1 and stepsize > 1:\n return np.zeros(0, dtype=a.dtype), a[::stepsize]\n\n # the original length of the input array\n l = a.shape[0]\n\n if dopad:\n p = _wpad(l, windowsize, stepsize)\n # pad the array with enough zeros so that there are no leftover samples\n a = pad(a, p)\n # no leftovers; an empty array\n leftover = np.zeros((0,) + a.shape[1:], dtype=a.dtype)\n else:\n # cut the array so that any leftover samples are returned seperately\n c, lc = _wcut(l, windowsize, stepsize)\n leftover = a[lc:]\n a = a[:c]\n\n if 0 == a.shape[0]:\n return leftover, np.zeros(a.shape, dtype=a.dtype)\n\n n = 1 + (a.shape[0] - windowsize) // (stepsize)\n s = a.strides[0]\n newshape = (n, windowsize) + a.shape[1:]\n newstrides = (stepsize * s, s) + a.strides[1:]\n\n out = np.ndarray.__new__( \\\n np.ndarray,\n strides=newstrides,\n shape=newshape,\n buffer=a,\n dtype=a.dtype)\n\n return leftover, out"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nappends a single item to the array growing the wrapped numpy array if necessary", "response": "def append(self, item):\n \"\"\"\n append a single item to the array, growing the wrapped numpy array\n if necessary\n \"\"\"\n try:\n self._data[self._position] = item\n except IndexError:\n self._grow()\n self._data[self._position] = item\n self._position += 1\n return self"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef extend(self, items):\n items = np.array(items)\n pos = items.shape[0] + self.logical_size\n if pos > self.physical_size:\n # more physical memory is needed than has been allocated so far\n amt = self._tmp_size()\n if self.physical_size + amt < pos:\n # growing the memory by the prescribed amount still won't \n # make enough room. Allocate exactly as much memory as is needed\n # to accommodate items\n amt = pos - self.physical_size\n\n # actually grow the array\n self._grow(amt=amt)\n\n stop = self._position + items.shape[0]\n self._data[self._position: stop] = items\n self._position += items.shape[0]\n return self", "response": "extend the numpy array with multiple items growing the wrapped array if necessary\n "} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndisplays the source cutout into the display list.", "response": "def display(self, cutout, use_pixel_coords=False):\n \"\"\"\n :param cutout: source cutout object to be display\n :type cutout: source.SourceCutout\n \"\"\"\n logging.debug(\"Current display list contains: {}\".format(self._displayables_by_cutout.keys()))\n logging.debug(\"Looking for {}\".format(cutout))\n assert isinstance(cutout, SourceCutout)\n if cutout in self._displayables_by_cutout:\n displayable = self._displayables_by_cutout[cutout]\n else:\n displayable = self._create_displayable(cutout)\n self._displayables_by_cutout[cutout] = displayable\n self._detach_handlers(self.current_displayable)\n self.current_cutout = cutout\n self.current_displayable = displayable\n self._attach_handlers(self.current_displayable)\n self._do_render(self.current_displayable)\n self.mark_apertures(cutout, pixel=use_pixel_coords)\n self.draw_uncertainty_ellipse(cutout)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndraws an ErrEllipse with the spcified dimensions.", "response": "def draw_uncertainty_ellipse(self, cutout):\n \"\"\"\n Draws an ErrEllipse with the spcified dimensions. Only one ErrEllipse can be drawn and\n only once (not movable).\n \"\"\"\n if not self.mark_prediction:\n return\n\n try:\n colour = cutout.reading.from_input_file and 'b' or 'g'\n colour = cutout.reading.null_observation and 'c' or colour\n dash = cutout.reading.null_observation and 1 or 0\n sky_coord = cutout.reading.sky_coord\n\n\n # RA/DEC value of observation / prediction goes to X/Y using PV keywords\n predict_ra = sky_coord.ra.to(units.degree)\n predict_dec = sky_coord.dec.to(units.degree)\n x, y, extno = cutout.world2pix(predict_ra, predict_dec, usepv=True)\n\n # The X/Y pixel values are then converted to RA/DEC without PV for use with DS9.\n ra, dec = cutout.pix2world(x, y, extno, usepv=False)\n uncertainty_ellipse = cutout.reading.uncertainty_ellipse\n\n self.current_displayable.place_ellipse((ra, dec), uncertainty_ellipse, colour=colour, dash=dash)\n\n # Also mark on the image where source is localted, not just the ellipse.\n try:\n measured_ra = cutout.reading.mpc_observation.coordinate.ra.to(units.degree)\n measured_dec = cutout.reading.mpc_observation.coordinate.dec.to(units.degree)\n try:\n print \"{:5.2f} {:5.2f} || {:5.2f} {:5.2f} # {}\".format((predict_ra - measured_ra).to('arcsec'),\n (predict_dec - measured_dec).to('arcsec'),\n cutout.reading.uncertainty_ellipse.a,\n cutout.reading.uncertainty_ellipse.b,\n cutout.reading.mpc_observation.to_string())\n except Exception as ex:\n print \"Failed trying to write out the prevoiusly recorded measurement: {}\".format(ex)\n pass\n # x, y, extno = cutout.world2pix(measured_ra, measured_dec, usepv=True)\n # ra, dec = cutout.pix2world(x, y, extno, usepv=False)\n\n except Exception as ex:\n # print \"We had an error :{}\".format(ex)\n logging.debug(str(ex))\n logging.debug(\"Failed to get x/y from previous observation, using prediction.\")\n hdulist_index = cutout.get_hdulist_idx(cutout.reading.get_ccd_num())\n measured_ra, measured_dec = cutout.pix2world(cutout.pixel_x, cutout.pixel_y, hdulist_index, usepv=True)\n colour = 'm'\n\n self.current_displayable.place_marker(measured_ra, measured_dec, radius=8, colour=colour)\n except Exception as ex:\n print \"EXCEPTION while drawing the uncertainty ellipse, skipping.: {}\".format(ex)\n logger.debug(type(ex))\n logger.debug(str(ex))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef align(self, cutout, reading, source):\n if not self.current_displayable:\n return\n if not self.current_displayable.aligned:\n focus_sky_coord = reading.reference_sky_coord\n #focus_calculator = SingletFocusCalculator(source)\n #logger.debug(\"Got focus calculator {} for source {}\".format(focus_calculator, source))\n #focus = cutout.flip_flip(focus_calculator.calculate_focus(reading))\n #focus = cutout.get_pixel_coordinates(focus)\n #focus = cutout.pix2world(focus[0], focus[1], usepv=False)\n #focus_sky_coord = SkyCoord(focus[0], focus[1])\n self.current_displayable.pan_to(focus_sky_coord)", "response": "Aligns the display center of the given cutout with the source."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef phot(fits_filename, x_in, y_in, aperture=15, sky=20, swidth=10, apcor=0.3,\n maxcount=30000.0, exptime=1.0, zmag=None, extno=0, centroid=True):\n \"\"\"\n Compute the centroids and magnitudes of a bunch sources on fits image.\n\n :rtype : astropy.table.Table\n :param fits_filename: Name of fits image to measure source photometry on.\n :type fits_filename: str\n :param x_in: x location of source to measure\n :type x_in: float, numpy.array\n :param y_in: y location of source to measure\n :type y_in: float, numpy.array\n :param aperture: radius of circular aperture to use.\n :type aperture: float\n :param sky: radius of inner sky annulus\n :type sky: float\n :param swidth: width of the sky annulus\n :type swidth: float\n :param apcor: Aperture correction to take aperture flux to full flux.\n :type apcor: float\n :param maxcount: maximum linearity in the image.\n :type maxcount: float\n :param exptime: exposure time, relative to zmag supplied\n :type exptime: float\n :param zmag: zeropoint magnitude\n :param extno: extension of fits_filename the x/y location refers to.\n \"\"\"\n if not hasattr(x_in, '__iter__'):\n x_in = [x_in, ]\n if not hasattr(y_in, '__iter__'):\n y_in = [y_in, ]\n\n if (not os.path.exists(fits_filename) and\n not fits_filename.endswith(\".fits\")):\n # For convenience, see if we just forgot to provide the extension\n fits_filename += \".fits\"\n\n try:\n input_hdulist = fits.open(fits_filename)\n except Exception as err:\n logger.debug(str(err))\n raise TaskError(\"Failed to open input image: %s\" % err.message)\n\n # get the filter for this image\n filter_name = input_hdulist[extno].header.get('FILTER', 'DEFAULT')\n\n # Some nominal CFHT zeropoints that might be useful\n zeropoints = {\"I\": 25.77,\n \"R\": 26.07,\n \"V\": 26.07,\n \"B\": 25.92,\n \"DEFAULT\": 26.0,\n \"g.MP9401\": 32.0,\n 'r.MP9601': 31.9,\n 'gri.MP9603': 33.520}\n\n if zmag is None:\n logger.warning(\"No zmag supplied to daophot, looking for header or default values.\")\n zmag = input_hdulist[extno].header.get('PHOTZP', zeropoints[filter_name])\n logger.warning(\"Setting zmag to: {}\".format(zmag))\n # check for magic 'zeropoint.used' files\n for zpu_file in [\"{}.zeropoint.used\".format(os.path.splitext(fits_filename)[0]), \"zeropoint.used\"]:\n if os.access(zpu_file, os.R_OK):\n with open(zpu_file) as zpu_fh:\n zmag = float(zpu_fh.read())\n logger.warning(\"Using file {} to set zmag to: {}\".format(zpu_file, zmag))\n break\n\n photzp = input_hdulist[extno].header.get('PHOTZP', zeropoints.get(filter_name, zeropoints[\"DEFAULT\"]))\n if zmag != photzp:\n logger.warning((\"zmag sent to daophot: ({}) \"\n \"doesn't match PHOTZP value in image header: ({})\".format(zmag, photzp)))\n\n # setup IRAF to do the magnitude/centroid measurements\n iraf.set(uparm=\"./\")\n iraf.digiphot()\n iraf.apphot()\n iraf.daophot(_doprint=0)\n\n iraf.photpars.apertures = aperture\n iraf.photpars.zmag = zmag\n iraf.datapars.datamin = 0\n iraf.datapars.datamax = maxcount\n iraf.datapars.exposur = \"\"\n iraf.datapars.itime = exptime\n iraf.fitskypars.annulus = sky\n iraf.fitskypars.dannulus = swidth\n iraf.fitskypars.salgorithm = \"mode\"\n iraf.fitskypars.sloclip = 5.0\n iraf.fitskypars.shiclip = 5.0\n if centroid:\n iraf.centerpars.calgori = \"centroid\"\n iraf.centerpars.cbox = 5.\n iraf.centerpars.cthreshold = 0.\n iraf.centerpars.maxshift = 2.\n iraf.centerpars.clean = 'no'\n else:\n iraf.centerpars.calgori = \"none\"\n iraf.phot.update = 'no'\n iraf.phot.verbose = 'no'\n iraf.phot.verify = 'no'\n iraf.phot.interactive = 'no'\n\n # Used for passing the input coordinates\n coofile = tempfile.NamedTemporaryFile(suffix=\".coo\", delete=False)\n\n for i in range(len(x_in)):\n coofile.write(\"%f %f \\n\" % (x_in[i], y_in[i]))\n coofile.flush()\n # Used for receiving the results of the task\n # mag_fd, mag_path = tempfile.mkstemp(suffix=\".mag\")\n magfile = tempfile.NamedTemporaryFile(suffix=\".mag\", delete=False)\n\n # Close the temp files before sending to IRAF due to docstring:\n # \"Whether the name can be used to open the file a second time, while\n # the named temporary file is still open, varies across platforms\"\n coofile.close()\n magfile.close()\n os.remove(magfile.name)\n\n iraf.phot(fits_filename+\"[{}]\".format(extno), coofile.name, magfile.name)\n pdump_out = ascii.read(magfile.name, format='daophot')\n logging.debug(\"PHOT FILE:\\n\"+str(pdump_out))\n if not len(pdump_out) > 0:\n mag_content = open(magfile.name).read()\n raise TaskError(\"photometry failed. {}\".format(mag_content))\n\n # apply the aperture correction\n pdump_out['MAG'] -= apcor\n\n # if pdump_out['PIER'][0] != 0 or pdump_out['SIER'][0] != 0 or pdump_out['CIER'][0] != 0:\n # raise ValueError(\"Photometry failed:\\n {}\".format(pdump_out))\n\n # Clean up temporary files generated by IRAF\n os.remove(coofile.name)\n os.remove(magfile.name)\n logger.debug(\"Computed aperture photometry on {} objects in {}\".format(len(pdump_out), fits_filename))\n\n del input_hdulist\n return pdump_out", "response": "Compute the centroids and magnitudes of a bunch sources on a fits image."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nwrapping around phot which only returns the computed magnitude directly.", "response": "def phot_mag(*args, **kwargs):\n \"\"\"Wrapper around phot which only returns the computed magnitude directly.\"\"\"\n try:\n return phot(*args, **kwargs)\n except IndexError:\n raise TaskError(\"No photometric records returned for {0}\".format(kwargs))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a dict of all available blocks.", "response": "def all_blocks(self):\n\n status = OrderedDict.fromkeys(parameters.BLOCKS.keys())\n status['13AE'] = ['discovery complete', '50', '24.05']\n status['13AO'] = ['discovery complete', '36', '24.40']\n status['13BL'] = ['discovery complete', '79', '24.48']\n status['14BH'] = ['discovery running', '-', '-']\n status['15AP'] = ['discovery running', '-', '-']\n status['15AM'] = ['discovery running', '-', '-']\n\n '''Overview tal table is expecting:\n ID observations processing status discoveries m_r 40%\n '''\n bks = []\n for block in status.iterkeys():\n bk = [block, self.num_block_images(block)] # if set in the .fromkeys(), doesn't give a unique list\n if status[block] is not None:\n bk = bk + status[block]\n else:\n bk = bk + ['awaiting triplets', '-', '-']\n bks.append(bk)\n\n retval = {'blocks': bks, 'status': status}\n\n return retval"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef from_env(key: str, default: T = None) -> Union[str, Optional[T]]:\n return os.getenv(key, default)", "response": "Shortcut for safely reading environment var. By\n default is None."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ninitializes and return immutable Settings dictionary.", "response": "def immutable_settings(defaults: Settings,\n **optionals: Any) -> types.MappingProxyType:\n r\"\"\"Initialize and return immutable Settings dictionary.\n\n Settings dictionary allows you to setup settings values from multiple\n sources and make sure that values cannot be changed, updated by anyone else\n after initialization. This helps keep things clear and not worry about\n hidden settings change somewhere around your web application.\n\n :param defaults:\n Read settings values from module or dict-like instance.\n :param \\*\\*optionals:\n Update base settings with optional values.\n\n In common additional values shouldn't be passed, if settings values\n already populated from local settings or environment. But in case\n of using application factories this makes sense::\n\n from . import settings\n\n def create_app(**options):\n app = ...\n app.settings = immutable_settings(settings, **options)\n return app\n\n And yes each additional key overwrite default setting value.\n \"\"\"\n settings = {key: value for key, value in iter_settings(defaults)}\n for key, value in iter_settings(optionals):\n settings[key] = value\n return types.MappingProxyType(settings)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef inject_settings(mixed: Union[str, Settings],\n context: MutableMapping[str, Any],\n fail_silently: bool = False) -> None:\n \"\"\"Inject settings values to given context.\n\n :param mixed:\n Settings can be a string (that it will be read from Python path),\n Python module or dict-like instance.\n :param context:\n Context to assign settings key values. It should support dict-like item\n assingment.\n :param fail_silently:\n When enabled and reading settings from Python path ignore errors if\n given Python path couldn't be loaded.\n \"\"\"\n if isinstance(mixed, str):\n try:\n mixed = import_module(mixed)\n except Exception:\n if fail_silently:\n return\n raise\n\n for key, value in iter_settings(mixed):\n context[key] = value", "response": "Injects settings values to given context."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef iter_settings(mixed: Settings) -> Iterator[Tuple[str, Any]]:\n if isinstance(mixed, types.ModuleType):\n for attr in dir(mixed):\n if not is_setting_key(attr):\n continue\n yield (attr, getattr(mixed, attr))\n else:\n yield from filter(lambda item: is_setting_key(item[0]), mixed.items())", "response": "Iterate over settings values from settings module or dict - like instance."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef setup_timezone(timezone: str) -> None:\n if timezone and hasattr(time, 'tzset'):\n tz_root = '/usr/share/zoneinfo'\n tz_filename = os.path.join(tz_root, *(timezone.split('/')))\n\n if os.path.exists(tz_root) and not os.path.exists(tz_filename):\n raise ValueError('Incorrect timezone value: {0}'.format(timezone))\n\n os.environ['TZ'] = timezone\n time.tzset()", "response": "Shortcut helper to configure timezone for backend application."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreading through the HISTORY cards in an image header looking for detrend input lines and return the list of input lines.", "response": "def inputs(header):\n \"\"\"Read through the HISTORY cards in an image header looking for detrend\n input lines.\n\n Detrend inputs are given on lines like:\n HISTORY imcombred: file_id\n\n We require that the value in file_id be store in the CADC archive before\n adding to the inputs list.\n \"\"\"\n import string, re\n\n inputs=[]\n for h in header.ascardlist():\n if h.key==\"HISTORY\":\n g=h.value\n result=re.search('imcombred: (\\d{6}[bfopd])\\d{2} .*',g)\n if not result:\n continue\n file_id=result.group(1)\n import os\n status=os.system(\"adInfo -a CFHT -s \"+file_id)\n if status==0:\n result=re.search('(\\d{6}).*',file_id)\n if not result:\n continue\n expnum=result.group(1)\n inputs.append(expnum)\n\n if len(inputs)==0:\n ### try using the new FLIPS 2.0 keywords\n nit = header.get('IMCMB_NI',0)\n if nit==0:\n return(inputs)\n for nin in range(nit): \n kwd='IMCMB_'+str(nin).zfill(2)\n file=(header.get(kwd,''))\n result=re.search('.*(\\d{6}[bfopd]).*',g)\n if not result:\n continue\n file_id=result.group(1)\n import os\n status=os.system(\"adInfo -a CFHT -s \"+file_id)\n if status==0:\n result=re.search('(\\d{6}).*',file_id)\n if not result:\n continue\n expnum=result.group(1)\n inputs.append(expnum)\n \n return inputs"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndecodes an elixir style file name and returns a dictionary of the content.", "response": "def elixir_decode(elixir_filename):\n \"\"\"\n Takes an elixir style file name and decodes it's content.\n\n Values returned as a dictionary. Elixir filenames have the format\n RUNID.TYPE.FILTER/EXPTIME.CHIPID.VERSION.fits\n \"\"\"\n import re, pyfits\n \n parts_RE=re.compile(r'([^\\.\\s]+)')\n dataset_name = parts_RE.findall(elixir_filename)\n\n ### check that this was a valid elixir_filename\n if not dataset_name or len(dataset_name)<5 :\n raise ValueError('String %s does not parse as elixir filename'\n % elixir_filename )\n \n comments={'exptime': 'Integration time (seconds)',\n 'filter': 'Name of filter in position ',\n 'crunid': 'CFHT Q RunID',\n 'obstype': 'Observation or Exposure type',\n 'imageid': 'CCD chip number',\n 'filename': 'file name at creation of this MEF file'\n }\n \n keywords={}\n keywords['filename']=elixir_filename\n keywords['runid']=dataset_name[0]\n keywords['obstype']=dataset_name[1]\n keywords['exptime']=None\n keywords['filter']=None\n\n ### if the third part of the name is all numbers we assume exposure time\n if re.match(r'\\d+',dataset_name[2]):\n keyword['exptime']=int(dataset_name[2])\n else:\n keyword['filter']=dataset_name[2]\n \n keywords['imageid']=dataset_name[3]\n keywords['version']=dataset_name[4]\n\n header=pyfits.Header()\n for keyword in keywords.keys():\n if keywords[keyword]:\n header.update(keyword,keywords[keyword],comment=comment[keyword])\n\n return header"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate a MEF file called filename.", "response": "def create_mef(filename=None):\n \"\"\"\n Create a file an MEF fits file called filename. Generate a random\n filename if None given\n \"\"\"\n import pyfits, time\n\n if not filename:\n ### here I know what filename is to start with.\n import tempfile\n filename=tempfile.mktemp(suffix='.fits')\n else:\n import string, re\n ### filenames gotta be a string and no lead/trailing space\n filename=string.strip(str(filename))\n ### require that the filename ends in .fits\n suffix=re.match(r'^.*.fits$',filename)\n if not suffix:\n filename = filename+'.fits'\n\n ### create an HDU list\n temp = pyfits.HDUList()\n\n ### create a primary HDU\n prihdu = pyfits.PrimaryHDU()\n\n ### build the header\n h=prihdu.header\n h.update('EXTEND',pyfits.TRUE,after='NAXIS')\n h.update('NEXTEND',0,after='EXTEND')\n h.add_comment('MEF created at CADC')\n h.add_comment('Created using '+__name__+' '+__Version__)\n h.add_comment('Extensions may not be in CCD order')\n #h.update('cfh12k',__Version__,comment='split2mef software at CADC')\n h.add_comment('Use the EXTNAME keyword')\n h.add_history('Primary HDU created on '+time.asctime())\n\n ### stick the HDU onto the HDU list and write to file\n temp.append(prihdu)\n temp.writeto(filename)\n temp.close()\n\n return(filename)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef strip_pad(hdu):\n \n l = hdu.header.ascardlist()\n d = []\n for index in range(len(l)):\n if l[index].key in __comment_keys and str(l[index])==__cfht_padding:\n d.append(index)\n d.reverse()\n for index in d:\n del l[index]\n return(0)", "response": "Remove the padding lines that CFHT adds to headers"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nstacks the infiles into a single MEF file.", "response": "def stack(outfile,infiles,verbose=0):\n \"\"\"\n Stick infiles into outfiles as FITS extensions.\n\n outfile willl contain an MEF format file of the single extension FITS\n files named in the infiles array\n \"\"\"\n \n import os, sys, string, tempfile, shutil\n import pyfits, re, time\n\n### if there is a pre-existing MEF file for output then append to it\n### otherwise we need to create a PrimaryHDU\n if os.access(outfile,os.R_OK)!=1:\n if verbose:\n print \"Creating new MEF file: \",outfile\n outfile=create_mef(outfile)\n \n### get a handle for the output image, _open is the local variant of\n### pyfits.open and just does some error recovery if pyfits.open raises an\n### exception.\n out = pyfits.open(outfile,'append')\n hdr = out[0].header\n count=0\n\n### append the fits files given on the command line to the\n### output file.\n det_xmin=None\n det_xmax=None\n det_ymin=None\n det_ymax=None\n for infile in infiles:\n if verbose:\n print \"Adding \",infile,\" to \",outfile\n\n ### _open tries to handle bad fits format exceptions.\n file=_open(infile)\n if not file:\n raise IOError(\"Cann't get the HDU for \"+infile)\n\n for hdu in file:\n extname=None\n if hdu.header.has_key('EXTNAME') :\n extname=hdu.header['EXTNAME']\n elif hdu.header.has_key('EXTVER') :\n extname=\"ccd\"+string.zfill(hdu.header.has_key('EXTVER'),2)\n \n if hdu.header.has_key('EPOCH'):\n if hdu.header.has_key('EQUINOX'):\n del hdu.header['EPOCH']\n else:\n hdu.header.update('EQUINOX',hdu.header['EQUINOX'].value,\n comment=hdu.header['EQUINOX'].comment)\n\n ahdu=pyfits.ImageHDU(data=hdu.data, header=hdu.header, name=extname)\n out.append(ahdu)\n\n ### build the size of the overall detector\n if hdu.header.has_key('DETSEC'):\n values=re.findall(r'(\\d+)',\n hdu.header['DETSEC'])\n if len(values)==4:\n xmin=int(values[0])\n xmax=int(values[1])\n ymin=int(values[2])\n ymax=int(values[3])\n\n if xmin>xmax:\n t=xmin\n xmin=xmax\n xmax=t\n if ymin>ymax:\n t=ymin\n ymin=ymax\n ymax=t\n\n if xmindet_xmax or not det_xmax:\n det_xmax=xmax\n if ymindet_ymax or not det_ymax:\n det_ymax=ymax\n \n\n file.close()\n\n\n detsize='['+str(det_xmin)+':'+str(det_xmax)+','+str(det_ymin)+':'+str(det_ymax)+']'\n out[0].header.update('DETSIZE',detsize,comment='Size of Mosaic')\n out.close() \n if verbose:\n print \"Done building MEF: \",outfile\n return 0"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef\tadGet(file_id, archive=\"CFHT\", extno=None, cutout=None ):\n \n import os, string, re,urllib\n\n\n #proxy=\"http://www1.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/authProxy/getData\"\n proxy=\"http://test.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/authProxy/getData\"\n\n if file_id is None:\n return(-1)\n\n if extno is None:\n filename=file_id+\".fits\"\n else:\n filename=\"%s%s.fits\" % (file_id, string.zfill(extno,2))\n print filename\n\n if os.access(filename,os.R_OK):\n \treturn filename\n\n args={\n \"file_id\": file_id,\n \"archive\": archive\n }\n\n if extno is not None:\n args['cutout']=\"[\"+str(extno+1)+\"]\"\n else:\n args['cutout']=''\n\n if cutout is not None:\n args['cutout']=args['cutout']+cutout\n \n argline=\"\"\n sep=\"\"\n import sys\n\n ### get the directory that may contain the data\n mop_data_path=os.curdir\n if os.environ.has_key('MOP_DATA_PATH'):\n mop_data_path=os.environ['MOP_DATA_PATH']\n\n suffix=\"fits\"\n basefile=mop_data_path+\"/\"+file_id+\".fits\"\n print basefile\n if not os.access(basefile,os.R_OK):\n \targdict={}\n\targline=''\n\tsep=''\n for arg in args:\n if not args[arg]:\n continue\n\t argline+=sep+\"%s=%s\" % ( arg, args[arg])\n\t sep='&'\n\n\n url=proxy+\"?\"+argline\n command=\"curl --silent -g --fail --max-time 1800 --user jkavelaars:newone '\"+url+\"' | gunzip > \"+filename\n else:\n command=\"imcopy %s%s %s\" % ( basefile,args['cutout'],filename)\n\n\n print command\n try:\n status=os.system(command)\n except:\n sys.stderr.write(\"Failed to execute command: %s\\n\" % ( command))\n \traise TaskError, \"getData failed\"\n if status!=0:\n sys.stderr.write(\"Failed while executing command: %s\\n\" % ( command))\n \traise TaskError, \"getData failed\"\n\n return filename", "response": "Use get a fits image from the CADC."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _open(file,mode='copyonwrite'):\n import pyfits\n\n try:\n infits=pyfits.open(file,mode) \n hdu=infits\n except (ValueError,pyfits.VerifyError,pyfits.FITS_SevereError):\n import sys\n #### I really only know how to deal with one error right now.\n #if str(sys.exc_info()[1])=='mandatory keywords are not fixed format':\n hdu=_open_fix(file)\n #else:\n # print sys.exc_info()[1]\n # print \" Failed trying to repair \", file\n # raise\n \n for f in hdu:\n strip_pad(f)\n return hdu", "response": "Opens a FITS format file and returns a tuple of the first N elements."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef find_proc_date(header):\n import string, re\n for h in header.ascardlist():\n if h.key==\"HISTORY\":\n g=h.value\n if ( string.find(g,'FLIPS 1.0 -:') ):\n result=re.search('imred: FLIPS 1.0 - \\S{3} (.*) - ([\\s\\d]\\d:\\d\\d:\\d\\d)\\s*$',g)\n if result:\n date=result.group(1)\n time=result.group(2)\n datetime=date+\" \"+time\n return datetime\n return None", "response": "Search the HISTORY fields of a header looking for the FLIPS\n processing date."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef find_detrend_keyword(header, type):\n import re, string\n value='NULL'\n #print type\n for h in header:\n g = str(h)\n if ( string.find(g,'.'+type+'.')!= -1 ):\n result=re.search('[^\\s]*\\.'+type+'\\.[^\\s]*\\.\\d\\d\\.\\d\\d',g)\n if result:\n return result.group(0)", "response": "Search through header and find\n the elixir formated string that match the the\n input type."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nopens a FITS file and create a HDUList with the header keywords.", "response": "def _open_fix(file):\n \"\"\"Takes in a fits file name, open the file in binary mode and creates an HDU.\n\n Will attempt to fix some of the header keywords to match the standard FITS format.\n \"\"\"\n import pyfits, re, string\n temp = pyfits.HDUList()\n hdu = pyfits.PrimaryHDU()\n\n hdu._file=open(file,'rb')\n\n _number_RE = re.compile(\n r'(?P[+-])?0*(?P(\\.\\d+|\\d+(\\.\\d*)?)([deDE][+-]?\\d+)?)')\n\n ### here's the real difference between pyFits and cfh12kFits.\n ### I'm more flexible on the format of the header file so that allows me\n ### read more files.\n card_RE=re.compile(r\"\"\"\n (?P[-A-Z0-9_a-za ]{8}) ### keyword is the first 8 bytes... i'll allow small letters\n (\n (\n (?P=\\s) ### =\\s indicats a value coming.\n (\\s*\n ( \n (?P\\'[^\\']*[\\'/]) ### a string \n |\n (?P([+-]?(\\.\\d+|\\d+\\.\\d*)([dDEe][+-]?\\d+)?)) ### a floating point number\n |\n (?P[+-]?\\d+) ### an integer\n |\n (?P[TFtf]) ### perhaps value is boolian\n )\n \\s*\n (( / )?(?P.*))? ### value related comment.\n )\n )\n |\n (?P.*) ### strickly a comment field\n )\n \"\"\",re.VERBOSE)\n\n \n done=0\n while ( not done):\n\n ### read a line of 80 characters up to a new line from the file.\n block=hdu._file.readline(80)\n\n string_end=79\n if len(block)== 0:\n done=1\n continue\n if block[-1]=='\\n':\n string_end=len(block)-2\n\n line = re.match(r'[ -~]{0,'+str(string_end)+'}',block)\n\n line = string.ljust(line.group(0),80)[0:79]\n\n if line[0:8] == 'END ':\n done=1\n break\n\n card=card_RE.match(line)\n if not card or not card.group('KEY'):\n print card.groups()\n raise SyntaxError(\"Failed to get keyword from FITS Card %s\" % line)\n \n key=card.group('KEY')\n value=None\n if card.group('INT'):\n try:\n value=int(card.group('INT'))\n except:\n value=card.group('INT')\n elif card.group('FLOAT'):\n try:\n value=float(card.group('FLOAT'))\n except:\n value=float(card.group('FLOAT'))\n elif card.group('BOOL'):\n value=pyfits.Boolean(card.group('BOOL'))\n elif card.group('STRING'):\n value=card.group('STRING')[1:-1]\n \n if card.group('COMMENT'):\n _comment=card.group('COMMENT')\n elif card.group('C2'):\n _comment=card.group('C2')\n else:\n _comment=None\n\n try:\n if key =='COMMENT ':\n hdu.header.add_comment(_comment)\n elif key =='HISTORY ':\n hdu.header.add_history(_comment)\n elif key ==' ':\n hdu.header.add_blank(_comment)\n elif key:\n if key =='DATE-OBS' and value:\n value=string.replace(value,'/','-')\n hdu.header.update(key,value,comment=_comment)\n except:\n raise SyntaxError(\"Failed to convert line to FITS Card %s\" % line)\n\n ### set some internal variables to decided on data flow.\n hdu._bzero=hdu.header.get('BZERO',0)\n hdu._bscale=hdu.header.get('BSCALE',1)\n hdu._bitpix=hdu.header.get('BITPIX',-16)\n\n if hdu.header.get('NAXIS',0)>0:\n naxis1=hdu.header.get('NAXIS1',1)\n naxis2=hdu.header.get('NAXIS2',1)\n ### now read the data... this is a HACK from pyfits.py\n import numarray as num\n \n code = pyfits._ImageBaseHDU.NumCode[hdu._bitpix]\n dims = tuple([naxis2,naxis1])\n raw_data = num.fromfile(hdu._file,type=code,shape=dims)\n raw_data._byteorder='big'\n\n if ( hdu._bzero != 0\n or hdu._bscale!=1 ):\n if hdu._bitpix > 0 :\n hdu.data=num.array(raw_data,type=num.Float32)\n else:\n hdu.data=raw_data\n if hdu._bscale != 1:\n num.multiply(hdu.data,hdu._bscale,hdu.data)\n if hdu._bzero!=0:\n hdu.data=hdu.data + hdu._bzero\n\n del hdu.header['BSCALE']\n del hdu.header['BZERO']\n hdu.header['BITPIX']=pyfits._ImageBaseHDU.ImgCode[hdu.data.type()]\n \n temp.append(hdu)\n return temp"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef query_ssos(self, mpc_observations, lunation_count=None):\n\n # we observe ~ a week either side of new moon\n # but we don't know when in the dark run the discovery happened\n # so be generous with the search boundaries, add extra 2 weeks\n # current date just has to be the night of the triplet,\n\n if lunation_count is None:\n # Only using SSOS to find data acquired during the survey period, for now.\n search_start_date = Time('2013-02-08', scale='utc')\n search_end_date = Time(datetime.datetime.now().strftime('%Y-%m-%d'), scale='utc')\n else:\n search_start_date = Time((mpc_observations[0].date.jd * units.day - (\n self._nights_per_darkrun +\n lunation_count * self._nights_separating_darkruns)), format='jd', scale='utc')\n search_end_date = Time((mpc_observations[-1].date.jd * units.day + (\n self._nights_per_darkrun +\n lunation_count * self._nights_separating_darkruns)), format='jd', scale='utc')\n\n logger.info(\"Sending query to SSOS start_date: {} end_data: {}\\n\".format(search_start_date, search_end_date))\n query = Query(mpc_observations,\n search_start_date=search_start_date,\n search_end_date=search_end_date)\n logger.debug(\"Parsing query results...\")\n tracks_data = self.ssos_parser.parse(query.get(), mpc_observations=mpc_observations)\n\n tracks_data.mpc_observations = {}\n\n for mpc_observation in mpc_observations:\n # attach the input observations to the the SSOS query result.\n if isinstance(mpc_observation.comment, mpc.OSSOSComment):\n try:\n tracks_data.mpc_observations[mpc_observation.comment.frame.strip()] = mpc_observation\n except Exception as e:\n logger.error(str(e))\n logger.error(mpc_observation)\n\n ref_sky_coord = None\n\n min_radius = config.read('CUTOUTS.SINGLETS.RADIUS')\n if not isinstance(min_radius, units.Quantity):\n min_radius = min_radius * units.arcsec\n\n for source in tracks_data.get_sources():\n astrom_observations = tracks_data.observations\n source_readings = source.get_readings()\n foci = []\n # Loop over all the sources to determine which ones go which which focus location.\n # this is helpful to for blinking.\n for idx in range(len(source_readings)):\n source_reading = source_readings[idx]\n astrom_observation = astrom_observations[idx]\n self.orbit.predict(Time(astrom_observation.mjd, format='mjd', scale='utc'))\n assert isinstance(source_reading, SourceReading)\n if ref_sky_coord is None or source_reading.sky_coord.separation(ref_sky_coord) > min_radius * 0.8:\n foci.append([])\n ref_sky_coord = source_reading.sky_coord\n foci[-1].append(source_reading)\n for focus in foci:\n ra = numpy.zeros(len(focus))\n dec = numpy.zeros(len(focus))\n for idx in range(len(focus)):\n source_reading = focus[idx]\n ra[idx] = source_reading.sky_coord.ra.to('degree').value\n dec[idx] = source_reading.sky_coord.dec.to('degree').value\n ref_sky_coord = SkyCoord(ra.mean(), dec.mean(), unit='degree')\n for source_reading in focus:\n source_reading.reference_sky_coord = ref_sky_coord\n source_reading.pa = self.orbit.pa\n # why are these being recorded just in pixels? Because the error ellipse is drawn in pixels.\n # TODO: Modify error ellipse drawing routine to use WCS but be sure\n # that this does not cause trouble with the use of dra/ddec for cutout computer\n source_reading.dx = self.orbit.dra\n source_reading.dy = self.orbit.ddec\n frame = astrom_observation.rawname\n if frame in tracks_data.mpc_observations:\n source_reading.discovery = tracks_data.mpc_observations[frame].discovery\n\n return tracks_data", "response": "Send a query to the SSOS web service looking for available observations using the given track."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsends a query to the SSOS web service looking for available observations using the given target name.", "response": "def query_ssos(self, target_name, lunation_count=None):\n \"\"\"Send a query to the SSOS web service, looking for available observations using the given track.\n\n :param target_name: name of target to query against SSOIS db\n :param lunation_count: ignored\n :rtype: SSOSData\n \"\"\"\n\n # we observe ~ a week either side of new moon\n # but we don't know when in the dark run the discovery happened\n # so be generous with the search boundaries, add extra 2 weeks\n # current date just has to be the night of the triplet,\n from mp_ephem import horizons\n search_start_date = Time('1999-01-01', scale='utc')\n search_end_date = Time(datetime.datetime.now().strftime('%Y-%m-%d'), scale='utc')\n logger.info(\"Sending query to SSOS start_date: {} end_data: {}\\n\".format(search_start_date, search_end_date))\n query = Query(target_name,\n search_start_date=search_start_date,\n search_end_date=search_end_date)\n\n logger.debug(\"Parsing query results...\")\n tracks_data = self.ssos_parser.parse(query.get())\n\n tracks_data.mpc_observations = {}\n\n start_time = Time(search_start_date)\n stop_time = Time(search_end_date)\n step_size = 5 * units.hour\n self.orbit = horizons.Body(target_name, start_time, stop_time, step_size)\n\n ref_sky_coord = None\n\n for source in tracks_data.get_sources():\n astrom_observations = tracks_data.observations\n source_readings = source.get_readings()\n for idx in range(len(source_readings)):\n source_reading = source_readings[idx]\n assert isinstance(source_reading, SourceReading)\n if ref_sky_coord is None or source_reading.sky_coord.separation(ref_sky_coord) > 40 * units.arcsec:\n ref_sky_coord = source_reading.sky_coord\n source_reading.reference_sky_coord = ref_sky_coord\n astrom_observation = astrom_observations[idx]\n self.orbit.predict(Time(astrom_observation.mjd, format='mjd', scale='utc'))\n source_reading.pa = self.orbit.pa\n # why are these being recorded just in pixels? Because the error ellipse is drawn in pixels.\n # TODO: Modify error ellipse drawing routine to use WCS but be sure\n # that this does not cause trouble with the use of dra/ddec for cutout computer\n source_reading.dx = self.orbit.dra\n source_reading.dy = self.orbit.ddec\n logger.debug(\"Sending back set of observations that might contain the target: {}\".format(tracks_data))\n return tracks_data"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nbuild an astrom. Observation object for a SourceReading object for the given exposure number.", "response": "def build_source_reading(expnum, ccd=None, ftype='p'):\n \"\"\"\n Build an astrom.Observation object for a SourceReading\n\n :param expnum: (str) Name or CFHT Exposure number of the observation.\n :param ccd: (str) CCD is this observation associated with. (can be None)\n :param ftype: (str) exposure time (specific to CFHT imaging)\n\n :return: An astrom.Observation object for the observation.\n :rtype: astrom.Observation\n \"\"\"\n\n logger.debug(\"Building source reading for expnum:{} ccd:{} ftype:{}\".format(expnum, ccd, ftype))\n\n return astrom.Observation(expnum=str(expnum),\n ftype=ftype,\n ccdnum=ccd)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef parse(self, ssos_result_filename_or_lines, mpc_observations=None):\n table_reader = ascii.get_reader(Reader=ascii.Basic)\n table_reader.inconsistent_handler = self._skip_missing_data\n table_reader.header.splitter.delimiter = '\\t'\n table_reader.data.splitter.delimiter = '\\t'\n ssos_table = table_reader.read(ssos_result_filename_or_lines)\n\n dbimage_list = storage.list_dbimages(dbimages=storage.DBIMAGES)\n logger.debug(\"Comparing to {} observations in dbimages: {}\".format(len(dbimage_list), storage.DBIMAGES))\n sources = []\n observations = []\n source_readings = []\n\n if mpc_observations is not None and isinstance(mpc_observations[0], mpc.Observation):\n orbit = Orbfit(mpc_observations)\n else:\n from mp_ephem import horizons\n start_time = Time(min(ssos_table['MJD']), format='mjd')\n stop_time = Time(max(ssos_table['MJD']), format='mjd')\n step_size = 5.0 * units.hour\n orbit = horizons.Body(self.provisional_name, start_time, stop_time, step_size)\n\n warnings.filterwarnings('ignore')\n logger.info(\"Loading {} observations\\n\".format(len(ssos_table)))\n expnums_examined = []\n for row in ssos_table:\n # Trim down to OSSOS-specific images\n\n logger.debug(\"Checking row: {}\".format(row))\n if (row['Filter'] not in parameters.OSSOS_FILTERS) or row['Image_target'].startswith('WP'):\n logger.debug(\"Failed filter / target name check\")\n continue\n\n # check if a dbimages object exists\n # For CFHT/MegaCam strip off the trailing character to get the exposure number.\n ftype = row['Image'][-1]\n expnum = row['Image'][:-1]\n if str(expnum) not in dbimage_list:\n logger.debug(\"Expnum: {} Failed dbimage list check\".format(expnum))\n continue\n logger.debug(\"Expnum: {} Passed dbimage list check\".format(expnum))\n # The file extension is the ccd number + 1 , or the first extension.\n ccd = int(row['Ext'])-1\n if 39 < ccd < 0 or ccd < 0:\n ccd = None\n x = row['X'] * units.pix\n y = row['Y'] * units.pix\n ra = row['Object_RA'] * units.degree\n dec = row['Object_Dec'] * units.degree\n ssois_coordinate = SkyCoord(ra, dec)\n mjd = row['MJD'] * units.day\n\n # if not 0 < x.value < 2060 or not 0 < y.value < 4700:\n # continue\n\n obs_date = Time(mjd, format='mjd', scale='utc')\n logger.info(\"Calling predict\")\n orbit.predict(obs_date)\n logger.info(\"Done calling predict\")\n if orbit.dra > 4 * units.arcminute or orbit.ddec > 4.0 * units.arcminute:\n print \"Skipping entry as orbit uncertainty at date {} is large.\".format(obs_date)\n continue\n if expnum in expnums_examined:\n logger.debug(\"Already checked this exposure.\")\n continue\n expnums_examined.append(expnum)\n\n logger.debug((\"SSOIS Prediction: exposure:{} ext:{} \"\n \"ra:{} dec:{} x:{} y:{}\").format(expnum, ccd, ra, dec, x, y))\n\n logger.debug((\"Orbfit Prediction: \"\n \"ra:{} dec:{} \").format(orbit.coordinate.ra.to(units.degree),\n orbit.coordinate.dec.to(units.degree)))\n logger.info(\"Building Observation\")\n observation = SSOSParser.build_source_reading(expnum, ccd, ftype=ftype)\n observation.mjd = mjd\n from_input_file = observation.rawname in self.input_rawnames\n # compare to input observation list.\n previous = False\n mpc_observation = None\n if from_input_file:\n for mpc_observation in mpc_observations:\n try:\n if mpc_observation.comment.frame.strip() == observation.rawname:\n # only skip previous obseravtions if not discovery.\n previous = not mpc_observation.discovery\n break\n except Exception as e:\n logger.debug(str(e))\n pass\n mpc_observation = None\n\n # skip previously measured observations if requested.\n if self.skip_previous and ( previous or observation.rawname in self.null_observations):\n continue\n\n logger.info('built observation {}'.format(observation))\n observations.append(observation)\n null_observation = observation.rawname in self.null_observations\n\n ddec = orbit.ddec + abs(orbit.coordinate.dec - ssois_coordinate.dec)\n dra = orbit.dra + abs(orbit.coordinate.ra - ssois_coordinate.ra)\n\n logger.info(\" Building SourceReading .... \\n\")\n source_reading = astrom.SourceReading(x=x, y=y, x0=x, y0=y,\n ra=orbit.coordinate.ra.to(units.degree).value,\n dec=orbit.coordinate.dec.to(units.degree).value,\n xref=x, yref=y, obs=observation,\n ssos=True, from_input_file=from_input_file,\n dx=dra, dy=ddec, pa=orbit.pa,\n null_observation=null_observation)\n source_reading.mpc_observation = mpc_observation\n source_readings.append(source_reading)\n logger.info(\"Source Reading Built\")\n\n # build our array of SourceReading objects\n sources.append(source_readings)\n\n warnings.filterwarnings('once')\n\n return SSOSData(observations, sources, self.provisional_name)", "response": "Parse the result table create source objects and observations."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef search_end_date(self, search_end_date):\n assert isinstance(search_end_date, Time)\n self._search_end_date = search_end_date.replicate(format='iso')\n self._search_end_date.out_subfmt = 'date'", "response": "Search for frames take after the given date."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef params(self):\n params = dict(format=RESPONSE_FORMAT,\n verbose=self.verbose,\n epoch1=str(self.search_start_date),\n epoch2=str(self.search_end_date),\n search=self.orbit_method,\n eunits=self.error_units,\n eellipse=self.error_ellipse,\n extres=self.resolve_extension,\n xyres=self.resolve_position,\n telinst=self.telescope_instrument)\n\n if self.orbit_method == 'bynameHorizons':\n params['object'] = NEW_LINE.join((str(target_name) for target_name in self.observations))\n else:\n params['obs'] = NEW_LINE.join((str(observation) for observation in self.observations))\n return params", "response": "Returns a dictionary of SSOS query parameters."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting the TSV result from the SSOS parameter dictionary.", "response": "def get(self):\n \"\"\"\n :return: A string containing the TSV result from SSOS\n :rtype: str\n :raise: AssertionError\n \"\"\"\n params = self.param_dict_builder.params\n logger.debug(pprint.pformat(format(params)))\n response = requests.post(SSOS_URL,\n data=params,\n headers=self.headers)\n logger.debug(response.url)\n assert isinstance(response, requests.Response)\n assert (response.status_code == requests.codes.ok)\n\n lines = response.content\n # note: spelling 'occured' is in SSOIS\n if len(lines) < 2 or \"An error occured getting the ephemeris\" in lines:\n print lines\n print response.url\n raise IOError(os.errno.EACCES,\n \"call to SSOIS failed on format error\")\n\n if os.access(\"backdoor.tsv\", os.R_OK):\n lines += open(\"backdoor.tsv\").read()\n return lines"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef SetFocus(self, **kwargs):\n if self._focus is not None:\n self._focus.SetFocus()\n else:\n # fall back on the default behaviour\n super(_FocusablePanel, self).SetFocus()", "response": "Over - rides normal behaviour of shifting focus to any child."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef step3(expnums, ccd, version, rate_min,\n rate_max, angle, width, field=None, prefix=None, dry_run=False,\n maximum_flux_ratio=3, minimum_area=5, minimum_median_flux=1000.0):\n \"\"\"run the actual step3 on the given exp/ccd combo\"\"\"\n\n jmp_args = ['step3jmp']\n matt_args = ['step3jjk']\n\n idx = 0\n cmd_args = []\n for expnum in expnums:\n idx += 1\n for ext in ['unid.jmp', 'unid.matt']:\n storage.get_file(expnum, ccd=ccd, version=version, ext=ext, prefix=prefix)\n image = os.path.splitext(os.path.basename(storage.get_uri(expnum, ccd, version=version, prefix=prefix)))[0]\n cmd_args.append('-f%d' % idx)\n cmd_args.append(image)\n\n cmd_args.extend(['-rn', str(rate_min),\n '-rx', str(rate_max),\n '-a', str(angle),\n '-w', str(width)])\n jmp_args.extend(cmd_args)\n\n # Add some extra arguemnents for the ISO search.\n cmd_args.extend(['-fr', str(maximum_flux_ratio),\n '-ma', str(minimum_area),\n '-mf', str(minimum_median_flux)])\n matt_args.extend(cmd_args)\n\n logging.info(util.exec_prog(jmp_args))\n logging.info(util.exec_prog(matt_args))\n\n if dry_run:\n return\n\n if field is None:\n field = str(expnums[0])\n storage.mkdir(os.path.dirname(storage.get_uri(field,\n ccd=ccd,\n version=version,\n prefix=prefix)))\n\n for ext in ['moving.jmp', 'moving.matt']:\n uri = storage.get_uri(field,\n ccd=ccd,\n version=version,\n ext=ext,\n prefix=prefix)\n filename = '%s%d%s%s.%s' % (prefix, expnums[0],\n version,\n str(ccd).zfill(2),\n ext)\n storage.copy(filename, uri)\n\n return", "response": "run the actual step3 of the ISO search"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngives an instance extract the arguments that can be reconstructed by deserializing it", "response": "def extract_init_args(instance):\n \"\"\"\n Given an instance, and under the assumption that member variables have the\n same name as the __init__ arguments, extract the arguments so they can\n be used to reconstruct the instance when deserializing\n \"\"\"\n cls = instance.__class__\n args = [x for x in inspect.getargspec(cls.__init__).args if x != 'self']\n return [instance.__dict__[key] for key in args]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef list(self,header,choices):\n \"Display list of choices. As many as we can get in a page.\"\n\n if not self.__list_window:\n (y,x)=self.__main.getmaxyx()\n self.__list_window = self.__main.subwin(35,x,0,0)\n \n _lw=self.__list_window\n\n\n _lw.keypad(1)\n (y_max,x_max)=_lw.getmaxyx()\n (y_0, x_0)=_lw.getbegyx()\n x_start=1+x_0\n _lw.box()\n ## Number of list items allowed.\n ### first entry in the list appears at page_top\n page_top=y_0+2\n ### the last entry display will be at page_bottom\n page_bottom = y_max-2\n\n ### break the list into chunks.\n max_items_per_page = page_bottom-page_top\n\n ### start at the top of the list\n top_item=0\n f=open('log.msg','w')\n first_item=page_top\n current_item=0\n item_list=[]\n while 1:\n _lw.erase()\n _lw.box()\n _lw.addstr(page_top-1,x_start,header)\n if top_item > len(choices):\n top_item=0\n for i in range(max_items_per_page):\n item=i+top_item\n if not item in range(len(choices)):\n break\n _lw.addstr(i+page_top,x_start,choices[item])\n ### provide a hint that there is more info in the list\n ### setup where we are in the list\n last_item=item\n if top_item > 0 :\n _lw.addstr(page_bottom,x_start,\"P(revious)\")\n if last_item < len(choices):\n _lw.addstr(page_bottom,x_max-8,\"N(ext)\")\n while 1:\n c=_lw.getch(current_item-top_item+page_top,x_start)\n if c==curses.KEY_UP:\n current_item=current_item-1\n elif c==curses.KEY_DOWN:\n current_item=current_item+1\n elif c==ord(' '):\n if current_item in item_list:\n _lw.addstr(choices[current_item])\n item_list.remove(current_item)\n else:\n _lw.addstr(choices[current_item],curses.A_REVERSE)\n item_list.append(current_item)\n elif c==ord('P'):\n top_item=top_item-max_items_per_page\n current_item=top_item\n break\n elif c==ord('N'):\n top_item=top_item + max_items_per_page\n current_item=top_item\n break\n elif c==10:\n return(item_list)\n elif c==ord('q'):\n _lw.erase()\n return(None)\n\t\telif c==ord('x'):\n\t\t choices[current_item]=choices[current_item][:4]+\" \"+choices[current_item][5:]\n _lw.addstr(choices[current_item])\n\t\telse:\n\t\t choices[current_item]=choices[current_item][:7]+chr(c).capitalize()+choices[current_item][8:]\n _lw.addstr(choices[current_item])\n\n if current_item > last_item-1:\n if last_item < len(choices):\n top_item = top_item+1\n break\n else:\n current_item=current_item-1\n if current_item < top_item :\n if top_item > 0:\n top_item = top_item-1\n break\n else:\n current_item=current_item+1", "response": "Display list of choices. As many as we can get in a page."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nrecenters the reference position values for the center pixel of the reference frame.", "response": "def recenter(self):\n \"\"\"\n Reset the reference position values to correspond to the center\n of the reference frame.\n Algorithm used here developed by Colin Cox - 27-Jan-2004.\n \"\"\"\n if self.ctype1.find('TAN') < 0 or self.ctype2.find('TAN') < 0:\n print 'WCS.recenter() only supported for TAN projections.'\n raise TypeError\n\n # Check to see if WCS is already centered...\n if self.crpix1 == self.naxis1/2. and self.crpix2 == self.naxis2/2.:\n # No recentering necessary... return without changing WCS.\n return\n\n # This offset aligns the WCS to the center of the pixel, in accordance\n # with the 'align=center' option used by 'drizzle'.\n #_drz_off = -0.5\n _drz_off = 0.\n _cen = (self.naxis1/2.+ _drz_off,self.naxis2/2. + _drz_off)\n\n # Compute the RA and Dec for center pixel\n _cenrd = self.xy2rd(_cen)\n _cd = N.array([[self.cd11,self.cd12],[self.cd21,self.cd22]],type=N.Float64)\n _ra0 = DEGTORAD(self.crval1)\n _dec0 = DEGTORAD(self.crval2)\n _ra = DEGTORAD(_cenrd[0])\n _dec = DEGTORAD(_cenrd[1])\n\n # Set up some terms for use in the final result\n _dx = self.naxis1/2. - self.crpix1\n _dy = self.naxis2/2. - self.crpix2\n\n _dE,_dN = DEGTORAD(N.dot(_cd,(_dx,_dy)))\n _dE_dN = 1 + N.power(_dE,2) + N.power(_dN,2)\n _cosdec = N.cos(_dec)\n _sindec = N.sin(_dec)\n _cosdec0 = N.cos(_dec0)\n _sindec0 = N.sin(_dec0)\n\n _n1 = N.power(_cosdec,2) + _dE*_dE + _dN*_dN*N.power(_sindec,2)\n _dra_dE = (_cosdec0 - _dN*_sindec0)/_n1\n _dra_dN = _dE*_sindec0 /_n1\n\n _ddec_dE = -_dE*N.tan(_dec) / _dE_dN\n _ddec_dN = (1/_cosdec) * ((_cosdec0 / N.sqrt(_dE_dN)) - (_dN*N.sin(_dec) / _dE_dN))\n\n # Compute new CD matrix values now...\n _cd11n = _cosdec * (self.cd11*_dra_dE + self.cd21 * _dra_dN)\n _cd12n = _cosdec * (self.cd12*_dra_dE + self.cd22 * _dra_dN)\n _cd21n = self.cd11 * _ddec_dE + self.cd21 * _ddec_dN\n _cd22n = self.cd12 * _ddec_dE + self.cd22 * _ddec_dN\n\n _new_orient = RADTODEG(N.arctan2(_cd12n,_cd22n))\n\n # Update the values now...\n self.crpix1 = _cen[0]\n self.crpix2 = _cen[1]\n self.crval1 = RADTODEG(_ra)\n self.crval2 = RADTODEG(_dec)\n\n # Keep the same plate scale, only change the orientation\n self.rotateCD(_new_orient)\n\n # These would update the CD matrix with the new rotation\n # ALONG with the new plate scale which we do not want.\n self.cd11 = _cd11n\n self.cd12 = _cd12n\n self.cd21 = _cd21n\n self.cd22 = _cd22n"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _buildNewKeyname(self,key,prepend):\n\n if len(prepend+key) <= 8: _new_key = prepend+key\n else: _new_key = str(prepend+key)[:8]\n\n return _new_key", "response": "Builds a new keyword based on original keyword name and\n a prepend string."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef copy(self,deep=yes):\n if deep:\n return copy.deepcopy(self)\n else:\n return copy.copy(self)", "response": "Makes a deep copy of this object for use by other objects."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\npull the three images and then scramble the MJD - OBS keywords", "response": "def scrambleTriples(expnums,ccd):\n \"\"\"Pull the three images and then scramble the MJD-OBS keywords\"\"\"\n import pyfits, MOPfits\n mjd=[]\n fid=[]\n fs=[]\n filenames=[]\n for expnum in expnums:\n if int(ccd)<18:\n cutout=\"[-*,-*]\"\n else:\n cutout=\"[*,*]\"\n filenames.append(MOPfits.adGet(str(expnum)+opt.raw,extno=int(ccd),cutout=cutout))\n\tfs.append(pyfits.open(filenames[-1]))\n\tmjd.append(fs[-1][0].header.get(\"MJD-OBS\"))\n\tfid.append(fs[-1][0].header.get(\"EXPNUM\"))\n\n if not os.access('weight.fits',os.F_OK):\n os.symlink(MOPfits.adGet(\"weight\",extno=int(ccd),cutout=cutout),'weight.fits')\n\n for filename in filenames:\n if not os.access(filename,os.R_OK):\n sys.stderr.write(\"Ad Get Failed\\n\")\n raise TaskError, 'adGet Failed'\n\t \n order=[2,0,1]\n basenames=[]\n for i in range(len(fs)):\n fs[i][0].header.update(\"MJD-OBS\",mjd[order[i]]);\n fs[i][0].header.update(\"EXPNUM\",fid[order[i]]);\n filename=filenames[order[i]].replace(opt.raw,'s')\n basenames.append(os.path.splitext(filename)[0])\n if os.access(filename,os.F_OK):\n os.unlink(filename)\n fs[i][0].writeto(filename,output_verify='ignore')\n os.unlink(filenames[order[i]])\n\n return basenames"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\npull the three images and then scramble the MJD - OBS keywords", "response": "def getTriples(expnums,ccd):\n \"\"\"Pull the three images and then scramble the MJD-OBS keywords\"\"\"\n import pyfits, MOPfits\n filenames=[]\n for expnum in expnums:\n if int(ccd)<18:\n cutout=\"[-*,-*]\"\n else:\n cutout=\"[*,*]\"\n filenames.append(MOPfits.adGet(str(expnum)+opt.raw,extno=int(ccd),cutout=cutout))\n\n for filename in filenames:\n if not os.access(filename,os.R_OK):\n sys.stderr.write(\"Ad Get Failed\\n\")\n raise TaskError, 'adGet Failed'\n \n if not os.access('weight.fits',os.F_OK):\n os.symlink(MOPfits.adGet(\"weight\",extno=int(ccd),cutout=cutout),'weight.fits')\n \n basenames=[]\n for filename in filenames:\n filename=os.path.splitext(filename)\n basenames.append(filename[0])\n \n print filenames, basenames\n return basenames"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef searchTriples(filenames,plant=False):\n\n print filenames\n if opt.none :\n return\n\n\n import MOPfits,os \n import MOPdbaccess\n import string\n import os.path\n import pyfits\n\n \n if len(filenames)!=3:\n raise TaskError, \"got %d exposures\"%(len(expnums))\n\n\n ### Some program Constants\n proc_these_files=[]\n if not plant:\n proc_these_files.append(\"# Files to be planted and searched\\n\")\n proc_these_files.append(\"# image fwhm plant\\n\")\n\n \n for filename in filenames:\n \ttry: \n mysql=MOPdbaccess.connect('bucket','cfhls','MYSQL')\n bucket=mysql.cursor()\n\texcept:\n raise TaskError, \"mysql failed\"\n #bucket.execute(\"SELECT obs_iq_refccd FROM exposure WHERE expnum=%s\" , (expnum, ) )\n #row=bucket.fetchone()\n\t#mysql.close()\n #fwhm=row[0]\n #if not fwhm > 0:\n fwhm=1.0\n\n if not plant:\n #proc_these_files.append(\"%s %f %s \\n\" % ( filename[0], fwhm/0.183, 'no'))\n\t pstr='NO'\n else:\n\t pstr='YES'\n ### since we're planting we need a psf. JMPMAKEPSF will\n ### update the proc-these-files listing\n\n ### run the make psf script .. always. This creates proc-these-files\n ### which is needed by the find.pl script.\n command='jmpmakepsf.csh ./ %s %s' % ( filename, pstr )\n if opt.verbose:\n sys.stderr.write( command )\n try:\n os.system(command)\n except:\n raise TaskError, \"jmpmakepsf noexec\"\n if os.access(filename+'.jmpmakepsf.FAILED',os.R_OK) or not os.access(filename+\".psf.fits\", os.R_OK) :\n\t if plant:\n raise TaskError, \"jmpmakepsf failed\"\n#\tdo without plant\n else:\n\t plant=False\n\t\tpstr='NO'\n\t ### we're not planting so, lets keep going\n\t\t### but check that there is a line in proc_these_files\n\t add_line=True\n\t\tif not os.access('proc-these-files',os.R_OK):\n\t\t f=open('proc-these-files','w')\n\t\t for l in proc_these_files:\n\t\t f.write(l)\n\t\t f.close()\n\t \tf=open('proc-these-files','r')\n\t\tptf_lines=f.readlines()\n\t\tf.close()\n\t\tfor ptf_line in ptf_lines:\n\t\t if ptf_line[0]=='#':\n\t\t continue\n\t ptf_a=ptf_line.split()\n\t\t import re\n\t\t if re.search('%s' % (filename),ptf_a[0]):\n\t\t ### there's already a line for this one\n\t\t\tadd_line=False\n\t\t break\n if add_line:\n\t\t f=open('proc-these-files','a')\n\t\t f.write(\"%s %f %s \\n\" % ( filename, fwhm/0.183, 'no'))\n\t\t f.close()\n\n if opt.none:\n return(-1)\n prefix=''\n if plant:\n command=\"plant.csh ./ \" \n #command=\"plant.csh ./ -rmin %s -rmax %s -ang %s -width %s \" % ( opt.rmin, opt.rmax, opt.angle, opt.width)\n try: \n os.system(command)\n except:\n raise TaskError, 'plant exec. failed'\n if not os.access('plant.OK',os.R_OK):\n raise TaskError, 'plant failed'\n prefix='fk'\n #else:\n # f=open('proc-these-files','w')\n # for line in proc_these_files:\n # f.write(line)\n # f.flush()\n # f.close()\n \t\n if opt.rerun and os.access('find.OK',os.R_OK):\n os.unlink(\"find.OK\")\n\n #command=\"find.pl -p \"+prefix+\" -rn %s -rx %s -a %s -aw %s -d ./ \" % ( opt.rmin, opt.rmax, opt.angle, opt.width) \n command=\"find.pl -p \"+prefix+\" -d ./ \" \n if opt.union :\n command+=\" -u\"\n\n if opt.verbose:\n sys.stderr.write( command )\n\n try:\n os.system(command)\n except:\n raise TaskErorr, \"execute find\"\n \n\n if not os.access(\"find.OK\",os.R_OK):\n raise TaskError, \"find failed\"\n\n ### check the transformation file\n command = \"checktrans -p \"+prefix\n \n try:\n os.system(command)\n except:\n raise TaskError, \"execute checktrans\"\n \n if not os.access(\"checktrans.OK\",os.R_OK):\n raise TaskError, \"checktrans failed\"\n elif os.access(\"checktrans.FAILED\",os.R_OK):\n os.unlink(\"checktrans.FAILED\")\n\n if os.access(\"BAD_TRANS\"+prefix,os.R_OK):\n raise TaskError,\"BAD TRANS\"\n\n ## check that the transformation in .trans.jmp files look reasonable\n import math\n for filename in filenames:\n try:\n for line in open(filename+\".trans.jmp\"):\n for v in line.split():\n if math.fabs(float(v)) > 200:\n raise TaskError,\"BAD TRANS\"\n except:\n raise TaskError, \"TRAN_CHECK FAILED\"\n \n astrom=prefix+filenames[0]+\".cands.comb\"\n if opt.plant:\n for filename in filenames:\n\t try:\n\t ushort(prefix+filename+\".fits\")\n\t except:\n\t raise TaskError(\"ushort failed %s\" % (prefix+filename+\".fits\"))\n\n if opt.plant:\n astrom=prefix+filenames[0]+\".comb.found\"\n try:\n #make sure we have +5 lines in this file\n\t lines=file(astrom).readlines()\n\t if len(lines)<5:\n\t raise TaskError,\"Too few Found\"\n\texcept:\n\t raise TaskError, \"Error reading %s\" %(astrom)\n \n \n if os.access(astrom,os.R_OK):\n return(1)\n else:\n return(0)", "response": "This function searches for all the Triples in a set of exposures and returns a list of the KBOs that are found in that set."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef ushort(filename):\n import pyfits\n f=pyfits.open(filename,mode='update')\n f[0].scale('int16','',bzero=32768)\n f.flush()\n f.close()", "response": "Ushort a the pixels"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nparses a file into an AstromData structure.", "response": "def parse(self, filename):\n \"\"\"\n Parses a file into an AstromData structure.\n\n Args:\n filename: str\n The name of the file whose contents will be parsed.\n\n Returns:\n data: AstromData\n The file contents extracted into a data structure for programmatic\n access.\n \"\"\"\n filehandle = storage.open_vos_or_local(filename, \"rb\")\n assert filehandle is not None, \"Failed to open file {} \".format(filename)\n filestr = filehandle.read()\n filehandle.close()\n\n assert filestr is not None, \"File contents are None\"\n\n observations = self._parse_observation_list(filestr)\n\n self._parse_observation_headers(filestr, observations)\n\n sys_header = self._parse_system_header(filestr)\n\n sources = self._parse_source_data(filestr, observations)\n\n return AstromData(observations, sys_header, sources, discovery_only=self.discovery_only)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nwrites the header of the MJD - Obs model.", "response": "def _write_observation_headers(self, observations):\n \"\"\"\n See src/pipematt/step1matt-c\n \"\"\"\n for observation in observations:\n header = observation.header\n\n def get_header_vals(header_list):\n header_vals = []\n for key in header_list:\n val = header[key]\n\n if key == MJD_OBS_CENTER:\n header_vals.append(val)\n elif key == DETECTOR:\n header_vals.append(val.ljust(20))\n else:\n header_vals.append(float(val))\n\n return tuple(header_vals)\n\n self._write_line(\"## MOPversion\")\n self._write_line(\"# %s\" % header[MOPVERSION])\n self._write_line(\"## MJD-OBS-CENTER EXPTIME THRES FWHM MAXCOUNT CRVAL1 CRVAL2 EXPNUM\")\n self._write_line(\"# %s%8.2f%6.2f%6.2f%9.1f%11.5f%11.5f%9d\" % get_header_vals(\n [MJD_OBS_CENTER, EXPTIME, THRES, FWHM, MAXCOUNT, CRVAL1, CRVAL2, EXPNUM]))\n self._write_line(\"## SCALE CHIP CRPIX1 CRPIX2 NAX1 NAX2 DETECTOR PHADU RDNOIS\")\n self._write_line(\"# %6.3f%4d%10.2f%10.2f%6d%6d %s%5.2f %5.2f\" % get_header_vals(\n [SCALE, CHIP, CRPIX1, CRPIX2, NAX1, NAX2, DETECTOR, PHADU, RDNOIS]))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _write_sys_header(self, sys_header):\n header_vals = [sys_header[RMIN], sys_header[RMAX], sys_header[ANGLE],\n sys_header[AWIDTH]]\n self._write_line(\"## RMIN RMAX ANGLE AWIDTH\")\n self._write_line(\"# %8.1f%8.1f%8.1f%8.1f\" % tuple(map(float, header_vals)))", "response": "Write the sys header."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _write_source_data(self, sources):\n for i, source in enumerate(sources):\n self._write_source(source)", "response": "Write the source data to the file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nwrite the header part of the astrom file so that only the sourceCOOKIE and sourceCOOKIE are written.", "response": "def write_headers(self, observations, sys_header):\n \"\"\"\n Writes the header part of the astrom file so that only the source\n data has to be filled in.\n \"\"\"\n if self._header_written:\n raise AstromFormatError(\"Astrom file already has headers.\")\n\n self._write_observation_list(observations)\n self._write_observation_headers(observations)\n self._write_sys_header(sys_header)\n self._write_source_header()\n\n self._header_written = True"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef write_source(self, source):\n if not self._header_written:\n observations = [reading.get_observation() for reading in source.get_readings()]\n self.write_headers(observations, self.sys_header)\n\n self._write_source(source)", "response": "Writes out data for a single source."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef write_astrom_data(self, astrom_data):\n self.write_headers(astrom_data.observations, astrom_data.sys_header)\n self._write_source_data(astrom_data.sources)", "response": "Writes an AstromData structure at once."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns x y in the original frame based on a guess as much as anything.", "response": "def _original_frame(self, x, y):\n \"\"\"\n Return x/y in the original frame, based on a guess as much as anything.\n :param x: x pixel coordinate\n :type x: float\n :param y: y pixel coordinate\n :type y: float\n :return: x,y\n :rtype: float, float\n \"\"\"\n if self._inverted:\n return self.obs.naxis1 - x, self.obs.naxis2 - y\n return x, y"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting the _pix_coord attribute of the object", "response": "def pix_coord(self, pix_coord):\n \"\"\"\n :type pix_coord: list\n :param pix_coord: an x,y pixel coordinate, origin = 1\n \"\"\"\n try:\n pix_coord = list(pix_coord)\n except:\n pass\n if not isinstance(pix_coord, list) or len(pix_coord) != 2:\n raise ValueError(\"pix_coord needs to be set with an (x,y) coordinate pair, got {}\".format(pix_coord))\n x, y = pix_coord\n if not isinstance(x, Quantity):\n x = float(x) * units.pix\n if not isinstance(y, Quantity):\n y = float(y) * units.pix\n self._pix_coord = x, y"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the location of the source in terms of the reference image coordinates.", "response": "def reference_source_point(self):\n \"\"\"\n The location of the source in the reference image, in terms of the\n current image coordinates.\n \"\"\"\n xref = isinstance(self.xref, Quantity) and self.xref.value or self.xref\n yref = isinstance(self.yref, Quantity) and self.yref.value or self.yref\n\n return xref + self.x_ref_offset, yref + self.y_ref_offset"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_coordinate_offset(self, other_reading):\n my_x, my_y = self.reference_source_point\n other_x, other_y = other_reading.reference_source_point\n return my_x - other_x, my_y - other_y", "response": "Calculates the x and y offsets between this reading and the other reading s base - point of the other reading."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef compute_inverted(self):\n # astheader = storage.get_astheader(self.obs.expnum, self.obs.ccdnum, version=self.obs.ftype)\n # pvwcs = wcs.WCS(astheader)\n # (x, y) = pvwcs.sky2xy(self.ra, self.dec)\n # logger.debug(\"is_inverted: X,Y {},{} -> wcs X,Y {},{}\".format(self.x, self.y, x, y))\n # dr2 = ((x-self.x)**2 + (y-self.y)**2)\n # return dr2 > 2\n\n if self.ssos or self.obs.is_fake() or self.obs.ftype == 's':\n inverted = False\n else:\n inverted = True if self.get_ccd_num() - 1 in INVERTED_CCDS else False\n logger.debug(\"Got that {} is_inverted: {}\".format(self.obs.rawname, inverted))\n return inverted", "response": "Returns:\n inverted: bool\n True if the stored image is inverted."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef from_source_reference(expnum, ccd, x, y):\n\n image_uri = storage.dbimages_uri(expnum=expnum,\n ccd=None,\n version='p',\n ext='.fits',\n subdir=None)\n logger.debug('Trying to access {}'.format(image_uri))\n\n if not storage.exists(image_uri, force=False):\n logger.warning('Image not in dbimages? Trying subdir.')\n image_uri = storage.dbimages_uri(expnum=expnum,\n ccd=ccd,\n version='p')\n\n if not storage.exists(image_uri, force=False):\n logger.warning(\"Image doesn't exist in ccd subdir. %s\" % image_uri)\n return None\n\n if x == -9999 or y == -9999:\n logger.warning(\"Skipping {} as x/y not resolved.\".format(image_uri))\n return None\n\n mopheader_uri = storage.dbimages_uri(expnum=expnum,\n ccd=ccd,\n version='p',\n ext='.mopheader')\n if not storage.exists(mopheader_uri, force=False):\n # ELEVATE! we need to know to go off and reprocess/include this image.\n logger.critical('Image exists but processing incomplete. Mopheader missing. {}'.format(image_uri))\n return None\n\n # Build astrom.Observation\n observation = Observation(expnum=str(expnum),\n ftype='p',\n ccdnum=str(ccd),\n fk=\"\")\n\n # JJK commented this out, I think the following line is not true?\n # observation.rawname = os.path.splitext(os.path.basename(image_uri))[0]+str(ccd).zfill(2)\n\n return observation", "response": "Given the location of a source in the image create an Observation object and return it."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef write_source(self, source, comment=None, reject=False):\n if not self._header_written:\n observations = [reading.get_observation() for reading in source.get_readings()]\n self.write_headers(observations, self.sys_header)\n\n self._write_source(source, comment=comment, reject=reject)", "response": "Writes out data for a single source."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef mjd2gmst(mjd):\n\n tu = (mjd - MJD0) / (100*DPY)\n\n st = math.fmod(mjd, 1.0) * D2PI + (24110.54841 + (8640184.812866 + (0.093104 - 6.2e-6 * tu) * tu) * tu) * DS2R\n\n w = math.fmod(st, D2PI)\n if w >= 0.0:\n return w\n else:\n return w + D2PI", "response": "Convert Modfied Juian Date to GMST."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _gradient_penalty(self, real_samples, fake_samples, kwargs):\n import torch\n from torch.autograd import Variable, grad\n\n real_samples = real_samples.view(fake_samples.shape)\n\n subset_size = real_samples.shape[0]\n\n real_samples = real_samples[:subset_size]\n fake_samples = fake_samples[:subset_size]\n\n alpha = torch.rand(subset_size)\n if self.use_cuda:\n alpha = alpha.cuda()\n alpha = alpha.view((-1,) + ((1,) * (real_samples.dim() - 1)))\n\n interpolates = alpha * real_samples + ((1 - alpha) * fake_samples)\n interpolates = Variable(interpolates, requires_grad=True)\n if self.use_cuda:\n interpolates = interpolates.cuda()\n\n d_output = self.critic(interpolates, **kwargs)\n\n grad_ouputs = torch.ones(d_output.size())\n if self.use_cuda:\n grad_ouputs = grad_ouputs.cuda()\n\n gradients = grad(\n outputs=d_output,\n inputs=interpolates,\n grad_outputs=grad_ouputs,\n create_graph=True,\n retain_graph=True,\n only_inputs=True)[0]\n return ((gradients.norm(2, dim=1) - 1) ** 2).mean() * 10", "response": "Compute the norm of the gradients for each sample in a batch and penalize anything on either side of unit norm"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef remeasure(mpc_in, reset_pixel_coordinates=True):\n if mpc_in.null_observation:\n return mpc_in\n mpc_obs = deepcopy(mpc_in)\n logging.debug(\"rm start: {}\".format(mpc_obs.to_string()))\n\n if not isinstance(mpc_obs.comment, mp_ephem.ephem.OSSOSComment):\n logging.error(\"Failed to convert comment line\")\n return mpc_in\n\n parts = re.search('(?P\\d{7})(?P\\S)(?P\\d\\d)', str(mpc_obs.comment.frame))\n if not parts:\n logging.error(\"Failed to parse expnum from frame info in comment line\")\n return mpc_in\n ccd = int(parts.group('ccd'))\n expnum = int(parts.group('expnum'))\n exp_type = parts.group('type')\n\n try:\n header = _connection_error_wrapper(storage._get_sghead, expnum)[ccd+1]\n except IOError as ioerr:\n logging.error(str(ioerr))\n logging.error(\"Failed to get astrometric header for: {}\".format(mpc_obs))\n return mpc_in\n\n this_wcs = wcs.WCS(header)\n\n coordinate = this_wcs.xy2sky(mpc_obs.comment.x, mpc_obs.comment.y, usepv=True)\n mpc_obs.coordinate = coordinate[0].to('degree').value, coordinate[1].to('degree').value\n sep = mpc_in.coordinate.separation(mpc_obs.coordinate)\n\n if sep > TOLERANCE*20 and mpc_in.discovery and _flipped_ccd(ccd):\n logging.warn(\"Large ({}) offset using X/Y in comment line to compute RA/DEC\".format(sep))\n if reset_pixel_coordinates:\n logging.info(\"flipping/flopping the discvoery x/y position recorded.\")\n x = header['NAXIS1'] - mpc_obs.comment.x + 1\n y = header['NAXIS2'] - mpc_obs.comment.y + 1\n new_coordinate = this_wcs.xy2sky(x, y, usepv=True)\n new_sep = mpc_in.coordinate.separation(new_coordinate)\n if new_sep < TOLERANCE*2:\n mpc_obs.coordinate = new_coordinate\n mpc_obs.comment.x = x\n mpc_obs.comment.y = y\n sep = new_sep\n\n if sep > TOLERANCE:\n # use the old header RA/DEC to predict the X/Y and then use that X/Y to get new RA/DEC\n logging.warn(\"sep: {} --> large offset when using comment line X/Y to compute RA/DEC\")\n if reset_pixel_coordinates:\n logging.warn(\"Using RA/DEC and original WCS to compute X/Y and replacing X/Y in comment.\".format(sep))\n header2 = _connection_error_wrapper(storage.get_astheader, expnum, ccd)\n image_wcs = wcs.WCS(header2)\n (x, y) = image_wcs.sky2xy(mpc_in.coordinate.ra.degree, mpc_in.coordinate.dec.degree, usepv=False)\n mpc_obs.coordinate = this_wcs.xy2sky(x, y, usepv=True)\n mpc_obs.comment.x = x\n mpc_obs.comment.y = y\n logging.info(\"Coordinate changed: ({:5.2f},{:5.2f}) --> ({:5.2f},{:5.2f})\".format(mpc_obs.comment.x,\n mpc_obs.comment.y,\n x, y))\n\n if mpc_obs.comment.mag_uncertainty is not None:\n try:\n merr = float(mpc_obs.comment.mag_uncertainty)\n fwhm = float(_connection_error_wrapper(storage.get_fwhm, expnum, ccd))\n centroid_err = merr * fwhm * header['PIXSCAL1']\n logging.debug(\"Centroid uncertainty: {} {} => {}\".format(merr, fwhm, centroid_err))\n except Exception as err:\n logging.error(str(err))\n logging.error(\"Failed to compute centroid_err for observation:\\n\"\n \"{}\\nUsing default of 0.2\".format(mpc_obs.to_string()))\n centroid_err = 0.2\n else:\n centroid_err = 0.2\n\n mpc_obs.comment.astrometric_level = header.get('ASTLEVEL', \"0\")\n\n try:\n asterr = float(header['ASTERR'])\n residuals = (asterr ** 2 + centroid_err ** 2) ** 0.5\n logging.debug(\"Residuals: {} {} => {}\".format(asterr, centroid_err, residuals))\n except Exception as err:\n logging.error(str(err))\n logging.error(\"Failed while trying to compute plate uncertainty for\\n{}\".format(mpc_obs.to_string()))\n logging.error('Using default of 0.25')\n residuals = 0.25\n\n mpc_obs.comment.plate_uncertainty = residuals\n\n logging.debug(\"sending back: {}\".format(mpc_obs.to_string()))\n\n return mpc_obs", "response": "Compute the RA and DEC of the line of the given astrometric line."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nwrap a call to func in a try / except that repeats on ConnectionError and logs a warning if the call fails.", "response": "def _connection_error_wrapper(func, *args, **kwargs):\n \"\"\"\n Wrap a call to func in a try/except that repeats on ConnectionError\n @param func:\n @param args:\n @param kwargs:\n @return:\n \"\"\"\n\n counter = 0\n while counter < 5:\n try:\n result = func(*args, **kwargs)\n return result\n except Exception as ex:\n time.sleep(5)\n counter += 1\n logging.warning(str(ex))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef recompute_mag(mpc_in, skip_centroids=False):\n # TODO this really shouldn't need to build a 'reading' to get the cutout...\n\n from ossos.downloads.cutouts import downloader\n dlm = downloader.ImageCutoutDownloader()\n\n mpc_obs = deepcopy(mpc_in)\n assert isinstance(mpc_obs, mp_ephem.ephem.Observation)\n assert isinstance(mpc_obs.comment, mp_ephem.ephem.OSSOSComment)\n\n if mpc_obs.null_observation:\n return mpc_obs\n\n parts = re.search('(?P\\d{7})(?P\\S)(?P\\d\\d)', mpc_obs.comment.frame)\n if parts is None:\n return mpc_obs\n expnum = parts.group('expnum')\n ccd = parts.group('ccd')\n file_type = parts.group('type')\n\n observation = astrom.Observation(expnum, file_type, ccd)\n assert isinstance(observation, astrom.Observation)\n\n ast_header = _connection_error_wrapper(storage._get_sghead, int(expnum))[int(ccd)+1]\n\n filter_value = None\n for keyword in ['FILTER', 'FILT1 NAME']:\n filter_value = ast_header.get(keyword, None)\n if filter_value is not None:\n if filter_value.startswith('gri'):\n filter_value = 'w'\n else:\n filter_value = filter_value[0]\n break\n # The ZP for the current astrometric lines is the pipeline one. The new ZP is in the astheader file.\n new_zp = ast_header.get('PHOTZP')\n\n # The .zeropoint.used value is likely the one used for the original photometry.\n old_zp = _connection_error_wrapper(storage.get_zeropoint, int(expnum), int(ccd))\n\n reading = astrom.SourceReading(float(mpc_obs.comment.x), float(mpc_obs.comment.y), float(mpc_obs.comment.x),\n float(mpc_obs.comment.y), mpc_obs.coordinate.ra.degree,\n mpc_obs.coordinate.dec.degree, float(mpc_obs.comment.x), float(mpc_obs.comment.y),\n observation, ssos=True, from_input_file=True, null_observation=False,\n discovery=mpc_obs.discovery)\n\n cutout = _connection_error_wrapper(dlm.download_cutout, reading, needs_apcor=True)\n cutout._zmag = new_zp\n\n if math.fabs(cutout.zmag - old_zp) > 0.3:\n logging.warn(\"Large change in zeropoint detected: {} -> {}\".format(old_zp, cutout.zmag))\n\n try:\n PHOT = cutout.get_observed_magnitude(centroid=not skip_centroids and mpc_obs.note1 != \"H\")\n x = PHOT['XCENTER']\n y = PHOT['YCENTER']\n mag = PHOT['MAG']\n merr = PHOT['MERR']\n cutout.update_pixel_location((x, y), hdu_index=cutout.extno)\n x, y = cutout.observed_source_point\n except Exception as ex:\n logging.error(\"ERROR: {}\".format(str(ex)))\n return mpc_obs\n\n try:\n if mpc_obs.comment.mag_uncertainty is not None and mpc_obs.comment.mag is not None and math.fabs(mpc_obs.comment.mag - mag) > 3.5 * mpc_obs.comment.mag_uncertainty:\n logging.warn(\"recomputed magnitude shift large: {} --> {}\".format(mpc_obs.mag, mag[0]))\n if math.sqrt((x.value - mpc_obs.comment.x) ** 2 + (y.value - mpc_obs.comment.y) ** 2) > 1.9:\n logging.warn(\"Centroid shifted ({},{}) -> ({},{})\".format(mpc_obs.comment.x,\n mpc_obs.comment.y,\n x.value,\n y.value))\n except Exception as ex:\n logging.error(str(ex))\n\n # Don't use the new X/Y for Hand measured entries. (although call to get_observed_magnitude should have changed)\n if str(mpc_obs.note1) != \"H\" and not skip_centroids:\n mpc_obs.comment.x = x.value\n mpc_obs.comment.y = y.value\n\n try:\n mag = float(mag)\n except:\n return mpc_obs\n\n if math.isnan(mag):\n return mpc_obs\n\n if mag > 10:\n mpc_obs._band = filter_value\n mpc_obs.comment.mag = mag\n mpc_obs.comment.mag_uncertainty = merr\n\n # Update the mpc record magnitude if previous value existed here.\n if (mpc_obs.mag is not None or (mpc_obs.mag is None and mpc_in.comment.photometry_note[0] == \"Z\")) and mag > 10:\n mpc_obs.mag = mag\n\n return mpc_obs", "response": "Get the mag of the object given the mp_ephem. ephem. Observation object."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef compare_orbits(original_obs, modified_obs, cor_file):\n\n origin = orbfit.Orbfit(original_obs)\n modified = orbfit.Orbfit(modified_obs)\n\n orbpt = file(cor_file+\".orb\", 'w')\n\n # Dump summaries of the orbits\n orbpt.write(\"#\"*80+\"\\n\")\n orbpt.write(\"# ORIGINAL ORBIT\\n\")\n orbpt.write(origin.summarize()+\"\\n\")\n orbpt.write(\"#\"*80+\"\\n\")\n orbpt.write(\"# MODIFIED ORBIT\\n\")\n orbpt.write(modified.summarize()+\"\\n\")\n orbpt.write(\"#\"*80+\"\\n\")\n\n # Create a report of the change in orbit parameter uncertainty\n for element in ['a', 'e', 'inc', 'om', 'Node', 'T']:\n oval = getattr(origin, element).value\n doval = getattr(origin, \"d\"+element).value\n mval = getattr(modified, element).value\n dmval = getattr(modified, \"d\"+element).value\n precision = max(int(-1*math.floor(math.log10(dmval))), int(-1*math.floor(math.log10(doval)))) + 1\n precision = max(0, precision)\n vdigits = 12\n ddigits = 6\n vpadding = \" \"*int(vdigits-precision)\n dpadding = \" \"*int(ddigits-precision)\n\n orbpt.write(\"{element:>5s}: \"\n \"{oval[0]:>{vdigits}.{vdigits}}.{oval[1]:<{precision}.{precision}} {vpadding} +/- \"\n \"{doval[0]:>{ddigits}.{ddigits}}.{doval[1]:<{precision}.{precision}} {dpadding} ==> \"\n \"{mval[0]:>{vdigits}.{vdigits}}.{mval[1]:<{precision}.{precision}} {vpadding} +/- \"\n \"{dmval[0]:>{ddigits}.{ddigits}}.{dmval[1]:<{precision}.{precision}}\\n\".format(\n element=element,\n dpadding=dpadding,\n vpadding=vpadding,\n vdigits=vdigits,\n ddigits=ddigits,\n oval=\"{:12.12f}\".format(oval).split(\".\"),\n doval=\"{:12.12f}\".format(doval).split(\".\"),\n mval=\"{:12.12f}\".format(mval).split(\".\"),\n dmval=\"{:12.12f}\".format(dmval).split(\".\"),\n precision=precision)\n )\n\n delta = math.fabs(oval - mval)\n if delta > 3.5 * doval:\n logging.warn(\"large delta for element {}: {} --> {}\".format(element, oval, mval))\n\n\n # Compute the stdev of the residuals and report the change given the new observations\n orbpt.write(\"*\"*80+\"\\n\")\n orbpt.write(\"Change in orbital parameters \\n\")\n sep = \"Change in scatter between initial and recalibrated obseravtions. \\n\"\n for orb in [origin, modified]:\n orbpt.write(sep)\n sep = \"\\n ==> becomes ==> \\n\"\n residuals = orb.residuals\n dra = []\n ddec = []\n mags = {}\n for observation in orb.observations:\n if not observation.null_observation:\n dra.append(observation.ra_residual)\n ddec.append(observation.dec_residual)\n filter = observation.band\n if filter is not None:\n if filter not in mags:\n mags[filter] = []\n try:\n mags[filter].append(float(observation.mag))\n except:\n pass\n if observation.comment.plate_uncertainty * 5.0 < \\\n ((observation.ra_residual ** 2 + observation.dec_residual ** 2) ** 0.5):\n logging.warn(\"LARGE RESIDUAL ON: {}\".format(observation.to_string()))\n logging.warn(\"Fit residual unreasonably large.\")\n dra = numpy.array(dra)\n ddec = numpy.array(ddec)\n merr_str = \"\"\n for filter in mags:\n mag = numpy.percentile(numpy.array(mags[filter]), (50))\n mags[filter] = numpy.percentile(numpy.array(mags[filter]), (5,95))\n merr = (mags[filter][1] - mags[filter][0])/6.0\n merr_str += \" {}: {:8.2f} +/- {:8.2f}\".format(filter, mag, merr)\n orbpt.write(\"ra_std:{:8.4} dec_std:{:8.4} mag: {}\".format(dra.std(), ddec.std(), merr_str))\n orbpt.write(\"\\n\")\n orbpt.close()", "response": "Compare the orbit fit given the original astrometry and modified astrometry."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning this instance summed to mono.", "response": "def mono(self):\n \"\"\"\n Return this instance summed to mono. If the instance is already mono,\n this is a no-op.\n \"\"\"\n if self.channels == 1:\n return self\n x = self.sum(axis=1) * 0.5\n y = x * 0.5\n return AudioSamples(y, self.samplerate)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns audio samples encoded as bytes given a particular audio format", "response": "def encode(self, flo=None, fmt='WAV', subtype='PCM_16'):\n \"\"\"\n Return audio samples encoded as bytes given a particular audio format\n\n Args:\n flo (file-like): A file-like object to write the bytes to. If flo\n is not supplied, a new :class:`io.BytesIO` instance will be\n created and returned\n fmt (str): A libsndfile-friendly identifier for an audio encoding\n (detailed here: http://www.mega-nerd.com/libsndfile/api.html)\n subtype (str): A libsndfile-friendly identifier for an audio\n encoding subtype (detailed here:\n http://www.mega-nerd.com/libsndfile/api.html)\n\n Examples:\n >>> from zounds import SR11025, AudioSamples\n >>> import numpy as np\n >>> silence = np.zeros(11025*10)\n >>> samples = AudioSamples(silence, SR11025())\n >>> bio = samples.encode()\n >>> bio.read(10)\n 'RIFFx]\\\\x03\\\\x00WA'\n \"\"\"\n flo = flo or BytesIO()\n with SoundFile(\n flo,\n mode='w',\n channels=self.channels,\n format=fmt,\n subtype=subtype,\n samplerate=self.samples_per_second) as f:\n\n if fmt == 'OGG':\n # KLUDGE: Trying to write too-large chunks to an ogg file seems\n # to cause a segfault in libsndfile\n # KLUDGE: This logic is very similar to logic in the OggVorbis\n # processing node, and should probably be factored into a common\n # location\n factor = 20\n chunksize = self.samples_per_second * factor\n for i in range(0, len(self), chunksize):\n chunk = self[i: i + chunksize]\n f.write(chunk)\n else:\n # write everything in one chunk\n f.write(self)\n\n flo.seek(0)\n return flo"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef extend_with_default(validator_class: Any) -> Any:\n validate_properties = validator_class.VALIDATORS['properties']\n\n def set_defaults(validator: Any,\n properties: dict,\n instance: dict,\n schema: dict) -> Iterator[ValidationError]:\n for prop, subschema in properties.items():\n if 'default' in subschema:\n instance.setdefault(prop, subschema['default'])\n\n for error in validate_properties(\n validator, properties, instance, schema,\n ):\n yield error # pragma: no cover\n\n return extend(validator_class, {'properties': set_defaults})", "response": "Extend the given validator class with default values."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef weights(self, other):\n try:\n return self._wdata(other)\n except AttributeError:\n frequency_dim = other.dimensions[-1]\n return self._wdata(frequency_dim.scale)", "response": "Compute weights given a scale or time - frequency representation\n "} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a list of rates angles for each source in the fk*reals. astrom file.", "response": "def rates_angles(fk_candidate_observations):\n \"\"\"\n :param fk_candidate_observations: name of the fk*reals.astrom file to check against Object.planted\n \"\"\"\n\n detections = fk_candidate_observations.get_sources()\n for detection in detections:\n measures = detection.get_readings()\n for measure in measures:\n \n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument('--astrom-filename', default=None, help=\"Give the astrom file directly instead of looking-up \"\n \"using the field/ccd naming scheme.\")\n parser.add_argument('--reals', action='store_true', default=False)\n parser.add_argument('--type', choices=['o', 'p', 's'], help=\"Which type of image.\", default='s')\n parser.add_argument('--measure3', default='vos:OSSOS/measure3/2013B-L_redo/')\n parser.add_argument('--dbimages', default=None)\n parser.add_argument('--dry-run', action='store_true', default=False)\n parser.add_argument('--force', action='store_true', default=False)\n\n args = parser.parse_args()\n\n logging.basicConfig(level=logging.INFO)\n\n prefix = 'fk'\n ext = args.reals and 'reals' or 'cands'\n\n storage.MEASURE3 = args.measure3\n\n if args.dbimages is not None:\n storage.DBIMAGES = args.dbimages\n astrom.DATASET_ROOT = args.dbimages\n\n astrom_uri = storage.get_cands_uri(args.field,\n ccd=args.ccd,\n version=args.type,\n prefix=prefix,\n ext=\"measure3.{}.astrom\".format(ext))\n\n if args.astrom_filename is None:\n astrom_filename = os.path.basename(astrom_uri)\n else:\n astrom_filename = args.astrom_filename\n\n if not os.access(astrom_filename, os.F_OK):\n astrom_filename = os.path.dirname(astrom_uri) + \"/\" + astrom_filename\n\n # Load the list of astrometric observations that will be looked at.\n fk_candidate_observations = astrom.parse(astrom_filename)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndownloads a FITS image as a HDUList.", "response": "def download_hdulist(self, uri, **kwargs):\n \"\"\"\n Downloads a FITS image as a HDUList.\n\n Args:\n uri: The URI of the FITS image to download.\n kwargs: optional arguments to pass to the vos client.\n For example, passing view=\"cutout\" and cutout=[1] will result\n in a cutout of extension 1 from the FITS image specified by the\n URI.\n\n Returns:\n hdulist: astropy.io.fits.hdu.hdulist.HDUList\n The requests FITS image as an Astropy HDUList object\n (http://docs.astropy.org/en/latest/io/fits/api/hdulists.html).\n \"\"\"\n\n logger.debug(str(kwargs))\n hdulist = None\n try:\n vobj = storage.vofile(uri, **kwargs)\n try:\n fobj = cStringIO.StringIO(vobj.read())\n fobj.seek(0)\n hdulist = fits.open(fobj)\n except Exception as e:\n sys.stderr.write(\"ERROR: {}\\n\".format(str(e)))\n sys.stderr.write(\"While loading {} {}\\n\".format(uri, kwargs))\n pass\n finally:\n vobj.close()\n except Exception as e:\n sys.stderr.write(str(e)+\"\\n\")\n sys.stderr.write(\"While opening connection to {}.\\n\".format(uri))\n sys.stderr.write(\"Sending back FLAT instead, too keep display happy.\")\n hdulist = self.download_hdulist('vos:OSSOS/dbimages/calibrators/13AQ05_r_flat.fits', **kwargs)\n return hdulist"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef download_apcor(self, uri):\n\n local_file = os.path.basename(uri)\n if os.access(local_file, os.F_OK):\n fobj = open(local_file)\n else:\n fobj = storage.vofile(uri, view='data')\n fobj.seek(0)\n str = fobj.read()\n fobj.close()\n apcor_str = str\n return ApcorData.from_string(apcor_str)", "response": "Downloads the apcor data."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef from_string(cls, rawstr):\n try:\n args = map(float, rawstr.split())\n except Exception as ex:\n import sys\n logger.error(\"Failed to convert aperture correction: {}\".format(ex))\n raise ex\n return cls(*args)", "response": "Creates an ApcorData record from the raw string format."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nruns the OSSOS jmpmakepsf script.", "response": "def run(expnum, ccd, version, dry_run=False, prefix=\"\", force=False):\n \"\"\"Run the OSSOS jmpmakepsf script.\n\n \"\"\"\n\n message = storage.SUCCESS\n if storage.get_status(task, prefix, expnum, version=version, ccd=ccd) and not force:\n logging.info(\"{} completed successfully for {} {} {} {}\".format(task, prefix, expnum, version, ccd))\n return\n\n with storage.LoggingManager(task, prefix, expnum, ccd, version, dry_run):\n try:\n if not storage.get_status(dependency, prefix, expnum, \"p\", ccd=ccd):\n raise IOError(\"{} not yet run for {}\".format(dependency, expnum))\n\n # confirm destination directory exists.\n destdir = os.path.dirname(\n storage.dbimages_uri(expnum, ccd, prefix=prefix, version=version, ext='fits'))\n if not dry_run:\n storage.mkdir(destdir)\n\n # get image from the vospace storage area\n logging.info(\"Getting fits image from VOSpace\")\n filename = storage.get_image(expnum, ccd, version=version, prefix=prefix)\n\n # get mopheader from the vospace storage area\n logging.info(\"Getting mopheader from VOSpace\")\n mopheader_filename = storage.get_file(expnum, ccd, version=version, prefix=prefix, ext='mopheader')\n\n\n # run mkpsf process\n logging.info(\"Running mkpsf on %s %d\" % (expnum, ccd))\n logging.info(util.exec_prog(['jmpmakepsf.csh',\n './',\n filename,\n 'yes', 'yes']))\n \n if dry_run:\n return\n\n # place the results into VOSpace\n basename = os.path.splitext(filename)[0]\n\n for ext in ('mopheader', 'psf.fits',\n 'zeropoint.used', 'apcor', 'fwhm', 'phot'):\n dest = storage.dbimages_uri(expnum, ccd, prefix=prefix, version=version, ext=ext)\n source = basename + \".\" + str(ext)\n count = 0\n with open(source, 'r'):\n while True:\n count += 1\n try:\n logging.info(\"Attempt {} to copy {} -> {}\".format(count, source, dest))\n storage.copy(source, dest)\n break\n except Exception as ex:\n if count > 10:\n raise ex\n\n # set some data parameters associated with the image, determined in this step.\n storage.set_status('fwhm', prefix, expnum, version=version, ccd=ccd, status=str(storage.get_fwhm(\n expnum, ccd=ccd, prefix=prefix, version=version)))\n storage.set_status('zeropoint', prefix, expnum, version=version, ccd=ccd,\n status=str(storage.get_zeropoint(\n expnum, ccd=ccd, prefix=prefix, version=version)))\n logging.info(message)\n except Exception as e:\n message = str(e)\n logging.error(message)\n \n storage.set_status(task, prefix, expnum, version, ccd=ccd, status=message)\n\n return"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nruns the OSSOS mopheader script.", "response": "def run(expnum, ccd, version, dry_run=False, prefix=\"\", force=False, ignore_dependency=False):\n \"\"\"Run the OSSOS mopheader script.\n\n \"\"\"\n message = storage.SUCCESS\n logging.info(\"Attempting to get status on header for {} {}\".format(expnum, ccd))\n if storage.get_status(task, prefix, expnum, version, ccd) and not force:\n logging.info(\"{} completed successfully for {} {} {} {}\".format(task, prefix, expnum, version, ccd))\n return message\n\n with storage.LoggingManager(task, prefix, expnum, ccd, version, dry_run):\n try:\n logging.info(\"Building a mopheader \")\n if not storage.get_status(dependency, prefix, expnum, \"p\", 36) and not ignore_dependency:\n raise IOError(\"{} not yet run for {}\".format(dependency, expnum))\n\n # confirm destination directory exists.\n destdir = os.path.dirname(storage.dbimages_uri(expnum, ccd, prefix=prefix, version=version, ext='fits'))\n if not dry_run:\n storage.mkdir(destdir)\n\n # get image from the vospace storage area\n logging.info(\"Retrieving image from VOSpace\")\n filename = storage.get_image(expnum, ccd, version=version, prefix=prefix)\n\n # launch the stepZjmp program \n logging.info(\"Launching stepZ on %s %d\" % (expnum, ccd))\n expname = os.path.basename(filename).strip('.fits')\n logging.info(util.exec_prog(['stepZjmp',\n '-f',\n expname]))\n # if this is a dry run then we are finished\n if dry_run:\n return message\n\n # push the header to the VOSpace\n mopheader_filename = expname+\".mopheader\"\n destination = storage.dbimages_uri(expnum, ccd, prefix=prefix, version=version, ext='mopheader')\n source = mopheader_filename\n count = 0\n with open(source, 'r'):\n while True:\n try:\n count += 1\n logging.info(\"Attempt {} to copy {} -> {}\".format(count, source, destination))\n storage.copy(source, destination)\n break\n except Exception as ex:\n if count > 10:\n raise ex\n logging.info(message)\n except CalledProcessError as cpe:\n message = str(cpe.output)\n logging.error(message)\n except Exception as e:\n message = str(e)\n logging.error(message)\n\n if not dry_run:\n storage.set_status(task, prefix, expnum, version=version, ccd=ccd, status=message)\n \n return message"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef htmIndex(ra,dec,htm_level=3):\n import re\n if os.uname()[0] == \"Linux\": javabin = '/opt/java2/bin/java '\n htm_level = htm_level\n verc_htm_cmd = javabin+'-classpath /usr/cadc/misc/htm/htmIndex.jar edu.jhu.htm.app.lookup %s %s %s' % (htm_level, ra, dec)\n\n for result in os.popen( verc_htm_cmd ).readlines():\n result = result[:-1]\n if re.search(\"ID/Name cc\", result):\n (void, coord ) = result.split(\"=\")\n (void, junk, htm_index) = coord.split(\" \")\n return htm_index", "response": "Compute htm index of htm_level at position ra dec"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef circTOcutout(wcs,ra,dec,rad):\n (x1,y1)=wcs.rd2xy((ra+rad/2.0,dec-rad/2.0))\n (x2,y2)=wcs.rd2xy((ra-rad/2.0,dec+rad/2.0))\n xl=min(x1,x2)\n xr=max(x1,x2)\n yl=min(y1,y2)\n yu=max(y1,y2)\n \n ### constrain the cutout to be inside the image\n x1=max(xl,1)\n x1=int(min(x1,wcs.naxis1))\n \n x2=max(xr,x1,1)\n x2=int(min(x2,wcs.naxis1))\n \n y1=max(yl,1)\n y1=int(min(y1,wcs.naxis2))\n y2=max(yu,y1,1)\n y2=int(min(y2,wcs.naxis2))\n\n area=(x2-x1)*(y2-y1)\n cutout=\"[%d:%d,%d:%d]\" % ( x1,x2,y1,y2)\n\n if not y1 0:\n npts1 = len(pos1[:, 0])\n pos1_idx_array = numpy.arange(npts1, dtype=numpy.int16)\n\n if len(pos2) > 0:\n npts2 = len(pos2[:, 0])\n pos2_idx_array = numpy.arange(npts2, dtype=numpy.int16)\n\n # this is the array of final matched index, -1 indicates no match found.\n match1 = numpy.ma.zeros(npts1, dtype=numpy.int16)\n match1.mask = True\n\n # this is the array of matches in pos2, -1 indicates no match found.\n match2 = numpy.ma.zeros(npts2, dtype=numpy.int16)\n match2.mask = True\n\n # if one of the two input arrays are zero length then there is no matching to do.\n if npts1 * npts2 == 0:\n return match1, match2\n\n for idx1 in range(npts1):\n\n # compute the distance source idx1 to each member of pos2\n if not spherical :\n sep = numpy.sqrt((pos2[:, 0] - pos1[idx1, 0]) ** 2 + (pos2[:, 1] - pos1[idx1, 1]) ** 2)\n else:\n sep = numpy.sqrt((numpy.cos(numpy.radians(pos1[idx1,1]))*(pos2[:, 0] - pos1[idx1, 0])) ** 2 + (pos2[:, 1] - pos1[idx1, 1]) ** 2)\n \n\n # considered a match if sep is below tolerance and is the closest match available.\n match_condition = numpy.all((sep <= tolerance, sep == sep.min()), axis=0)\n\n # match_group_1 is list of the indexes of pos2 entries that qualified as possible matches to pos1[idx1]\n match_group_1 = pos2_idx_array[match_condition]\n\n # For each of those pos2 objects that could be a match to pos1[idx] find the best match in all of pos1\n for idx2 in match_group_1:\n # compute the distance from this pos2 object that is a possible match to pos1[idx1] to all members of pos1\n sep = numpy.sqrt((pos1[:, 0] - pos2[idx2, 0]) ** 2 + (pos1[:, 1] - pos2[idx2, 1]) ** 2)\n\n # considered a match if sep is below tolerance and is the closest match available.\n match_condition = numpy.all((sep <= tolerance, sep == sep.min()), axis=0)\n match_group_2 = pos1_idx_array[match_condition]\n\n # Are any of the pos1 members that were matches to the matched pos2 member the pos1[idx] entry?\n if idx1 in match_group_2:\n match1[idx1] = idx2\n match2[idx2] = idx1\n # this BREAK is in here since once we have a match we're done.\n break\n\n return match1, match2"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef stream(self):\n if self._stream is None:\n self._stream = tempfile.NamedTemporaryFile(delete=False)\n try:\n self._stream.write(self.client.open(self.filename, view='data').read())\n except:\n pass\n return self._stream", "response": "Returns a stream to write the log content."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef client(self):\n if self._client is not None:\n return self._client\n self._client = vospace.client\n return self._client", "response": "Return the client object for this instance."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse_string(self, timestr, subfmts):\n # Datetime components required for conversion to JD by ERFA, along\n # with the default values.\n components = ('year', 'mon', 'mday')\n defaults = (None, 1, 1, 0)\n # Assume that anything following \".\" on the right side is a\n # floating fraction of a second.\n try:\n idot = timestr.rindex('.')\n except:\n fracday = 0.0\n else:\n timestr, fracday = timestr[:idot], timestr[idot:]\n fracday = float(fracday)\n\n for _, strptime_fmt_or_regex, _ in subfmts:\n vals = []\n #print strptime_fmt_or_regex\n if isinstance(strptime_fmt_or_regex, six.string_types):\n try:\n #print timstr\n #print strptime_fmt_or_regex\n tm = time.strptime(timestr, strptime_fmt_or_regex)\n tm.tm_hour += int(24 * fracday)\n tm.tm_min += int(60 * (24 * fracday - tm.tm_hour))\n tm.tm_sec += 60 * (60 * (24 * fracday - tm.tm_hour) - tm.tm_min)\n except ValueError as ex:\n print ex\n continue\n else:\n vals = [getattr(tm, 'tm_' + component)\n for component in components]\n\n else:\n tm = re.match(strptime_fmt_or_regex, timestr)\n if tm is None:\n continue\n tm = tm.groupdict()\n vals = [int(tm.get(component, default)) for component, default\n in six.moves.zip(components, defaults)]\n\n hrprt = int(24 * fracday)\n vals.append(hrprt)\n mnprt = int(60 * (24 * fracday - hrprt))\n vals.append(mnprt)\n scprt = 60 * (60 * (24 * fracday - hrprt) - mnprt)\n vals.append(scprt)\n return vals\n else:\n raise ValueError('Time {0} does not match {1} format'\n .format(timestr, self.name))", "response": "Read time from a single string using a set of possible formats."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_db_options(args):\n import optik, getpass,sys\n from optik import OptionParser\n parser=OptionParser()\n parser.add_option(\"-d\",\"--database\",\n action=\"store\", type=\"string\", dest=\"database\",\n default=\"cfht\",\n help=\"Name of the SYBASE database containing TABLE\",\n metavar=\"FILE\")\n parser.add_option(\"-u\",\"--user\",\n action=\"store\", type=\"string\", dest=\"user\",\n default=getpass.getuser(),\n help=\"User name to access db with\",\n metavar=\"USER\")\n (opt, unused_args) = parser.parse_args(args)\n return opt.database,opt.user,unused_args", "response": "Parse through the command line arguments to over -ride the values\n in the users. dbrc file."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_db_password(dbSystem,db,user):\n import string, getpass, os\n dbrc = os.environ['HOME']+\"/.dbrc\"\n password={}\n if os.access(dbrc,os.R_OK):\n fd=open(dbrc)\n lines=fd.readlines()\n for line in lines:\n entry=line.split()\n if entry[0]==dbSystem and entry[1]==db and entry[2]==user:\n return entry[3]\n\n return getpass.getpass()", "response": "Read through the users. dbrc file to get the password for the db and user."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates a connection to the database specified on the command line", "response": "def _get_db_connect(dbSystem,db,user,password):\n \"\"\"Create a connection to the database specified on the command line\n \"\"\"\n if dbSystem=='SYBASE':\n import Sybase \n try:\n dbh = Sybase.connect(dbSystem,\n user,\n password,\n database=db )\n except:\n dbh=None\n elif dbSystem=='MYSQL':\n import MySQLdb\n try:\n dbh = MySQLdb.connect(user=user,\n passwd=password,\n db=db ,\n host='gimli')\n except:\n dbh=None\n \n return dbh"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef caom2(mpc_filename, search_date=\"2014 07 24.0\"):\n columns = ('Image',\n 'Ext',\n 'X',\n 'Y',\n 'MJD',\n 'Filter',\n 'Exptime',\n 'Object_RA',\n 'Object_Dec',\n 'Image_target',\n 'Telescope/Instrument',\n 'MetaData',\n 'Datalink')\n\n ephem_table = Table(names=columns,\n dtypes=('S10', 'i4', 'f8', 'f8',\n 'f8', 'S10', 'f8', 'f8', 'f8', 'S20', 'S20', 'S20', 'S50'))\n\n ephem_table.pprint()\n\n o = orbfit.Orbfit(mpc.MPCReader(mpc_filename).mpc_observations)\n o.predict(search_date)\n fields = storage.cone_search(o.coordinate.ra.degrees, o.coordinate.dec.degrees, dra=0.3, ddec=0.3,\n calibration_level=1)\n mjdates = numpy.unique(fields['mjdate'])\n\n collectionIDs = []\n for mjdate in mjdates:\n jd = 2400000.5 + mjdate\n o.predict(jd)\n for field in storage.cone_search(o.coordinate.ra.degrees, o.coordinate.dec.degrees,\n dra=30./3600.0, ddec=30./3600.0,\n mjdate=mjdate, calibration_level=1):\n collectionIDs.append(field['collectionID'])\n\n expnums = numpy.unique(numpy.array(collectionIDs))\n\n for expnum in expnums:\n header = storage.get_astheader(expnum, 22)\n o.predict(header['MJDATE']+2400000.5)\n print o.time.iso, o.coordinate.ra.degrees, o.coordinate.dec.degrees\n for ccd in range(36):\n header = storage.get_astheader(expnum, ccd)\n w = wcs.WCS(header)\n (x, y) = w.sky2xy(o.coordinate.ra.degrees, o.coordinate.dec.degrees)\n print ccd, x, y\n if 0 < x < header['NAXIS1'] and 0 < y < header['NAXIS2']:\n ephem_table.add_row([expnum, ccd+1, x, y,\n header['MJDATE'], header['FILTER'], header['EXPTIME'],\n o.coordinate.ra.degrees, o.coordinate.dec.degrees,\n header['OBJECT'],\n 'CFHT/MegaCam',\n None,\n \"http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/pub/CFHT/{}p[{}]\".format(expnum, ccd)])\n break\n\n ephem_table.pprint()\n ephem_table.write('backdoor.tsv', format='ascii', delimiter='\\t')", "response": "Builds a sequence of TSVs from a CADC file."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef download_cutout(self, reading, focus=None, needs_apcor=False):\n logger.debug(\"Doing download_cutout with inputs: reading:{} focus:{} needs_apcor:{}\".format(reading,\n focus,\n needs_apcor))\n assert isinstance(reading, SourceReading)\n\n min_radius = config.read('CUTOUTS.SINGLETS.RADIUS')\n if not isinstance(min_radius, Quantity):\n min_radius = min_radius * units.arcsec\n\n radius = max(reading.uncertainty_ellipse.a,\n reading.uncertainty_ellipse.b) * 2.5 + min_radius\n\n logger.debug(\"got radius for cutout: {}\".format(radius))\n image_uri = reading.get_image_uri()\n logger.debug(\"Getting cutout at {} for {}\".format(reading.reference_sky_coord, image_uri))\n hdulist = storage._cutout_expnum(reading.obs,\n reading.reference_sky_coord, radius)\n # hdulist = storage.ra_dec_cutout(image_uri, reading.reference_sky_coord, radius)\n logger.debug(\"Getting the aperture correction.\")\n source = SourceCutout(reading, hdulist, radius=radius)\n # Accessing the attribute here to trigger the download.\n try:\n apcor = source.apcor\n zmag = source.zmag\n source.reading.get_observation_header()\n except Exception as ex:\n if needs_apcor:\n import sys, traceback\n sys.stderr.write(\"Failed to retrieve apcor but apcor required. Raising error, see logs for more details\")\n sys.stderr.write(traceback.print_exc())\n pass\n logger.debug(\"Sending back the source reading.\")\n return source", "response": "Downloads a cutout of the FITS image for a given source reading."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef load_objects(self, directory_name=None):\n # for name in Neptune:\n # self.kbos[name] = Neptune[name]\n\n if directory_name is not None:\n # defaults to looking at .ast files only\n if directory_name == parameters.REAL_KBO_AST_DIR:\n kbos = parsers.ossos_discoveries(all_objects=True, data_release=None)\n else:\n kbos = parsers.ossos_discoveries(directory_name, all_objects=False, data_release=None)\n\n for kbo in kbos:\n # if kbo.orbit.arc_length > 30.: # cull the short ones for now\n self.kbos[kbo.name] = kbo.orbit\n self.kbos[kbo.name].mag = kbo.mag\n # else:\n # print(\"Arc very short, large uncertainty. Skipping {} for now.\\n\".format(kbo.name))\n\n self.doplot()", "response": "Load the targets from a file."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef p2c(self, p=None):\n if p is None:\n p = [0, 0]\n x = (p[0] - self.x1) * self.xscale + self.cx1\n y = (p[1] - self.y1) * self.yscale + self.cy1\n # logging.debug(\"p2c: ({},{}) -> ({},{})\".format(p[0],p[1], x, y))\n\n return (x, y)", "response": "convert from plot to canvas coordinates."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef p2s(self, p=None):\n if not p: p = [0, 0]\n\n s = self.p2c(p)\n return self.c2s(s)", "response": "Convert from plot to screen coordinates"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef c2s(self, p=None):\n if not p:\n p = [0, 0]\n\n return p[0] - self.canvasx(self.cx1), p[1] - self.canvasy(self.cy2)", "response": "Convert from canvas to screen coordinates"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef centre(self):\n return self.p2s(self.canvasx(self.width / 2.0)), self.p2s(self.canvasy(self.height / 2.0))", "response": "Return the RA and DEC of the center of the visible Canvas."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndraw a grid of the current RA and DEC on Canvas.", "response": "def coord_grid(self):\n \"\"\"Draw a grid of RA/DEC on Canvas.\"\"\"\n\n ra2 = math.pi * 2\n ra1 = 0\n dec1 = -1 * math.pi / 2.0\n dec2 = math.pi / 2.0\n\n dra = math.fabs((self.x2 - self.x1) / 4.0)\n ddec = math.fabs((self.y2 - self.y1) / 4.0)\n logging.debug(\"Drawing the grid.\")\n label = True\n ra = ra1\n while ra <= ra2:\n # create a line that goes from dec1 to dec2 at this ra\n (cx1, cy1) = self.p2c((ra, dec1))\n # ly is the y location mid-way between the top and bottom of the plot\n (cx2, cy2) = self.p2c((ra, dec2))\n # create a dashed line for un-labeled lines\n self.create_line(cx1, cy1, cx2, cy2, dash=(20, 20), fill=\"grey\")\n dec = dec1\n while dec <= dec2:\n if label:\n # lx is the the x screen coordinate location of the ra\n (lx, ly) = self.p2c((ra, dec + ddec / 2.0))\n self.create_text(lx + 5, ly, justify=LEFT,\n font=('Times', '-20'),\n text=\"\\n\".join(str(ephem.hours(ra))[:-3]), fill='grey')\n label = not label\n dec += ddec\n ra += dra\n\n # plot dec grid lines bewtween South (dec1) and North (dec2) limits of plot\n dec = dec1\n # should we label the current grid line?\n label = True\n while dec <= dec2:\n # create a line the goes from ra1 to ra2 at this dec\n (cx1, cy1) = self.p2c((ra1, dec))\n (cx2, cy2) = self.p2c((ra2, dec))\n self.create_line(cx1, cy1, cx2, cy2, dash=(20, 20), fill='grey')\n ra = ra1\n while ra <= ra2:\n if label:\n # lx/ly are the screen coordinates of the label\n (lx, ly) = self.p2c((ra + dra / 2.0, dec))\n self.create_text(lx, ly - 5, font=('Times', '-20'),\n text=str(ephem.degrees(dec)), fill='grey')\n ra += dra\n dec += ddec\n label = not label"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndrawing a line of size and orientation at x y", "response": "def tickmark(self, x, y, size=10, orientation=90):\n \"\"\"Draw a line of size and orientation at x,y\"\"\"\n\n (x1, y1) = self.p2c([x, y])\n x2 = x1 + size * math.cos(math.radians(orientation))\n y2 = y1 - size * math.sin(math.radians(orientation))\n self.create_line(x1, y1, x2, y2)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nset the coordinate boundaries of the plot", "response": "def limits(self, x1, x2, y1, y2):\n \"\"\"Set the coordinate boundaries of plot\"\"\"\n\n self.x1 = x1\n self.x2 = x2\n self.y1 = y1\n self.y2 = y2\n self.xscale = (self.cx2 - self.cx1) / (self.x2 - self.x1)\n self.yscale = (self.cy2 - self.cy1) / (self.y2 - self.y1)\n\n # Determine the limits of the canvas\n (cx1, cy1) = self.p2c((0, -math.pi / 2.0))\n (cx2, cy2) = self.p2c((2 * math.pi, math.pi / 2.0))\n # # set the scroll region to the size of the camvas plus a boundary to allow the canvas edge to be at centre\n self.config(scrollregion=(\n cx2 - self.width / 2.0, cy2 - self.height / 2.0, cx1 + self.width / 2.0, cy1 + self.height / 2.0))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef reset(self):\n\n sun = ephem.Sun()\n this_time = Time(self.date.get(), scale='utc')\n sun.compute(this_time.iso)\n self.sun = Coord((sun.ra, sun.dec))\n\n self.doplot()\n self.plot_pointings()", "response": "Expand to the full scale"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef updateObj(self, event):\n\n name = self.objList.get(\"active\")\n self.SearchVar.set(name)\n self.object_info.set(str(self.kbos.get(name, '')))\n return", "response": "Update the search box with the current object"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef relocate(self):\n\n name = self.SearchVar.get()\n if self.kbos.has_key(name):\n kbo = self.kbos[name]\n assert isinstance(kbo, orbfit.Orbfit)\n this_time = Time(self.date.get(), scale='utc')\n try:\n kbo.predict(this_time)\n self.recenter(kbo.coordinate.ra.radian, kbo.coordinate.dec.radian)\n self.create_point(kbo.coordinate.ra.radian, kbo.coordinate.dec.radian, color='blue', size=4)\n except:\n logging.error(\"failed to compute KBO position\")", "response": "Move to the position of the KBO."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef zoom(self, event=None, scale=2.0):\n\n # compute the x,y of the center of the screen\n sx1 = (self.cx1 + self.cx2) / 2.0\n sy1 = (self.cy1 + self.cy2) / 2.0\n\n if not event is None:\n logging.debug(\"EVENT: {},{}\".format(event.x, event.y))\n sx1 = event.x\n sy1 = event.y\n\n (x, y) = self.c2p((self.canvasx(sx1), self.canvasy(sy1)))\n xw = (self.x2 - self.x1) / 2.0 / scale\n yw = (self.y2 - self.y1) / 2.0 / scale\n\n # reset the limits to be centered at x,y with\n # area of xw*2,y2*2\n self.limits(x - xw, x + xw, y - yw, y + yw)\n\n self.delete(ALL)\n self.doplot()", "response": "Zoom in the area of the current area."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a circle of size at this x y location", "response": "def create_point(self, xcen, ycen, size=10, color='red', fill=None):\n \"\"\"Plot a circle of size at this x,y location\"\"\"\n\n if fill is None:\n fill = color\n\n (x, y) = self.p2c((xcen, ycen))\n x1 = x - size\n x2 = x + size\n y1 = y - size\n y2 = y + size\n self.create_rectangle(x1, y1, x2, y2, fill=fill, outline=color)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting the color of the currently selected pointing to black", "response": "def current_pointing(self, index):\n \"\"\"set the color of the currently selected pointing to 'blue'\"\"\"\n if self.current is not None:\n for item in self.pointings[self.current]['items']:\n self.itemconfigure(item, outline=\"black\")\n self.current = index\n for item in self.pointings[self.current]['items']:\n self.itemconfigure(item, outline=\"blue\")"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef delete_pointing(self, event):\n\n if self.current is None:\n return\n for item in self.pointings[self.current]['items']:\n self.delete(item)\n self.delete(self.pointings[self.current]['label']['id'])\n del (self.pointings[self.current])\n\n self.current = None", "response": "Delete the currently active pointing"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load_pointings(self, filename=None):\n\n filename = ( filename is None and tkFileDialog.askopenfilename() or filename)\n\n if filename is None:\n return\n\n f = storage.open_vos_or_local(filename)\n lines = f.readlines()\n f.close()\n points = []\n if lines[0][0:5] == \" 0 ):\n points.append((d[0], d[1], d[2]))\n else:\n points.append(('', math.radians(float(d[1])), math.radians(float(d[2]))))\n elif len(d) == 8:\n line = \"%s %s:%s:%s %s:%s:%s %s\" % (d[0], d[1], d[2], d[3], d[4], d[5], d[6], d[7] )\n d = line.split()\n # this one seems unfinished...no append\n else:\n sys.stderr.write(\"Don't understand pointing format\\n%s\\n\" % ( line))\n continue\n\n self.plot_points_list(points)\n return", "response": "Load some pointings from a file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating the pointing area for the current camera.", "response": "def create_pointing(self, event, label_text=None):\n \"\"\"Plot the sky coverage of pointing at event.x,event.y on the canvas.\n\n \"\"\"\n x = self.canvasx(event.x)\n y = self.canvasy(event.y)\n (ra, dec) = self.c2p((x, y))\n this_camera = Camera(ra=float(ra) * units.radian, dec=float(dec)*units.radian, camera=self.camera.get())\n\n ccds = numpy.radians(numpy.array(this_camera.geometry))\n items = []\n for ccd in ccds:\n if len(ccd) == 4:\n (x1, y1) = self.p2c((ccd[0], ccd[1]))\n (x2, y2) = self.p2c((ccd[2], ccd[3]))\n item = self.create_rectangle(x1, y1, x2, y2, stipple='gray25', fill=None)\n else:\n (x1, y1) = self.p2c((ccd[0] - ccd[2], ccd[1] - ccd[2]))\n (x2, y2) = self.p2c((ccd[0] + ccd[2], ccd[1] + ccd[2]))\n item = self.create_oval(x1, y1, x2, y2)\n items.append(item)\n label = {}\n if label_text is None:\n label_text = self.plabel.get()\n label['text'] = label_text\n label['id'] = self.label(this_camera.ra.radian, this_camera.dec.radian, label['text'])\n self.pointings.append({\n \"label\": label,\n \"items\": items,\n \"camera\": this_camera})\n self.current = len(self.pointings) - 1\n self.current_pointing(len(self.pointings) - 1)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef plot_pointings(self, pointings=None):\n\n if pointings is None:\n pointings = self.pointings\n\n i = 0\n for pointing in pointings:\n items = []\n i = i + 1\n label = {}\n label['text'] = pointing['label']['text']\n for ccd in numpy.radians(pointing[\"camera\"].geometry):\n if len(ccd) == 4:\n ccd = numpy.radians(numpy.array(ccd))\n (x1, y1) = self.p2c((ccd[0], ccd[1]))\n (x2, y2) = self.p2c((ccd[2], ccd[3]))\n item = self.create_rectangle(x1, y1, x2, y2, stipple='gray25', fill=pointing.get('color', ''))\n else:\n (x1, y1) = self.p2c((ccd[0] - ccd[2]), ccd[1] - ccd[2])\n (x2, y2) = self.p2c((ccd[0] + ccd[2]), ccd[1] + ccd[2])\n item = self.create_oval(x1, y1, x2, y2)\n items.append(item)\n if self.show_labels.get() == 1:\n label['id'] = self.label(pointing[\"camera\"].ra.radian, pointing[\"camera\"].dec.radian, label['text'])\n pointing[\"items\"] = items\n pointing[\"label\"] = label", "response": "Plot pointings on canavs"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nlets the label of the current pointing to the value in the plabel box", "response": "def set_pointing_label(self):\n \"\"\"Let the label of the current pointing to the value in the plabel box\"\"\"\n\n current = self.current\n pointing = self.pointings[current]\n self.delete_pointing(None)\n pointing[\"label\"]['text'] = self.plabel.get()\n self.pointings.append(pointing)\n self.plot_pointings([pointing])\n self.current = current"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nmoves the nearest pointing to event. x event. y and with cursor.", "response": "def move_pointing(self, event):\n \"\"\"Grab nearest pointing to event.x,event.y and with cursor\"\"\"\n\n (ra, dec) = self.c2p((self.canvasx(event.x),\n self.canvasy(event.y)))\n closest = None\n this_pointing = None\n this_index = -1\n index = -1\n for pointing in self.pointings:\n index = index + 1\n # Find the camera we clicked closest too\n ds = pointing[\"camera\"].separation(ra, dec)\n if this_pointing is None or ds < closest:\n this_index = index\n closest = ds\n this_pointing = pointing\n if this_pointing is None:\n return\n self.plabel.set(this_pointing['label']['text'])\n this_pointing[\"camera\"].set_coord((ra*units.radian, dec*units.radian))\n ccds = numpy.radians(this_pointing[\"camera\"].geometry)\n items = this_pointing[\"items\"]\n label = this_pointing[\"label\"]\n (x1, y1) = self.p2c((this_pointing[\"camera\"].ra.radian, this_pointing[\"camera\"].dec.radian))\n self.coords(label[\"id\"], x1, y1)\n for i in range(len(ccds)):\n ccd = ccds[i]\n item = items[i]\n if len(ccd) == 4:\n (x1, y1) = self.p2c((ccd[0], ccd[1]))\n (x2, y2) = self.p2c((ccd[2], ccd[3]))\n else:\n (x1, y1) = self.p2c((ccd[0] - ccd[2]), ccd[1] - ccd[2])\n (x2, y2) = self.p2c((ccd[0] + ccd[2]), ccd[1] + ccd[2])\n self.coords(item, x1, y1, x2, y2)\n self.current_pointing(this_index)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nplot an OSSOS observation on the OSSOS plot.", "response": "def ossos_pointings(self):\n \"\"\"\n plot an OSSOS observation on the OSSOS plot.\n \"\"\"\n match = re.match('(\\d+)\\D(\\d+)', self.expnum.get())\n if match is not None:\n expnum = int(match.group(1))\n ccd = int(match.group(2))\n x = 2112 / 2.0\n y = 4644 / 2.0\n else:\n expnum = int(str(self.expnum.get()))\n ccd = 22\n x = 1000\n y = 4644 - 15 / 0.185\n header = None\n try:\n header = storage.get_astheader(expnum, ccd=ccd)\n except:\n if header is None:\n print \"Didn't get a header... \"\n return\n\n ossos_wcs = wcs.WCS(header)\n (ra, dec) = ossos_wcs.xy2sky(x, y)\n\n class MyEvent(object):\n def __init__(self, x, y):\n self.x = x\n self.y = y\n\n (x, y) = self.p2s((math.radians(ra), math.radians(dec)))\n event = MyEvent(x, y)\n self.create_pointing(event, label_text=header['OBJECT'] + ' ccd{}'.format(ccd))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nretrieves the MEGACAM pointings that overlap with the current FOV and plot.", "response": "def get_pointings(self):\n \"\"\"\n Retrieve the MEGACAM pointings that overlap with the current FOV and plot.\n @return: None\n \"\"\"\n\n self.camera.set(\"MEGACAM_40\")\n (ra1, dec1) = self.c2p((self.canvasx(1), self.canvasy(1)))\n (ra2, dec2) = self.c2p((self.canvasx(480 * 2), self.canvasy(360 * 2)))\n ra_cen = math.degrees((ra2 + ra1) / 2.0)\n dec_cen = math.degrees((dec2 + dec1) / 2.0)\n # width = math.degrees(math.fabs(ra1 - ra2))\n width = 180\n # height = math.degrees(math.fabs(dec2 - dec1))\n height = 90\n date = mpc.Time(self.date.get(), scale='utc').iso\n table = cadc.cfht_megacam_tap_query(ra_cen, dec_cen, width, height, date=date)\n\n for row in table:\n ra = row['RAJ2000']\n dec = row['DEJ2000']\n (x, y) = self.p2s((math.radians(ra), math.radians(dec)))\n event = MyEvent(x, y)\n\n self.create_pointing(event, label_text=\"\")"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef save_pointings(self):\n i = 0\n if self.pointing_format.get() in ['GEMINI ET', 'CFHT ET', 'CFHT API']:\n logging.info('Beginning table pointing save.')\n for pointing in self.pointings:\n name = pointing[\"label\"][\"text\"]\n camera = pointing[\"camera\"]\n ccds = numpy.radians(camera.geometry)\n polygons = []\n for ccd in ccds:\n polygon = Polygon.Polygon(((ccd[0], ccd[1]),\n (ccd[0], ccd[3]),\n (ccd[2], ccd[3]),\n (ccd[2], ccd[1]),\n (ccd[0], ccd[1])))\n polygons.append(polygon)\n et = EphemTarget(name, ephem_format=self.pointing_format.get())\n # determine the mean motion of target KBOs in this field.\n field_kbos = []\n center_ra = 0\n center_dec = 0\n\n pointing_date = mpc.Time(self.date.get(), scale='utc')\n start_date = mpc.Time(self.date.get(), scale='utc') - TimeDelta(8.1*units.day)\n end_date = start_date + TimeDelta(17*units.day)\n time_step = TimeDelta(3.0*units.hour)\n\n # Compute the mean position of KBOs in the field on current date.\n for kbo_name, kbo in self.kbos.items():\n if kbo_name in Neptune or kbo_name in tracking_termination:\n print 'skipping', kbo_name\n continue\n kbo.predict(pointing_date)\n ra = kbo.coordinate.ra\n dec = kbo.coordinate.dec\n if kbo_name in name:\n print \"{} matches pointing {} by name, adding to field.\".format(kbo_name, name)\n field_kbos.append(kbo)\n center_ra += ra.radian\n center_dec += dec.radian\n else:\n for polygon in polygons:\n if polygon.isInside(ra.radian, dec.radian):\n print \"{} inside pointing {} polygon, adding to field.\".format(kbo_name, name)\n field_kbos.append(kbo)\n center_ra += ra.radian\n center_dec += dec.radian\n\n # logging.critical(\"KBOs in field {0}: {1}\".format(name, ', '.join([n.name for n in field_kbos])))\n\n today = start_date\n while today < end_date:\n today += time_step\n mean_motion = (0, 0)\n max_mag = 0.0\n if len(field_kbos) > 0:\n current_ra = 0\n current_dec = 0\n for kbo in field_kbos:\n kbo.predict(today)\n max_mag = max(max_mag, kbo.mag)\n current_ra += kbo.coordinate.ra.radian\n current_dec += kbo.coordinate.dec.radian\n mean_motion = ((current_ra - center_ra) / len(field_kbos),\n (current_dec - center_dec) / len(field_kbos))\n ra = pointing['camera'].coordinate.ra.radian + mean_motion[0]\n dec = pointing['camera'].coordinate.dec.radian + mean_motion[1]\n cc = SkyCoord(ra=ra,\n dec=dec,\n unit=(units.radian, units.radian),\n obstime=today)\n dt = pointing_date - today\n cc.dra = (mean_motion[0] * units.radian / dt.to(units.hour)).to(units.arcsec/units.hour).value*math.cos(dec)\n cc.ddec = (mean_motion[1] * units.radian / dt.to(units.hour)).to(units.arcsec/units.hour).value\n cc.mag = max_mag\n et.append(cc)\n\n et.save()\n return\n\n f = tkFileDialog.asksaveasfile()\n if self.pointing_format.get() == 'Subaru':\n for pointing in self.pointings:\n (sra, sdec) = str(pointing[\"camera\"]).split()\n ra = sra.replace(\":\", \"\")\n dec = sdec.replace(\":\", \"\")\n name = pointing[\"label\"][\"text\"]\n f.write(\"\"\"{}=OBJECT=\"{}\" RA={} DEC={} EQUINOX=2000.0 INSROT_PA=90\\n\"\"\".format(name,\n name,\n ra,\n dec))\n return\n if self.pointing_format.get() == 'CFHT PH':\n f.write(\"\"\"\n\n\n
\nFixed Targets\nFixed Targets for CFHT QSO\n\n\n Name of target\n\n\n Right ascension of target\n \n\n Declination of target\n \n\n Epoch of coordinates\n \n\nPointing name\n \n \n= 0:\n dec[0] = '+' + dec[0]\n if self.pointing_format.get() == 'Palomar':\n f.write(\"%5d %16s %2s %2s %4s %3s %2s %4s 2000\\n\" % (i, name,\n ra[0].zfill(2),\n ra[1].zfill(2),\n ra[2].zfill(2),\n dec[0].zfill(3),\n dec[1].zfill(2),\n dec[2].zfill(2)))\n elif self.pointing_format.get() == 'CFHT PH':\n # f.write(\"%f %f\\n\" % (pointing[\"camera\"].ra,pointing[\"camera\"].dec))\n f.write(\"%-20s|%11s|%11s|%6.1f|%-5d|\\n\" % (name, sra, sdec, 2000.0, 1))\n elif self.pointing_format.get() == 'KPNO/CTIO':\n str1 = sra.replace(\":\", \" \")\n str2 = sdec.replace(\":\", \" \")\n f.write(\"%16s %16s %16s 2000\\n\" % ( name, str1, str2))\n elif self.pointing_format.get() == 'SSim':\n ra = []\n dec = []\n for ccd in pointing[\"camera\"].geometry:\n ra.append(ccd[0])\n ra.append(ccd[2])\n dec.append(ccd[1])\n dec.append(ccd[3])\n\n dra = math.degrees(math.fabs(max(ra) - min(ra)))\n ddec = math.degrees(math.fabs(max(dec) - min(dec)))\n f.write(\"%f %f %16s %16s DATE 1.00 1.00 500 FILE\\n\" % (dra, ddec, sra, sdec ))\n if self.pointing_format.get() == 'CFHT PH':\n f.write(\"\"\"]]\n
\n\n\"\"\")\n f.close()", "response": "Print the currently defined FOVs and the mean position of target KBOs in this field."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nclearing the plot and redraw it.", "response": "def doplot(self):\n \"\"\"\n Clear the plot and then redraw it.\n\n \"\"\"\n w = self\n w.delete(ALL)\n w.coord_grid()\n w.objList.delete(0, END)\n self._plot()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndraw the actual plot.", "response": "def _plot(self):\n \"\"\"Draw the actual plot.\n \"\"\"\n\n w = self\n kbos = self.kbos\n re_string = w.FilterVar.get()\n vlist = []\n for name in kbos:\n if not re.search(re_string, name):\n continue\n vlist.append(name)\n fill = None\n is_colossos_target = False\n for cname in parameters.COLOSSOS:\n if cname in name:\n is_colossos_target = True\n print \"ColOSSOS: \", cname\n break\n is_terminated = False\n for cname in tracking_termination:\n if cname in name:\n is_terminated = True\n print \"Terminated\", cname\n break\n is_double = False\n for cname in doubles:\n if cname in name:\n is_double = True\n print 'Needs double:', cname\n break\n if type(kbos[name]) == type(ephem.EllipticalBody()):\n try:\n kbos[name].compute(w.date.get())\n except Exception as e:\n logging.error(\"Failed to compute KBO position. {}\".format(name))\n continue\n ra = kbos[name].ra\n dec = kbos[name].dec\n a = math.radians(10.0 / 3600.0)\n b = a\n ang = 0.0\n point_size = 1\n yoffset = +10\n xoffset = +10\n elif isinstance(kbos[name], orbfit.Orbfit):\n yoffset = -10\n xoffset = -10\n kbo = kbos[name]\n pointing_date = mpc.Time(w.date.get(), scale='utc').jd\n trail_mid_point = 0\n for days in range(trail_mid_point * 2 + 1):\n point_size = days == trail_mid_point and 5 or 1\n today = mpc.Time(pointing_date - trail_mid_point + days, scale='utc', format='jd')\n kbo.predict(today, 568)\n ra = kbo.coordinate.ra.radian\n dec = kbo.coordinate.dec.radian\n from astropy import units\n a = kbo.dra.to(units.radian).value\n b = kbo.ddec.to(units.radian).value\n ang = kbo.pa.to(units.radian).value\n lost = False\n if a > math.radians(0.3):\n lost = True\n\n fill_colour = (lost and \"red\") or \"grey\"\n fill_colour = (is_double and \"magenta\") or fill_colour\n fill_colour = (is_colossos_target and \"blue\") or fill_colour\n point_colour = (is_terminated and \"red\") or \"black\"\n w.create_point(ra, dec, size=point_size, color=point_colour, fill=fill_colour)\n if w.show_ellipse.get() == 1 and days == trail_mid_point:\n if a < math.radians(5.0):\n w.create_ellipse(ra,\n dec,\n a,\n b,\n ang)\n if w.show_labels.get() == 1:\n w.label(ra, dec, name, offset=[xoffset, yoffset])\n else:\n ra = kbos[name]['RA']\n dec = kbos[name]['DEC']\n w.create_point(ra, dec, size=4, color='cyan')\n w.label(ra, dec, name[-2:], offset=[-15, +15]) # truncate object name for plot clutter clarity\n\n vlist.sort()\n for v in vlist:\n w.objList.insert(END, v)\n w.plot_pointings()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef planted(fk_candidate_observations, planted_objects, tolerance=10):\n\n found_pos = []\n detections = fk_candidate_observations.get_sources()\n for detection in detections:\n reading = detection.get_reading(0)\n # create a list of positions, to be used later by match_lists\n found_pos.append([reading.x, reading.y])\n\n # The match_list method expects a list that contains a position, not an x and a y vector, so we transpose.\n planted_objects_table = planted_objects.table\n planted_pos = numpy.transpose([planted_objects_table['x'].data, planted_objects_table['y'].data])\n\n # match_idx is an order list. The list is in the order of the first list of positions and each entry\n # is the index of the matching position from the second list.\n (match_idx, match_fnd) = util.match_lists(numpy.array(planted_pos), numpy.array(found_pos), tolerance=tolerance)\n return match_fnd, match_idx", "response": "This function takes a fk_candidate_observations and matches the Object. planted file with found sources."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconvert Modfied Juian Date to GMST.", "response": "def mjd2gmst(mjd):\n \"\"\"Convert Modfied Juian Date (JD = 2400000.5) to GMST\n \n Take from P.T. Walace routines\"\"\"\n \n D2PI= 6.2831853071795864769252867665590057683943387987502\n DS2R= 7.2722052166430399038487115353692196393452995355905e-5\n \n tu = (mjd-51544.5)/36525.0;\n \n st = fmod(mjd,1.0) * D2PI + ( 24110.54841 + ( 8640184.812866 + ( 0.093104 - 6.2e-6 * tu ) * tu ) * tu ) * DS2R \n \n w = fmod(st,D2PI)\n if w >= 0.0 :\n return w\n else:\n return w+D2PI"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef ec2eq(self):\n\n import math\n #from numpy.matlib import sin, cos, arcsin, arctan2\n from math import sin, cos\n from math import asin as arcsin\n from math import atan2 as arctan2\n from math import acos as arccos\n \n eb=self.eb\n el=self.el\n ob=math.radians(23.439281)\n \n dec = arcsin(sin(eb)*cos(ob)+cos(eb)*sin(ob)*sin(el))\n sra = (sin(dec)*cos(ob)-sin(eb))/(cos(dec)*sin(ob))\n\n cra = cos(el)*cos(eb)/cos(dec)\n if sra < 1 and sra > -1 :\n sa= arcsin(sra)\n else:\n sa = 0\n ca= arccos(cra)\n tsa=sa\n tca=ca\n if tsa<0 :\n ca=2.0*math.pi-ca\n if tca>=math.pi/2.0:\n sa=math.pi-sa\n if ca >= math.pi*2.0:\n ca=ca-math.pi*2.0\n self.tsa=sra\n self.tca=cra\n self.ra=ca\n self.dec=dec", "response": "Convert ecliptic coordinates to equatorial coordinates"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nplots the ecliptic plane line on the given axes", "response": "def plot_line(axes, fname, ltype):\n \"\"\"plot the ecliptic plane line on the given axes.\"\"\"\n x = np.genfromtxt(fname, unpack=True)\n axes.plot(x[0], x[1], ltype)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating the apparent magnitude of a TNO given its absolute magnitude H and a given distance d.", "response": "def apmag_at_absmag(H, d, phi=1):\n \"\"\"\n Calculate the apparent magnitude of a TNO given its absolute magnitude H, for a given distance.\n\n :param H: TNO absolute magnitude (unitless)\n :param d: barycentric distance (AU)\n :param phi: phase angle (0-1, always v close to 1 for TNOs)\n :return: apparent magnitude of TNO\n \"\"\"\n d_observer = 1. # 1 AU\n # approximate object's distance d_heliocentric and d_geocentric as the same, d, because TNO\n m_r = H + 2.5 * math.log10((d ** 4) / (phi * d_observer ** 4))\n print(\"m_r = {:2.2f} for a H = {} TNO at {} AU at opposition.\".format(\n m_r, H, d))\n return m_r"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef from_summary_line(cls, summaryLine, version=4, existing_object=None):\n ''' Summary format:\n object mag stdev dist ..E nobs time av_xres av_yres max_x max_y\n a ..E e ..E i ..E node ..E argperi ..E M ..E ra_dis dec_dis\n '''\n if not summaryLine:\n raise ValueError('No summary line given')\n if version == 4:\n params = summaryLine.split()\n if len(params) != 25:\n print params\n raise TypeError('Expected 25 columns, {0} given'.format(len(params)))\n input_params = params[0:1] + params[3:23]\n if not existing_object:\n retval = cls(*input_params)\n else:\n assert isinstance(existing_object, tno)\n assert existing_object.name == params[0]\n retval = existing_object\n retval.mean_mag = float(params[1])\n retval.mean_mag_stdev = float(params[2])\n retval.ra_discov = float(params[23])\n retval.dec_discov = float(params[24])\n else:\n raise VersionError('Unknown version \"{0}\"'.format(version))\n assert retval\n return retval", "response": "Create a new instance of the class from a summary line."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates a new instance of the class from a class line.", "response": "def from_class_line(cls, classLine, version=4, existing_object=None):\n '''\n Class format:\n class wrt n m security object mag stdev F H_sur dist ..E nobs time av_xres av_yres max_x max_y\n a ..E e ..E i ..E node ..E argperi ..E time_peri ..E rate\n '''\n if not classLine:\n raise ValueError('No class file line given.')\n if version == 4:\n params = classLine.split()\n if len(params) != 31:\n print params\n raise TypeError('Expected 31 columns, {0} given'.format(len(params)))\n input_params = [params[5]] + params[10:30] # the elements that are in common\n if not existing_object:\n # can later add more tests that the values match those in the existing object\n retval = cls(*input_params)\n else:\n assert isinstance(existing_object, tno)\n assert existing_object.name == params[5]\n retval = existing_object\n retval.classification = params[0]\n retval.wrt = params[1]\n retval.n = int(params[2])\n retval.m = int(params[3])\n retval.security = params[4]\n retval.mag_discov = float(params[6])\n retval.mag_discov_e = float(params[7])\n retval.filter = params[8]\n retval.H = float(\n params[9]) # H_r magnitude, using the average m_r and the discovery geometry (same error as m_r)\n retval.rate = float(params[30])\n elif version == 5:\n # Upgrade?\n # input_params =\n retval = cls(*input_params)\n else:\n raise VersionError('Unknown version \"{0}\"'.format(version))\n assert retval\n return retval"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef scci_cmd(host, userid, password, cmd, port=443, auth_method='basic',\n client_timeout=60, do_async=True, **kwargs):\n \"\"\"execute SCCI command\n\n This function calls SCCI server modules\n :param host: hostname or IP of iRMC\n :param userid: userid for iRMC with administrator privileges\n :param password: password for userid\n :param cmd: SCCI command\n :param port: port number of iRMC\n :param auth_method: irmc_username\n :param client_timeout: timeout for SCCI operations\n :param do_async: async call if True, sync call otherwise\n :returns: requests.Response from SCCI server\n :raises: SCCIInvalidInputError if port and/or auth_method params\n are invalid\n :raises: SCCIClientError if SCCI failed\n \"\"\"\n auth_obj = None\n try:\n protocol = {80: 'http', 443: 'https'}[port]\n auth_obj = {\n 'basic': requests.auth.HTTPBasicAuth(userid, password),\n 'digest': requests.auth.HTTPDigestAuth(userid, password)\n }[auth_method.lower()]\n\n except KeyError:\n raise SCCIInvalidInputError(\n (\"Invalid port %(port)d or \" +\n \"auth_method for method %(auth_method)s\") %\n {'port': port, 'auth_method': auth_method})\n\n try:\n header = {'Content-type': 'application/x-www-form-urlencoded'}\n if kwargs.get('upgrade_type') == 'irmc':\n with open(cmd, 'rb') as file:\n data = file.read()\n config_type = '/irmcupdate?flashSelect=255'\n elif kwargs.get('upgrade_type') == 'bios':\n with open(cmd, 'rb') as file:\n data = file.read()\n config_type = '/biosupdate'\n else:\n data = cmd\n config_type = '/config'\n r = requests.post(protocol + '://' + host + config_type,\n data=data,\n headers=header,\n verify=False,\n timeout=client_timeout,\n allow_redirects=False,\n auth=auth_obj)\n\n if not do_async:\n time.sleep(5)\n if DEBUG:\n print(cmd)\n print(r.text)\n print(\"do_async = %s\" % do_async)\n if r.status_code not in (200, 201):\n raise SCCIClientError(\n ('HTTP PROTOCOL ERROR, STATUS CODE = %s' %\n str(r.status_code)))\n\n result_xml = ET.fromstring(r.text)\n status = result_xml.find(\"./Value\")\n # severity = result_xml.find(\"./Severity\")\n error = result_xml.find(\"./Error\")\n message = result_xml.find(\"./Message\")\n if not int(status.text) == 0:\n raise SCCIClientError(\n ('SCCI PROTOCOL ERROR, STATUS CODE = %s, '\n 'ERROR = %s, MESSAGE = %s' %\n (str(status.text), error.text, message.text)))\n else:\n return r\n\n except IOError as input_error:\n raise SCCIClientError(input_error)\n\n except ET.ParseError as parse_error:\n raise SCCIClientError(parse_error)\n\n except requests.exceptions.RequestException as requests_exception:\n raise SCCIClientError(requests_exception)", "response": "execute SCCI command\n\n This function calls SCCI server modules\n :param host: hostname or IP of iRMC\n :param userid: userid for iRMC with administrator privileges\n :param password: password for userid\n :param cmd: SCCI command\n :param port: port number of iRMC\n :param auth_method: irmc_username\n :param client_timeout: timeout for SCCI operations\n :param do_async: async call if True, sync call otherwise\n :returns: requests.Response from SCCI server\n :raises: SCCIInvalidInputError if port and/or auth_method params\n are invalid\n :raises: SCCIClientError if SCCI failed"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_client(host, userid, password, port=443, auth_method='basic',\n client_timeout=60, **kwargs):\n \"\"\"get SCCI command partial function\n\n This function returns SCCI command partial function\n :param host: hostname or IP of iRMC\n :param userid: userid for iRMC with administrator privileges\n :param password: password for userid\n :param port: port number of iRMC\n :param auth_method: irmc_username\n :param client_timeout: timeout for SCCI operations\n :returns: scci_cmd partial function which takes a SCCI command param\n \"\"\"\n\n return functools.partial(scci_cmd, host, userid, password,\n port=port, auth_method=auth_method,\n client_timeout=client_timeout, **kwargs)", "response": "get SCCI command partial function"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_virtual_cd_set_params_cmd(remote_image_server,\n remote_image_user_domain,\n remote_image_share_type,\n remote_image_share_name,\n remote_image_deploy_iso,\n remote_image_username,\n remote_image_user_password):\n \"\"\"get Virtual CD Media Set Parameters Command\n\n This function returns Virtual CD Media Set Parameters Command\n :param remote_image_server: remote image server name or IP\n :param remote_image_user_domain: domain name of remote image server\n :param remote_image_share_type: share type of ShareType\n :param remote_image_share_name: share name\n :param remote_image_deploy_iso: deploy ISO image file name\n :param remote_image_username: username of remote image server\n :param remote_image_user_password: password of the username\n :returns: SCCI command\n \"\"\"\n\n cmd = _VIRTUAL_MEDIA_CD_SETTINGS % (\n remote_image_server,\n remote_image_user_domain,\n remote_image_share_type,\n remote_image_share_name,\n remote_image_deploy_iso,\n remote_image_username,\n remote_image_user_password)\n\n return cmd", "response": "get Virtual CD Media Set Parameters Command"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget Virtual FD Media Set Parameters Command", "response": "def get_virtual_fd_set_params_cmd(remote_image_server,\n remote_image_user_domain,\n remote_image_share_type,\n remote_image_share_name,\n remote_image_floppy_fat,\n remote_image_username,\n remote_image_user_password):\n \"\"\"get Virtual FD Media Set Parameters Command\n\n This function returns Virtual FD Media Set Parameters Command\n :param remote_image_server: remote image server name or IP\n :param remote_image_user_domain: domain name of remote image server\n :param remote_image_share_type: share type of ShareType\n :param remote_image_share_name: share name\n :param remote_image_deploy_iso: deploy ISO image file name\n :param remote_image_username: username of remote image server\n :param remote_image_user_password: password of the username\n :returns: SCCI command\n \"\"\"\n cmd = _VIRTUAL_MEDIA_FD_SETTINGS % (\n remote_image_server,\n remote_image_user_domain,\n remote_image_share_type,\n remote_image_share_name,\n remote_image_floppy_fat,\n remote_image_username,\n remote_image_user_password)\n\n return cmd"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_report(host, userid, password,\n port=443, auth_method='basic', client_timeout=60):\n \"\"\"get iRMC report\n\n This function returns iRMC report in XML format\n :param host: hostname or IP of iRMC\n :param userid: userid for iRMC with administrator privileges\n :param password: password for userid\n :param port: port number of iRMC\n :param auth_method: irmc_username\n :param client_timeout: timeout for SCCI operations\n :returns: root element of SCCI report\n :raises: ISCCIInvalidInputError if port and/or auth_method params\n are invalid\n :raises: SCCIClientError if SCCI failed\n \"\"\"\n\n auth_obj = None\n try:\n protocol = {80: 'http', 443: 'https'}[port]\n auth_obj = {\n 'basic': requests.auth.HTTPBasicAuth(userid, password),\n 'digest': requests.auth.HTTPDigestAuth(userid, password)\n }[auth_method.lower()]\n\n except KeyError:\n raise SCCIInvalidInputError(\n (\"Invalid port %(port)d or \" +\n \"auth_method for method %(auth_method)s\") %\n {'port': port, 'auth_method': auth_method})\n\n try:\n r = requests.get(protocol + '://' + host + '/report.xml',\n verify=False,\n timeout=(10, client_timeout),\n allow_redirects=False,\n auth=auth_obj)\n\n if r.status_code not in (200, 201):\n raise SCCIClientError(\n ('HTTP PROTOCOL ERROR, STATUS CODE = %s' %\n str(r.status_code)))\n\n root = ET.fromstring(r.text)\n return root\n\n except ET.ParseError as parse_error:\n raise SCCIClientError(parse_error)\n\n except requests.exceptions.RequestException as requests_exception:\n raise SCCIClientError(requests_exception)", "response": "get iRMC report in XML format"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets essential properties from the report element", "response": "def get_essential_properties(report, prop_keys):\n \"\"\"get essential properties\n\n This function returns a dictionary which contains keys as in\n prop_keys and its values from the report.\n\n :param report: SCCI report element\n :param prop_keys: a list of keys for essential properties\n :returns: a dictionary which contains keys as in\n prop_keys and its values.\n \"\"\"\n v = {}\n v['memory_mb'] = int(report.find('./System/Memory/Installed').text)\n v['local_gb'] = sum(\n [int(int(size.text) / 1024)\n for size in report.findall('.//PhysicalDrive/ConfigurableSize')])\n v['cpus'] = sum([int(cpu.find('./CoreNumber').text)\n for cpu in report.find('./System/Processor')\n if cpu.find('./CoreNumber') is not None])\n # v['cpus'] = sum([int(cpu.find('./LogicalCpuNumber').text)\n # for cpu in report.find('./System/Processor')])\n v['cpu_arch'] = 'x86_64'\n\n return {k: v[k] for k in prop_keys}"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_capabilities_properties(d_info,\n capa_keys,\n gpu_ids,\n fpga_ids=None,\n **kwargs):\n \"\"\"get capabilities properties\n\n This function returns a dictionary which contains keys\n and their values from the report.\n\n\n :param d_info: the dictionary of ipmitool parameters for accessing a node.\n :param capa_keys: a list of keys for additional capabilities properties.\n :param gpu_ids: the list of string contains /\n for GPU.\n :param fpga_ids: the list of string contains /\n for CPU FPGA.\n :param kwargs: additional arguments passed to scciclient.\n :returns: a dictionary which contains keys and their values.\n \"\"\"\n\n snmp_client = snmp.SNMPClient(d_info['irmc_address'],\n d_info['irmc_snmp_port'],\n d_info['irmc_snmp_version'],\n d_info['irmc_snmp_community'],\n d_info['irmc_snmp_security'])\n try:\n v = {}\n if 'rom_firmware_version' in capa_keys:\n v['rom_firmware_version'] = \\\n snmp.get_bios_firmware_version(snmp_client)\n\n if 'irmc_firmware_version' in capa_keys:\n v['irmc_firmware_version'] = \\\n snmp.get_irmc_firmware_version(snmp_client)\n\n if 'server_model' in capa_keys:\n v['server_model'] = snmp.get_server_model(snmp_client)\n\n # Sometime the server started but PCI device list building is\n # still in progress so system will response error. We have to wait\n # for some more seconds.\n if kwargs.get('sleep_flag', False) and \\\n any(k in capa_keys for k in ('pci_gpu_devices', 'cpu_fpga')):\n time.sleep(5)\n\n if 'pci_gpu_devices' in capa_keys:\n v['pci_gpu_devices'] = ipmi.get_pci_device(d_info, gpu_ids)\n\n if fpga_ids is not None and 'cpu_fpga' in capa_keys:\n v['cpu_fpga'] = ipmi.get_pci_device(d_info, fpga_ids)\n\n if 'trusted_boot' in capa_keys:\n v['trusted_boot'] = ipmi.get_tpm_status(d_info)\n\n return v\n except (snmp.SNMPFailure, ipmi.IPMIFailure) as err:\n raise SCCIClientError('Capabilities inspection failed: %s' % err)", "response": "get capabilities properties\n\n This function returns a dictionary which contains keys\n and their values from the report.\n\n\n :param d_info: the dictionary of ipmitool parameters for accessing a node.\n :param capa_keys: a list of keys for additional capabilities properties.\n :param gpu_ids: the list of string contains /\n for GPU.\n :param fpga_ids: the list of string contains /\n for CPU FPGA.\n :param kwargs: additional arguments passed to scciclient.\n :returns: a dictionary which contains keys and their values."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef process_session_status(irmc_info, session_timeout, upgrade_type):\n session_expiration = time.time() + session_timeout\n\n while time.time() < session_expiration:\n try:\n # Get session status to check\n session = get_firmware_upgrade_status(irmc_info, upgrade_type)\n except SCCIClientError:\n # Ignore checking during rebooted server\n time.sleep(10)\n continue\n\n status = session.find(\"./Value\").text\n severity = session.find(\"./Severity\").text\n message = session.find(\"./Message\").text\n result = {}\n\n if severity == 'Information' and status != '0':\n if 'FLASH successful' in message:\n result['upgrade_status'] = 'Complete'\n return result\n # Sleep a bit\n time.sleep(5)\n elif severity == 'Error':\n result['upgrade_status'] = 'Error'\n return result\n else:\n # Error occurred, get session log to see what happened\n session_log = message\n raise SCCIClientError('Failed to set firmware upgrade. '\n 'Session log is %s.' % session_log)\n\n else:\n raise SCCISessionTimeout('Failed to time out mechanism with %s.'\n % session_expiration)", "response": "process session status for Bios config backup or restore or RAID config operation"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngathering fgi status of the RAID configuration and return a dict of activity status and log drive number.", "response": "def get_raid_fgi_status(report):\n \"\"\"Gather fgi(foreground initialization) information of raid configuration\n\n This function returns a fgi status which contains activity status\n and its values from the report.\n\n :param report: SCCI report information\n :returns: dict of fgi status of logical_drives, such as Initializing (10%)\n or Idle. e.g: {'0': 'Idle', '1': 'Initializing (10%)'}\n :raises: SCCIInvalidInputError: fail report input.\n SCCIRAIDNotReady: waiting for RAID configuration to complete.\n \"\"\"\n fgi_status = {}\n raid_path = \"./Software/ServerView/ServerViewRaid\"\n\n if not report.find(raid_path):\n raise SCCIInvalidInputError(\n \"ServerView RAID not available in Bare metal Server\")\n if not report.find(raid_path + \"/amEMSV/System/Adapter/LogicalDrive\"):\n raise SCCIRAIDNotReady(\n \"RAID configuration not configure in Bare metal Server yet\")\n\n logical_drives = report.findall(raid_path +\n \"/amEMSV/System/Adapter/LogicalDrive\")\n for logical_drive_name in logical_drives:\n status = logical_drive_name.find(\"./Activity\").text\n name = logical_drive_name.find(\"./LogDriveNumber\").text\n fgi_status.update({name: status})\n\n return fgi_status"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_firmware_upgrade_status(irmc_info, upgrade_type):\n\n host = irmc_info.get('irmc_address')\n userid = irmc_info.get('irmc_username')\n password = irmc_info.get('irmc_password')\n port = irmc_info.get('irmc_port', 443)\n auth_method = irmc_info.get('irmc_auth_method', 'digest')\n client_timeout = irmc_info.get('irmc_client_timeout', 60)\n\n auth_obj = None\n try:\n protocol = {80: 'http', 443: 'https'}[port]\n auth_obj = {\n 'basic': requests.auth.HTTPBasicAuth(userid, password),\n 'digest': requests.auth.HTTPDigestAuth(userid, password)\n }[auth_method.lower()]\n except KeyError:\n raise SCCIInvalidInputError(\n (\"Invalid port %(port)d or \" +\n \"auth_method for method %(auth_method)s\") %\n {'port': port, 'auth_method': auth_method})\n try:\n if upgrade_type == 'bios':\n upgrade_type = '/biosprogress'\n elif upgrade_type == 'irmc':\n upgrade_type = '/irmcprogress'\n r = requests.get(protocol + '://' + host + upgrade_type,\n verify=False,\n timeout=(10, client_timeout),\n allow_redirects=False,\n auth=auth_obj)\n\n if r.status_code not in (200, 201):\n raise SCCIClientError(\n ('HTTP PROTOCOL ERROR, STATUS CODE = %s' %\n str(r.status_code)))\n\n upgrade_status_xml = ET.fromstring(r.text)\n return upgrade_status_xml\n except ET.ParseError as parse_error:\n raise SCCIClientError(parse_error)", "response": "get firmware upgrade status of bios or irmc"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndo it row by row to save memory...", "response": "def median(ma):\n \"\"\" do it row by row, to save memory....\"\"\"\n _median = 0*ma[0].filled(fill_value=0)\n for i in range(ma.shape[-1]):\n\tt=xmedian(ma[:,:,i])\n \t_median[:,i]=t\n\tt=None\n return _median"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngive a masked numpy array return the median value of the array.", "response": "def xmedian(ma):\n \"\"\" Given a masked numpy array (build using numpy.ma class) return\n the median value of the array.\"\"\"\n import numpy\n _medianIndex = numpy.floor(ma.count(axis=0)/2.0)\n _sortIndex = ma.argsort(kind='heapsort',axis=0)\n _median = ma[0].filled(fill_value=0)*0\n for idx in range(len(_sortIndex)):\n _median = _median+_sortIndex[idx]*(_medianIndex==idx)\n _medianIndex=_median\n _median=0*_median\n for idx in range(len(ma)):\n _median = _median + ma[idx].filled(fill_value=0)*(_medianIndex==idx)\n _sortIndex=None\n _medianIndex=None\n return _median"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _generate_provisional_name(q, astrom_header, fits_header):\n while True:\n ef = get_epoch_field(astrom_header, fits_header)\n epoch_field = ef[0] + ef[1]\n count = storage.increment_object_counter(storage.MEASURE3, epoch_field)\n try:\n q.put(ef[1] + count)\n except:\n break", "response": "Generates a provisional name for an object given the information in the astromaton header and FITS header."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntoggles autoplay mode when the user presses the keybind for toggling autoplay.", "response": "def on_toggle_autoplay_key(self):\n \"\"\"\n The user has pressed the keybind for toggling autoplay.\n \"\"\"\n if self.autoplay_manager.is_running():\n self.autoplay_manager.stop_autoplay()\n self.view.set_autoplay(False)\n else:\n self.autoplay_manager.start_autoplay()\n self.view.set_autoplay(True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ninitiate acceptance procedure, gathering required data. @param auto: Set on_accept to automatic measure of source?", "response": "def on_accept(self, auto=False):\n \"\"\"\n Initiates acceptance procedure, gathering required data.\n @param auto: Set on_accept to automatic measure of source?\n \"\"\"\n\n if self.model.is_current_source_named():\n provisional_name = self.model.get_current_source_name()\n else:\n provisional_name = self._generate_provisional_name()\n\n band = self.model.get_current_band()\n logger.debug(\"Got band {} and provisional_name {}\".format(band, provisional_name))\n default_comment = \"\"\n\n source_cutout = self.model.get_current_cutout()\n\n if not auto:\n result = self.view.ds9.get('imexam key coordinate wcs fk5 degrees')\n # result = display.get(\"\"\"imexam key coordinate $x $y $filename\"\"\")\n if not isinstance(result, str):\n print result\n result = str(result)\n values = result.split()\n ra = Quantity(float(values[1]), unit=units.degree)\n dec = Quantity(float(values[2]), unit=units.degree)\n key = values[0]\n (x, y, hdulist_index) = source_cutout.world2pix(ra, dec, usepv=False)\n source_cutout.update_pixel_location((float(x), float(y)), hdulist_index)\n source_cutout.reading.inverted = False\n (ra, dec) = source_cutout.pix2world(x, y, hdulist_index, usepv=True)\n self.place_marker(ra, dec, radius=int(source_cutout.apcor.ap_in*0.185+1)*units.arcsec,\n colour='green', force=True)\n else:\n key = isinstance(auto, bool) and \" \" or auto\n ra = source_cutout.reading.ra * units.degree\n dec = source_cutout.reading.dec * units.degree\n self.place_marker(ra, dec, radius=int(source_cutout.apcor.ap_in*0.185+1)*units.arcsec,\n colour='cyan', force=True)\n (x, y, hdulist_index) = source_cutout.world2pix(ra, dec, usepv=False)\n source_cutout.update_pixel_location((float(x), float(y)), hdulist_index)\n # now we've reset the pixel locations, so they are no longer inverted.\n source_cutout.reading.inverted = False\n\n marked_skycoord = source_cutout.reading.sky_coord\n\n try:\n phot = self.model.get_current_source_observed_magnitude()\n cen_x = phot['XCENTER'][0]\n cen_y = phot['YCENTER'][0]\n obs_mag = phot['MAG'][0]\n obs_mag_err = phot['MERR'][0]\n if phot.mask[0]['MAG'] or phot.mask[0]['MERR']:\n obs_mag = None\n obs_mag_err = None\n phot_failure = phot['PIER'][0] != 0 or phot.mask[0]['MAG']\n sky_failure = phot['SIER'][0] != 0 or phot.mask[0]['MAG']\n cen_failure = phot['CIER'][0] != 0\n if key != 'h':\n source_cutout.update_pixel_location((cen_x, cen_y), hdulist_index)\n except Exception as er:\n print(\"DAOPhot failure: {}\".format(er))\n logger.critical(\"PHOT ERROR: {}\".format(er))\n phot_failure = sky_failure = cen_failure = True\n obs_mag = None\n obs_mag_err = None\n band = ''\n default_comment = str(er)\n\n obs_mag = phot_failure and None or obs_mag\n obs_mag_err = phot_failure and None or obs_mag_err\n self.place_marker(source_cutout.ra * units.degree, source_cutout.dec*units.degree,\n radius=int(source_cutout.apcor.ap_in*0.185+1)*units.arcsec,\n colour='white',\n force=True)\n\n # compare the RA/DEC position of the reading now that we have measured it to the initial value.\n if marked_skycoord.separation(source_cutout.reading.sky_coord) > 1 * units.arcsec or cen_failure:\n # check if the user wants to use the selected location or the DAOPhot centroid.\n self.view.show_offset_source_dialog(source_cutout.reading.sky_coord, marked_skycoord)\n\n note1_default = \"\"\n if self.model.is_current_source_adjusted():\n note1_default = config.read(\"MPC.NOTE1_HAND_ADJUSTED\")\n else:\n for note in config.read(\"MPC.NOTE1OPTIONS\"):\n if note.lower().startswith(key):\n note1_default = note\n break\n note1 = len(note1_default) > 0 and note1_default[0] or note1_default\n\n if isinstance(self, ProcessTracksController):\n this_observation = mpc.Observation(\n null_observation=False,\n provisional_name=provisional_name,\n note1=note1,\n note2=config.read('MPC.NOTE2DEFAULT')[0],\n date=self.model.get_current_observation_date(),\n ra=self.model.get_current_ra(),\n dec=self.model.get_current_dec(),\n mag=obs_mag,\n mag_err=obs_mag_err,\n band=band,\n observatory_code=config.read(\"MPC.DEFAULT_OBSERVATORY_CODE\"),\n discovery=self.is_discovery,\n comment=None,\n xpos=source_cutout.reading.x,\n ypos=source_cutout.reading.y,\n frame=source_cutout.reading.obs.rawname,\n astrometric_level=source_cutout.astrom_header.get('ASTLEVEL', None))\n\n try:\n previous_observations = self.model.get_writer().get_chronological_buffered_observations()\n for idx, observation in enumerate(previous_observations):\n try:\n if observation.comment.frame.strip() == this_observation.comment.frame.strip():\n previous_observations[idx] = this_observation\n this_observation = False\n break\n except Exception as ex:\n print type(ex), str(ex)\n if this_observation:\n previous_observations.append(this_observation)\n print Orbfit(previous_observations).summarize()\n except Exception as ex:\n logger.error(str(type(ex))+\" \"+str(ex))\n print \"Failed to compute preliminary orbit.\"\n\n if obs_mag < 24 and auto is not False:\n self.on_do_accept(None,\n provisional_name,\n sky_failure and \"S poor sky\" or note1_default,\n config.read(\"MPC.NOTE2DEFAULT\"),\n self.model.get_current_observation_date(),\n self.model.get_current_ra(),\n self.model.get_current_dec(),\n obs_mag,\n obs_mag_err,\n band,\n config.read(\"MPC.DEFAULT_OBSERVATORY_CODE\"), \"\"\n )\n else:\n self.view.show_accept_source_dialog(\n provisional_name,\n self.model.get_current_observation_date(),\n self.model.get_current_ra(),\n self.model.get_current_dec(),\n obs_mag,\n obs_mag_err,\n band,\n note1_choices=config.read(\"MPC.NOTE1OPTIONS\"),\n note2_choices=config.read(\"MPC.NOTE2OPTIONS\"),\n note1_default=sky_failure and \"S poor sky\" or note1_default,\n note2_default=config.read(\"MPC.NOTE2DEFAULT\"),\n default_observatory_code=config.read(\"MPC.DEFAULT_OBSERVATORY_CODE\"),\n default_comment=default_comment,\n phot_failure=phot_failure,\n pixel_x=source_cutout.pixel_x,\n pixel_y=source_cutout.pixel_y)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef on_do_accept(self,\n minor_planet_number,\n provisional_name,\n note1,\n note2,\n date_of_obs,\n ra,\n dec,\n obs_mag,\n obs_mag_err,\n band,\n observatory_code,\n comment):\n \"\"\"\n After a source has been mark for acceptance create an MPC Observation record.\n\n @param minor_planet_number: The MPC Number associated with the object\n @param provisional_name: A provisional name associated with the object\n @param note1: The observational quality note\n @param note2: The observational circumstance note\n @param date_of_obs: Date of the observation as a Time object.\n @param ra: RA in degrees\n @param dec: DE in degrees\n @param obs_mag: observed magnitude.\n @param obs_mag_err: Uncertainty in the observed magnitude.\n @param band: filter/band of the observations\n @param observatory_code: MPC Observatory Code of telescope.\n @param comment: A free form comment (not part of MPC standard record)\n \"\"\"\n # Just extract the character code from the notes, not the\n # full description\n note1_code = note1.split(\" \")[0]\n note2_code = note2.split(\" \")[0]\n\n self.view.close_accept_source_dialog()\n self.model.set_current_source_name(provisional_name)\n\n source_cutout = self.model.get_current_cutout()\n\n mpc_observation = mpc.Observation(\n null_observation=False,\n provisional_name=provisional_name,\n note1=note1_code,\n note2=note2_code,\n date=date_of_obs,\n ra=ra,\n dec=dec,\n mag=obs_mag,\n mag_err=obs_mag_err,\n band=band,\n observatory_code=observatory_code,\n discovery=self.is_discovery,\n comment=comment,\n xpos=source_cutout.reading.x,\n ypos=source_cutout.reading.y,\n frame=source_cutout.reading.obs.rawname,\n astrometric_level=source_cutout.astrom_header.get('ASTLEVEL', None)\n )\n\n # Store the observation into the model.\n data = self.model.get_current_workunit().data\n key = mpc_observation.comment.frame.strip()\n data.mpc_observations[key] = mpc_observation\n\n # And write this observation out.\n self.model.get_writer().write(mpc_observation)\n\n # Mark the current item of the work unit as accepted.\n self.model.accept_current_item()\n\n # Detemine if the display should be reset.\n reset_frame = False\n if self.model.get_current_workunit().get_current_source_readings().is_on_last_item():\n self.view.clear()\n reset_frame = True\n self.model.next_item()\n if reset_frame:\n self.view.frame(1)", "response": "This method is called when the user is about to accept an object."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef on_do_accept(self, comment):\n self.view.close_vetting_accept_source_dialog()\n\n # Set to None if blank\n if len(comment.strip()) == 0:\n comment = None\n\n writer = self.model.get_writer()\n writer.write_source(self.model.get_current_source(), comment=comment, reject=False)\n\n self.model.accept_current_item()\n self.view.clear()\n self.model.next_item()", "response": "Called when a vetting candidate is accepted."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndisplays the comparison image", "response": "def on_load_comparison(self, research=False):\n \"\"\"\n Display the comparison image\n @param research: find a new comparison image even if one already known?\n \"\"\"\n\n cutout = self.model.get_current_cutout()\n if research:\n cutout.comparison_image_index = None\n comparison_image = cutout.comparison_image\n if comparison_image is None:\n print \"Failed to load comparison image: {}\".format(cutout.comparison_image_list[cutout.comparison_image_index])\n else:\n self.view.display(cutout.comparison_image, self.use_pixel_coords)\n self.view.align(self.model.get_current_cutout(),\n self.model.get_current_reading(),\n self.model.get_current_source())\n self.model.get_current_workunit().previous_obs()\n self.model.acknowledge_image_displayed()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncompute the x y transform of each image and compare to trans. jmp", "response": "def compute_trans(expnums, ccd, version, prefix=None, default=\"WCS\"):\n \"\"\"\n Pull the astrometric header for each image, compute an x/y transform and compare to trans.jmp\n\n this one overides trans.jmp if they are very different.\n @param expnums:\n @param ccd:\n @param version:\n @param prefix:\n @return: None\n \"\"\"\n wcs_dict = {}\n for expnum in expnums:\n try:\n # TODO This assumes that the image is already N/E flipped.\n # If compute_trans is called after the image is retrieved from archive then we get the disk version.\n filename = storage.get_image(expnum, ccd, version, prefix=prefix)\n this_wcs = wcs.WCS(fits.open(filename)[0].header)\n except Exception as err:\n logging.warning(\"WCS Trans compute failed. {}\".format(str(err)))\n return\n wcs_dict[expnum] = this_wcs\n x0 = wcs_dict[expnums[0]].header['NAXIS1'] / 2.0\n y0 = wcs_dict[expnums[0]].header['NAXIS2'] / 2.0\n (ra0, dec0) = wcs_dict[expnums[0]].xy2sky(x0, y0)\n result = \"\"\n for expnum in expnums:\n filename = storage.get_file(expnum, ccd, version, ext='.trans.jmp', prefix=prefix)\n jmp_trans = file(filename, 'r').readline().split()\n (x, y) = wcs_dict[expnum].sky2xy(ra0, dec0)\n x1 = float(jmp_trans[0]) + float(jmp_trans[1]) * x + float(jmp_trans[2]) * y\n y1 = float(jmp_trans[3]) + float(jmp_trans[4]) * x + float(jmp_trans[5]) * y\n dr = math.sqrt((x1 - x0) ** 2 + (y1 - y0) ** 2)\n if dr > 0.5:\n result += \"WARNING: WCS-JMP transforms mis-matched {} reverting to using {}.\\n\".format(expnum, default)\n if default == \"WCS\": \n uri = storage.dbimages_uri(expnum, ccd, version, ext='.trans.jmp', prefix=prefix)\n filename = os.path.basename(uri)\n trans = file(filename, 'w')\n trans.write(\"{:5.2f} 1. 0. {:5.2f} 0. 1.\\n\".format(x0 - x, y0 - y))\n trans.close()\n else:\n result += \"WCS-JMP transforms match {}\\n\".format(expnum)\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nruns the actual step2 on the given expnum and ccd combo", "response": "def step2(expnums, ccd, version, prefix=None, dry_run=False, default=\"WCS\"):\n \"\"\"run the actual step2 on the given exp/ccd combo\"\"\"\n\n jmp_trans = ['step2ajmp']\n jmp_args = ['step2bjmp']\n matt_args = ['step2matt_jmp']\n\n idx = 0\n for expnum in expnums:\n jmp_args.append(\n storage.get_file(expnum, ccd=ccd, version=version, ext='obj.jmp', prefix=prefix)[0:-8]\n )\n jmp_trans.append(\n storage.get_file(expnum, ccd=ccd, version=version, ext='obj.jmp', prefix=prefix)[0:-8]\n )\n idx += 1\n matt_args.append('-f%d' % idx)\n matt_args.append(\n storage.get_file(expnum, ccd=ccd, version=version, ext='obj.matt', prefix=prefix)[0:-9]\n )\n\n logging.info(util.exec_prog(jmp_trans))\n\n if default == \"WCS\":\n logging.info(compute_trans(expnums, ccd, version, prefix, default=default))\n\n logging.info(util.exec_prog(jmp_args))\n logging.info(util.exec_prog(matt_args))\n\n ## check that the shifts from step2 are rational\n check_args = ['checktrans']\n if os.access('proc-these-files', os.R_OK):\n os.unlink('proc-these-files')\n ptf = open('proc-these-files', 'w')\n ptf.write(\"# A dummy file that is created so checktrans could run.\\n\")\n ptf.write(\"# Frame FWHM PSF?\\n\")\n for expnum in expnums:\n filename = os.path.splitext(storage.get_image(expnum, ccd, version=version, prefix=prefix))[0]\n if not os.access(filename + \".bright.psf\", os.R_OK):\n os.link(filename + \".bright.jmp\", filename + \".bright.psf\")\n if not os.access(filename + \".obj.psf\", os.R_OK):\n os.link(filename + \".obj.jmp\", filename + \".obj.psf\")\n ptf.write(\"{:>19s}{:>10.1f}{:>5s}\\n\".format(filename,\n _FWHM,\n \"NO\"))\n ptf.close()\n if os.access('BAD_TRANS', os.F_OK):\n os.unlink('BAD_TRANS')\n\n logging.info(util.exec_prog(check_args))\n\n if os.access('BAD_TRANS', os.F_OK):\n raise OSError(errno.EBADMSG, 'BAD_TRANS')\n\n if os.access('proc-these-files', os.F_OK):\n os.unlink('proc-these-files')\n\n if dry_run:\n return\n\n for expnum in expnums:\n for ext in ['unid.jmp', 'unid.matt', 'trans.jmp']:\n uri = storage.dbimages_uri(expnum, ccd=ccd, version=version, ext=ext, prefix=prefix)\n filename = os.path.basename(uri)\n storage.copy(filename, uri)\n\n return"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef parse_nate_sims(path):\n '''\n parts0.dat) contains the id number, particle fraction (ignore) a, ecc, inc, long. asc., arg. per, and mean anomaly\n for every particle in the simulation at t=0.\n The second (parts3999.dat) contains the same info at t=3.999 Gyrs for these particles.\n\n :return:\n '''\n zerostate = pandas.read_table(path + 'parts0.dat', delim_whitespace=True)\n endstate = pandas.read_table(path + 'parts3999.dat', delim_whitespace=True)\n\n # add perihelion\n zerostate['q'] = zerostate['a'] * (1 - zerostate['e'])\n endstate['q'] = endstate['a'] * (1 - endstate['e'])\n\n return zerostate, endstate", "response": "Parse the nate simulation file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _convert_netmask(mask):\n if mask not in range(0, 33):\n raise scci.SCCIInvalidInputError(\n 'Netmask value is invalid.')\n\n return socket.inet_ntoa(struct.pack(\n '!L', int('1' * mask + '0' * (32 - mask), 2)))", "response": "Convert netmask from CIDR format to doted decimal string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\napply the configuration to iRMC.", "response": "def apply(self, reboot=False):\n \"\"\"Apply the configuration to iRMC.\"\"\"\n self.root.use_virtual_addresses = True\n self.root.manage.manage = True\n self.root.mode = 'new'\n self.root.init_boot = reboot\n\n self.client.set_profile(self.root.get_json())"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef terminate(self, reboot=False):\n self.root.manage.manage = False\n self.root.mode = 'delete'\n self.root.init_boot = reboot\n self.client.set_profile(self.root.get_json())", "response": "Terminate VIOM configuration from iRMC."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset the LAN port information to the configuration.", "response": "def set_lan_port(self, port_id, mac=None):\n \"\"\"Set LAN port information to configuration.\n\n :param port_id: Physical port ID.\n :param mac: virtual MAC address if virtualization is necessary.\n \"\"\"\n port_handler = _parse_physical_port_id(port_id)\n port = self._find_port(port_handler)\n if port:\n port_handler.set_lan_port(port, mac)\n else:\n self._add_port(port_handler, port_handler.create_lan_port(mac))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset iSCSI volume information to configuration.", "response": "def set_iscsi_volume(self, port_id,\n initiator_iqn, initiator_dhcp=False,\n initiator_ip=None, initiator_netmask=None,\n target_dhcp=False, target_iqn=None, target_ip=None,\n target_port=3260, target_lun=0, boot_prio=1,\n chap_user=None, chap_secret=None,\n mutual_chap_secret=None):\n \"\"\"Set iSCSI volume information to configuration.\n\n :param port_id: Physical port ID.\n :param initiator_iqn: IQN of initiator.\n :param initiator_dhcp: True if DHCP is used in the iSCSI network.\n :param initiator_ip: IP address of initiator. None if DHCP is used.\n :param initiator_netmask: Netmask of initiator as integer. None if\n DHCP is used.\n :param target_dhcp: True if DHCP is used for iSCSI target.\n :param target_iqn: IQN of target. None if DHCP is used.\n :param target_ip: IP address of target. None if DHCP is used.\n :param target_port: Port number of target. None if DHCP is used.\n :param target_lun: LUN number of target. None if DHCP is used,\n :param boot_prio: Boot priority of the volume. 1 indicates the highest\n priority.\n \"\"\"\n\n initiator_netmask = (_convert_netmask(initiator_netmask)\n if initiator_netmask else None)\n\n port_handler = _parse_physical_port_id(port_id)\n iscsi_boot = _create_iscsi_boot(\n initiator_iqn,\n initiator_dhcp=initiator_dhcp,\n initiator_ip=initiator_ip,\n initiator_netmask=initiator_netmask,\n target_dhcp=target_dhcp,\n target_iqn=target_iqn,\n target_ip=target_ip,\n target_port=target_port,\n target_lun=target_lun,\n boot_prio=boot_prio,\n chap_user=chap_user,\n chap_secret=chap_secret,\n mutual_chap_secret=mutual_chap_secret)\n\n port = self._find_port(port_handler)\n if port:\n port_handler.set_iscsi_port(port, iscsi_boot)\n else:\n port = port_handler.create_iscsi_port(iscsi_boot)\n self._add_port(port_handler, port)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef set_fc_volume(self, port_id,\n target_wwn, target_lun=0, boot_prio=1,\n initiator_wwnn=None, initiator_wwpn=None):\n \"\"\"Set FibreChannel volume information to configuration.\n\n :param port_id: Physical port ID.\n :param target_wwn: WWN of target.\n :param target_lun: LUN number of target.\n :param boot_prio: Boot priority of the volume. 1 indicates the highest\n priority.\n :param initiator_wwnn: Virtual WWNN for initiator if necessary.\n :param initiator_wwpn: Virtual WWPN for initiator if necessary.\n \"\"\"\n port_handler = _parse_physical_port_id(port_id)\n fc_target = elcm.FCTarget(target_wwn, target_lun)\n fc_boot = elcm.FCBoot(boot_prio=boot_prio, boot_enable=True)\n fc_boot.add_target(fc_target)\n\n port = self._find_port(port_handler)\n if port:\n port_handler.set_fc_port(port, fc_boot,\n wwnn=initiator_wwnn, wwpn=initiator_wwpn)\n else:\n port = port_handler.create_fc_port(fc_boot,\n wwnn=initiator_wwnn,\n wwpn=initiator_wwpn)\n self._add_port(port_handler, port)", "response": "Set FibreChannel volume information to configuration."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _pad_former_ports(self, port_handler):\n if not port_handler.need_padding():\n return\n for port_idx in range(1, port_handler.port_idx):\n pad_handler = port_handler.__class__(\n port_handler.slot_type,\n port_handler.card_type,\n port_handler.slot_idx,\n port_handler.card_idx,\n port_idx)\n if not self._find_port(pad_handler):\n self._add_port(pad_handler,\n pad_handler.create_lan_port())", "response": "Create ports with former port index."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the number of samples in a chunk of a file.", "response": "def chunk_size_samples(sf, buf):\n \"\"\"\n Black magic to account for the fact that libsndfile's behavior varies\n depending on file format when using the virtual io api.\n\n If you ask for more samples from an ogg or flac file than are available\n at that moment, libsndfile will give you no more samples ever, even if\n more bytes arrive in the buffer later.\n \"\"\"\n byte_depth = _lookup[sf.subtype]\n channels = sf.channels\n bytes_per_second = byte_depth * sf.samplerate * channels\n secs = len(buf) / bytes_per_second\n secs = max(1, secs - 6)\n return int(secs * sf.samplerate)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts an integer to a base n string where n is the length of the alphabet.", "response": "def encode(number, alphabet):\n \"\"\"\n Converts an integer to a base n string where n is the length of the\n provided alphabet.\n\n Modified from http://en.wikipedia.org/wiki/Base_36\n \"\"\"\n if not isinstance(number, (int, long)):\n raise TypeError(\"Number must be an integer.\")\n\n base_n = \"\"\n sign = \"\"\n\n if number < 0:\n sign = \"-\"\n number = -number\n\n if 0 <= number < len(alphabet):\n return sign + alphabet[number]\n\n while number != 0:\n number, i = divmod(number, len(alphabet))\n base_n = alphabet[i] + base_n\n\n return sign + base_n"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef categorical(x, mu=255, normalize=True):\n\n if normalize:\n # normalize the signal\n mx = x.max()\n x = np.divide(x, mx, where=mx != 0)\n\n # mu law compression\n x = mu_law(x)\n\n # translate and scale to [0, 1]\n x = (x - x.min()) * 0.5\n\n # convert to the range [0, 255]\n x = (x * mu).astype(np.uint8)\n\n # create the array to house the categorical representation\n c = np.zeros((np.product(x.shape), mu + 1), dtype=np.uint8)\n c[np.arange(len(c)), x.flatten()] = 1\n\n return ArrayWithUnits(\n c.reshape(x.shape + (mu + 1,)),\n x.dimensions + (IdentityDimension(),))", "response": "Categorical compression of a block of audio samples and convert them into a categorical distribution of the specified mu - law."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ninverses of categorical samples.", "response": "def inverse_categorical(x, mu=255):\n \"\"\"\n Invert categorical samples\n \"\"\"\n flat = x.reshape((-1, x.shape[-1]))\n indices = np.argmax(flat, axis=1).astype(np.float32)\n indices = (indices / mu) - 0.5\n inverted = inverse_mu_law(indices, mu=mu).reshape(x.shape[:-1])\n return ArrayWithUnits(inverted, x.dimensions[:2])"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsynthesizes one or more sine waves", "response": "def synthesize(self, duration, freqs_in_hz=[440.]):\n \"\"\"\n Synthesize one or more sine waves\n\n Args:\n duration (numpy.timdelta64): The duration of the sound to be\n synthesized\n freqs_in_hz (list of float): Numbers representing the frequencies\n in hz that should be synthesized\n \"\"\"\n freqs = np.array(freqs_in_hz)\n scaling = 1 / len(freqs)\n sr = int(self.samplerate)\n cps = freqs / sr\n ts = (duration / Seconds(1)) * sr\n ranges = np.array([np.arange(0, ts * c, c) for c in cps])\n raw = (np.sin(ranges * (2 * np.pi)) * scaling).sum(axis=0)\n return AudioSamples(raw, self.samplerate)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef synthesize(self, duration, tick_frequency):\n sr = self.samplerate.samples_per_second\n # create a short, tick sound\n tick = np.random.uniform(low=-1., high=1., size=int(sr * .1))\n tick *= np.linspace(1, 0, len(tick))\n # create silence\n samples = np.zeros(int(sr * (duration / Seconds(1))))\n ticks_per_second = Seconds(1) / tick_frequency\n # introduce periodic ticking sound\n step = int(sr // ticks_per_second)\n for i in range(0, len(samples), step):\n size = len(samples[i:i + len(tick)])\n samples[i:i + len(tick)] += tick[:size]\n return AudioSamples(samples, self.samplerate)", "response": "Synthesize periodic ticks generated from white noise and an envelope sound to be synthesized\n "} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsynthesizes white noise with a given duration.", "response": "def synthesize(self, duration):\n \"\"\"\n Synthesize white noise\n\n Args:\n duration (numpy.timedelta64): The duration of the synthesized sound\n \"\"\"\n sr = self.samplerate.samples_per_second\n seconds = duration / Seconds(1)\n samples = np.random.uniform(low=-1., high=1., size=int(sr * seconds))\n return AudioSamples(samples, self.samplerate)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef query_ssos(self):\n self._ssos_queried = True\n mpc_filename = self.save()\n return self.builder.build_workunit(mpc_filename)", "response": "Query the SSOS file for this workunit."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsave the current state of the SouceReading information for the currently recorded observations.", "response": "def save(self):\n \"\"\"\n Update the SouceReading information for the currently recorded observations and then flush those to a file.\n @return: mpc_filename of the resulting save.\n \"\"\"\n self.get_writer().flush()\n mpc_filename = self.get_writer().get_filename()\n self.get_writer().close()\n self._writer = None\n return mpc_filename"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget a MPCWriter object for the current entry.", "response": "def get_writer(self):\n \"\"\"\n Get a writer.\n\n This method also makes the output filename be the same as the .track file but with .mpc.\n (Currently only works on local filesystem)\n :rtype MPCWriter\n \"\"\"\n if self._writer is None:\n suffix = tasks.get_suffix(tasks.TRACK_TASK)\n try:\n base_name = re.search(\"(?P.*?)\\.\\d*{}\".format(suffix), self.filename).group('base_name')\n except:\n base_name = os.path.splitext(self.filename)[0]\n mpc_filename_pattern = self.output_context.get_full_path(\n \"{}.?{}\".format(base_name, suffix))\n mpc_file_count = len(glob(mpc_filename_pattern))\n mpc_filename = \"{}.{}{}\".format(base_name, mpc_file_count, suffix)\n self._writer = self._create_writer(mpc_filename)\n\n return self._writer"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn true if filename matches the name_filter regex and should be filtered out of the list.", "response": "def _filter(self, filename):\n \"\"\"\n return 'true' if filename doesn't match name_filter regex and should be filtered out of the list.\n @param filename:\n @return:\n \"\"\"\n return self.name_filter is not None and re.search(self.name_filter, filename) is None"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_workunit(self, ignore_list=None):\n if ignore_list is None:\n ignore_list = []\n\n potential_files = self.get_potential_files(ignore_list)\n\n while len(potential_files) > 0:\n potential_file = self.select_potential_file(potential_files)\n potential_files.remove(potential_file)\n if self._filter(potential_file):\n continue\n\n if self.directory_context.get_file_size(potential_file) == 0:\n continue\n\n if self.progress_manager.is_done(potential_file):\n self._done.append(potential_file)\n continue\n else:\n try:\n self.progress_manager.lock(potential_file)\n except FileLockedException:\n continue\n\n self._already_fetched.append(potential_file)\n return self.builder.build_workunit(\n self.directory_context.get_full_path(potential_file))\n\n logger.info(\"No eligible workunits remain to be fetched.\")\n\n raise NoAvailableWorkException()", "response": "Gets a new workunit from the directory."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_potential_files(self, ignore_list):\n exclude_prefix = self.taskid == tasks.suffixes.get(tasks.REALS_TASK, '') and 'fk' or None\n filenames = [filename for filename in\n self.directory_context.get_listing(self.taskid, exclude_prefix=exclude_prefix)\n if filename not in ignore_list and\n filename not in self._done and\n filename not in self._already_fetched]\n\n # if the extension is .mpc. then we look for the largest numbered MPC file.\n # look for the largest numbered MPC file only.\n if self.taskid == tasks.suffixes.get(tasks.TRACK_TASK, ''):\n basenames = {}\n for filename in filenames:\n fullname = os.path.splitext(filename)[0]\n if fullname in basenames:\n continue\n basename = os.path.splitext(fullname)[0]\n # only do the 'maximum' search when the 2nd extension is an integer value\n try:\n idx = int(filename.split('.')[-2])\n if idx > basenames.get(basename, 0):\n basenames[basename] = idx\n except:\n # since we failed, just keep the file in the list\n basenames[fullname] = ''\n filenames = []\n for basename in basenames:\n # sometimes the version is empty, so no '.' is needed\n version = basenames[basename]\n version = len(str(version)) > 0 and \".{}\".format(version) or version\n filenames.append(\"{}{}{}\".format(basename, version, self.taskid))\n # print basename, basenames[basename], filenames[-1]\n return filenames", "response": "Get a listing of files for the appropriate task which may or may not be locked and not done."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nmoving the discovery triplet to the front of the reading list.", "response": "def move_discovery_to_front(self, data):\n \"\"\"\n Moves the discovery triplet to the front of the reading list.\n Leaves everything else in the same order.\n \"\"\"\n readings = self.get_readings(data)\n discovery_index = self.get_discovery_index(data)\n\n reordered_readings = (readings[discovery_index:discovery_index + 3] +\n readings[:discovery_index] +\n readings[discovery_index + 3:])\n\n self.set_readings(data, reordered_readings)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndo a query of the CADC Megacam table. Returns a file - like object.", "response": "def TAPQuery(RAdeg=180.0, DECdeg=0.0, width=1, height=1):\n \"\"\"Do a query of the CADC Megacam table. Get all observations insize the box. Returns a file-like object\"\"\"\n\n\n QUERY =( \"\"\" SELECT \"\"\"\n \"\"\" COORD1(CENTROID(Plane.position_bounds)) AS \"RAJ2000\", COORD2(CENTROID(Plane.position_bounds)) AS \"DEJ2000\", Plane.time_bounds_lower as \"MJDATE\" \"\"\"\n \"\"\" FROM \"\"\"\n \"\"\" caom2.Observation as o JOIN caom2.Plane as Plane on o.obsID=Plane.obsID \"\"\"\n \"\"\" WHERE \"\"\"\n \"\"\" o.collection = 'CFHT' \"\"\"\n \"\"\" AND o.instrument_name = 'MegaPrime' \"\"\"\n \"\"\" AND INTERSECTS( BOX('ICRS', {}, {}, {}, {}), Plane.position_bounds ) = 1 \"\"\"\n \"\"\" AND ( o.proposal_id LIKE '%P05') \"\"\")\n # \"\"\" AND ( o.proposal_id LIKE '%P05' OR o.proposal_id LIKE '%L03' or o.proposal_id LIKE '%L06' or o.proposal_id\n # in ( '06AF33', '06BF98' ) ) \"\"\" )\n\n QUERY = QUERY.format( RAdeg, DECdeg, width, height)\n\n data={\"QUERY\": QUERY,\n \"REQUEST\": \"doQuery\",\n \"LANG\": \"ADQL\",\n \"FORMAT\": \"votable\"}\n \n url=\"http://www.cadc.hia.nrc.gc.ca/tap/sync\"\n\n print url, data\n\n return urllib.urlopen(url,urllib.urlencode(data))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef check_tags(my_expnum, ops_set, my_ccds, dry_run=True):\n tags = storage.get_tags(my_expnum)\n count = 0\n outcount = 0\n fails = []\n for ccd in my_ccds:\n success = True\n count += 1\n for ops in ops_set:\n for fake in ops[0]:\n for my_program in ops[1]:\n for version in ops[2]:\n #print my_expnum, fake, my_program, version, ccd\n key = storage.get_process_tag(fake + my_program, ccd, version)\n uri = storage.tag_uri(key)\n if \"Failed to get image\" in tags.get(uri, \"Failed to get image\"):\n #print tags.get(uri, None)\n fails.append(ccd)\n success = False\n if success:\n outcount += 1\n sys.stderr.write(\"{} {} {:5.1f}%\\n\".format(outcount, count,100* float(outcount)/count))\n #print fails\n return set(fails)", "response": "check the tags for the given expnum and ccd set"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngiving a list of exposure numbers and CCD return the filename of the file that is flipped.", "response": "def get_flipped_ext(file_id,ccd):\n \"\"\"Given a list of exposure numbers and CCD, get them from the DB\"\"\"\n \n import MOPfits\n import os, shutil\n \n filename=MOPfits.adGet(file_id,extno=int(ccd))\n if int(ccd)<18:\n tfname=filename+\"F\"\n\tshutil.move(filename, tfname)\n os.system(\"imcopy %s[-*,-*] %s\" % (tfname, filename))\n\tos.unlink(tfname)\n if not os.access(filename,os.R_OK):\n return(None)\n return(filename)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets the exposure for a particular line in the meausre table", "response": "def get_file_ids(object):\n \"\"\"Get the exposure for a particular line in the meausre table\"\"\"\n import MOPdbaccess\n mysql = MOPdbaccess.connect('cfeps','cfhls',dbSystem='MYSQL')\n cfeps=mysql.cursor()\n sql=\"SELECT file_id FROM measure WHERE provisional LIKE %s\"\n cfeps.execute(sql,(object, ))\n file_ids=cfeps.fetchall()\n\n return (file_ids)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef main():\n parser = argparse.ArgumentParser(\n description='replace image header')\n parser.add_argument('--extname',\n help='name of extension to in header')\n parser.add_argument('expnum', type=str,\n help='exposure to update')\n parser.add_argument('-r', '--replace',\n action='store_true',\n help='store modified image back to VOSpace?')\n parser.add_argument('-v', '--verbose', action='store_true')\n parser.add_argument('--debug', action='store_true')\n parser.add_argument('--force', action='store_true', help=\"Re-run even if previous success recorded\")\n parser.add_argument('--dbimages', help=\"VOSpace DATA storage area.\", default=\"vos:OSSOS/dbimages\")\n\n args = parser.parse_args()\n task = util.task()\n dependency = 'preproc'\n prefix = \"\"\n\n storage.DBIMAGES = args.dbimages\n\n level = logging.CRITICAL\n message_format = \"%(message)s\"\n if args.verbose:\n level = logging.INFO\n if args.debug:\n level = logging.DEBUG\n message_format = \"%(module)s %(funcName)s %(lineno)s %(message)s\"\n logging.basicConfig(level=level, format=message_format)\n storage.set_logger(task, prefix, args.expnum, None, None, False)\n\n message = storage.SUCCESS\n expnum = args.expnum\n\n exit_status = 0\n try:\n # skip if already succeeded and not in force mode\n if storage.get_status(task, prefix, expnum, \"p\", 36) and not args.force:\n logging.info(\"Already updated, skipping\")\n sys.exit(0)\n \n image_hdulist = storage.get_image(args.expnum, return_file=False)\n ast_hdulist = storage.get_astheader(expnum, ccd=None)\n\n run_update_header(image_hdulist, ast_hdulist)\n image_filename = os.path.basename(storage.get_uri(expnum))\n image_hdulist.writeto(image_filename)\n if args.replace:\n dest = storage.dbimages_uri(expnum)\n storage.copy(image_filename, dest)\n storage.set_status('update_header', \"\", expnum, 'p', 36, message)\n except Exception as e:\n message = str(e)\n if args.replace:\n storage.set_status(task, prefix, expnum, 'p', 36, message)\n exit_status = message\n logging.error(message)\n\n return exit_status", "response": "Main entry point for the main script."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts lat / lon from the ecliptic to the invariable plane", "response": "def convert(lat, lon):\n \"\"\"convert lat/lon from the ecliptic to the invariable plane.\"\"\"\n\n x = numpy.cos(lon) * numpy.cos(lat)\n y = numpy.sin(lon) * numpy.cos(lat)\n z = numpy.sin(lat)\n\n # Invariable plane: values in arcseconds.\n epsilon = 5713.86\n omega = 387390.8\n\n coseps = numpy.cos(epsilon * secrad)\n sineps = numpy.sin(epsilon * secrad)\n cosom = numpy.cos(omega * secrad)\n sinom = numpy.sin(omega * secrad)\n\n xi = x * cosom + y * sinom\n yi = coseps * (-sinom * x + cosom * y) + sineps * z\n zi = - sineps * (-sinom * x + cosom * y) + coseps * z\n\n lat = numpy.arcsin(zi)\n lon = numpy.arctan2(yi, xi)\n return (lat, lon)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngiving a list of exposure numbers, find all the KBOs in that set of exposures", "response": "def searchTriples(expnums,ccd):\n \"\"\"Given a list of exposure numbers, find all the KBOs in that set of exposures\"\"\"\n import MOPfits,os \n import MOPdbaccess\n \n if len(expnums)!=3:\n return(-1)\n\n\n \n\n\n mysql=MOPdbaccess.connect('bucket','cfhls','MYSQL')\n bucket=mysql.cursor()\n ### Some program Constants\n\n proc_file = open(\"proc-these-files\",\"w\")\n proc_file.write(\"# Files to be planted and searched\\n\")\n proc_file.write(\"# image fwhm plant\\n\")\n \n import string\n import os.path\n filenames=[]\n import pyfits\n for expnum in expnums:\n bucket.execute(\"SELECT obs_iq_refccd FROM exposure WHERE expnum=%s\" , (expnum, ) )\n row=bucket.fetchone()\n fwhm=row[0]\n if not fwhm > 0:\n fwhm=1.0\n\n if int(ccd)<18:\n cutout=\"[-*,-*]\"\n else:\n cutout=None\n filename=MOPfits.adGet(str(expnum)+\"p\",extno=int(ccd),cutout=cutout)\n print filename\n if not os.access(filename,os.R_OK):\n return(-3)\n filename=os.path.splitext(filename)\n filenames.append(filename[0])\n proc_file.write(\"%s %f %s \\n\" % ( filename[0], fwhm/0.183, \"no\"))\n\n proc_file.flush()\n proc_file.close()\n \n command=\"find.pl -p '' -d ./ \"\n sys.stderr.write(command)\n try:\n os.system(command)\n except:\n sys.stderr.write(\"Failed while running find\")\n \n\n file_extens=[\n \"cands.comb\",\n \"measure3.cands.astrom\",\n \"measure3.WARNING\",\n \"measure3.astrom.scatter\"]\n \n if os.access(\"find.OK\",os.R_OK):\n os.system(\"touch /home/cadc/kavelaar/results/05AQ06B/\"+filenames[0]+\".OK\")\n else:\n os.system(\"touch /home/cadc/kavelaar/results/05AQ06B/\"+filenames[0]+\".FAILED\")\n \n ### look for the cand.comb file and store in the DB\n import shutil\n for ext in file_extens:\n if os.access(filenames[0]+\".\"+ext,os.R_OK):\n shutil.copy(filenames[0]+\".\"+ext,\"/home/cadc/kavelaar/results/05AQ06B\")\n astrom=filenames[0]+\".measure3.cands.astrom\"\n print astrom\n cmd = \"mpc_gen.pl -c \"+astrom\n print os.access(astrom,os.R_OK)\n if os.access(astrom,os.R_OK):\n print cmd\n os.system(cmd)\n os.system(\"mpcIngest.pl *.MPC\")\n\tos.system(\"cp *.MPC /home/cadc/kavelaar/results/05AQ06B\")\n\treturn(1)\n return(0)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nhandle an error that occurred while downloading the object.", "response": "def handle_error(self, error, download_request):\n \"\"\"\n Checks what error occured and looks for an appropriate solution.\n\n Args:\n error: Exception\n The error that has occured.\n download_request:\n The request which resulted in the error.\n \"\"\"\n if hasattr(error, \"errno\") and error.errno == errno.EACCES:\n self.handle_certificate_problem(str(error))\n else:\n self.handle_general_download_error(str(error), download_request)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets the date of the current observation by looking in the header of the last HDU element and the EXPTIME keywords.", "response": "def get_current_observation_date(self):\n \"\"\"\n Get the date of the current observation by looking in the header\n of the observation for the DATE and EXPTIME keywords.\n\n The 'DATE AT MIDDLE OF OBSERVATION' of the observation is returned\n @return: Time\n \"\"\"\n # All HDU elements have the same date and time so just use\n # last one, sometimes the first one is missing the header, in MEF\n header = self.get_current_cutout().hdulist[-1].header\n mjd_obs = float(header.get('MJD-OBS'))\n exptime = float(header.get('EXPTIME'))\n mpc_date = Time(mjd_obs,\n format='mjd',\n scale='utc',\n precision=config.read('MPC.DATE_PRECISION'))\n mpc_date += TimeDelta(exptime * units.second) / 2.0\n mpc_date = mpc_date.mpc\n return mpc_date"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef square(m, eff_max,c,m0,sigma,m1=21):\n\n return (eff_max-c*(m-21)**2)/(1+numpy.exp((m-m0)/sigma))", "response": "Compute the square of the drop - off in the peak"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nparsing the line from the. eff file that contains the efficiency function parameters for a square", "response": "def parse_square_param(line):\n \"\"\"\n Parse the line from the .eff file that contains the efficiency function\n parameters for a 'square' function\n line : the line containt the parameters, must start with 'square_param'\n \"\"\"\n if not line.startswith(\"square_param=\"):\n raise ValueError(\"Not a valid square_param line\")\n values = line.split()\n params = {'sigma': float(values.pop()),\n 'm0': float(values.pop()),\n 'c': float(values.pop()),\n 'eff_max': float(values.pop())\n }\n return params"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef parse_eff(filename):\n blocks = []\n block = {}\n\n with open(filename) as efile:\n for line in efile.readlines():\n if line.lstrip().startswith(\"#\"):\n continue\n keyword = line.lstrip().split(\"=\")[0]\n funcs = {'square_param': parse_square_param, \n 'rates': rates}\n block[keyword] = funcs.get(keyword, dummy)(line)\n if keyword == 'mag_lim':\n blocks.append(block)\n block = {}\n\n return blocks", "response": "Parse the Jean - Marcs. eff files."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef read(keypath, configfile=None):\n if configfile in _configs:\n appconfig = _configs[configfile]\n else:\n appconfig = AppConfig(configfile=configfile)\n _configs[configfile] = appconfig\n\n return appconfig.read(keypath)", "response": "Reads a value from the configuration file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a CDATA section for the current locale.", "response": "def _cdata_header(self, colsep=\"|\"):\n \"\"\"\n Create a header for the CDATA section, as a visual guide.\n \"\"\"\n fields = self.fields\n header_lines = []\n line = \"\"\n for fieldName in self.field_names:\n width = int(fields[fieldName]['attr']['width'])\n line += self._entry(fieldName, width, colsep)\n header_lines.append(line)\n\n line = \"\"\n for fieldName in self.field_names:\n width = int(fields[fieldName]['attr']['width'])\n line += self._entry(fields[fieldName]['attr']['format'], width=width, colsep=colsep)\n header_lines.append(line)\n\n line = \"\"\n for fieldName in self.field_names:\n width = int(fields[fieldName]['attr']['width'])\n (l, m) = divmod(width, 10)\n guide = \"\"\n for i in range(l):\n guide += \"\".join(map(str, range(10)))\n guide += \"\".join(map(str, range(m)))\n line += self._entry(guide, width=width, colsep=colsep)\n header_lines.append(line)\n\n line = \"\"\n for fieldName in self.field_names:\n width = int(fields[fieldName]['attr']['width'])\n guide = \"-\" * width\n line += self._entry(guide, width=width, colsep=colsep)\n header_lines.append(line)\n\n return header_lines"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _append_cdata(self, coordinate):\n fields = self.fields\n sra = coordinate.ra.to_string(units.hour, sep=':', precision=2, pad=True)\n sdec = coordinate.dec.to_string(units.degree, sep=':', precision=1, alwayssign=True)\n coord = SkyCoord(sra + \" \" + sdec, unit=(units.hour, units.degree))\n sra = coord.ra.to_string(units.hour, sep=\":\", precision=2, pad=True)\n sdec = coord.dec.to_string(units.degree, sep=\":\", precision=1, pad=True, alwayssign=True)\n sdate = str(coordinate.obstime.replicate(format('iso')))\n self.cdata.appendData(self._entry(sdate, fields[\"DATE_UTC\"]['attr']['width'], colsep=self.column_separator))\n self.cdata.appendData(self._entry(sra, fields[\"RA_J2000\"]['attr']['width'], colsep=self.column_separator))\n self.cdata.appendData(self._entry(sdec, fields[\"DEC_J2000\"][\"attr\"][\"width\"], colsep=self.column_separator))\n self.cdata.appendData(\"\\n\")", "response": "Append a data column to the ephemeris listing."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nwrites out a GEMINI formated OT ephemeris.", "response": "def gemini_writer(self, f_handle):\n \"\"\"\n Write out a GEMINI formated OT ephemeris. This is just a hack of SSD Horizons output.\n \"\"\"\n f_handle.write(GEMINI_HEADER)\n # Date__(UT)__HR:MN Date_________JDUT R.A.___(ICRF/J2000.0)___DEC dRA*cosD d(DEC)/dt\n # 1 2 3 4 5 6 7 8 9\n # 123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890\n # ' 2019-Jan-30 00:00 01 46 56.46 +10 28 54.9 01 47 56.17 +10 34 27.6 3.520\n\n for coordinate in self.coordinates:\n date = coordinate.obstime.datetime.strftime('%Y-%b-%d %H:%M')[:17]\n f_handle.write(\" {:16} {:17.9f} {:27} {:+8.5f} {:+8.5f}\\n\".format(date,\n coordinate.obstime.jd,\n coordinate.to_string('hmsdms',\n sep=' ',\n precision=4,\n pad=True)[:27],\n float(coordinate.dra),\n float(coordinate.ddec)),\n )\n f_handle.write(GEMINI_FOOTER)\n return"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef mkpsf_failures():\n\n for expnum in storage.list_dbimages():\n for ccd in range(36):\n if not storage.get_status(MKPSF, \"\", expnum, \"p\", ccd):\n # get_status returns FALSE if process didn't succeed,\n # with return_message=True it returns the error message.\n print expnum, ccd, storage.get_status(MKPSF, \"\", expnum, \"p\", ccd, return_message=True)", "response": "A simple script to loop over the standard tags for the mkpsf and\n and check if the process failed."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngive a list of exposures find all the KBOs in that set of exposures and return a list of the KBOs that are found in the same order as the original ones.", "response": "def searchTriples(expnums,ccd,plant=False):\n \"\"\"Given a list of exposure numbers, find all the KBOs in that set of exposures\"\"\"\n import MOPfits,os \n import MOPdbaccess\n \n if len(expnums)!=3:\n raise TaskError, \"got %d exposures\"%(len(expnums))\n\n\n ### Some program Constants\n proc_these_files=[]\n if not plant:\n proc_these_files.append(\"# Files to be planted and searched\\n\")\n proc_these_files.append(\"# image fwhm plant\\n\")\n \n import string\n import os.path\n filenames=[]\n import pyfits\n for expnum in expnums:\n ### Get the processed images from AD\n if int(ccd)<18:\n cutout=\"[-*,-*]\"\n else:\n cutout=\"[*,*]\"\n filename=MOPfits.adGet(str(expnum)+opt.raw,extno=int(ccd),cutout=cutout)\n\n if not os.access(filename,os.R_OK):\n sys.stderr.write(\"Ad Get Failed\\n\")\n raise TaskError, 'adGet Failed'\n\t \n\tif opt.none:\n\t continue\n\n filename=os.path.splitext(filename)\n filenames.append(filename[0])\n\n \ttry: \n mysql=MOPdbaccess.connect('bucket','cfhls','MYSQL')\n bucket=mysql.cursor()\n\texcept:\n raise TaskError, \"mysql failed\"\n bucket.execute(\"SELECT obs_iq_refccd FROM exposure WHERE expnum=%s\" , (expnum, ) )\n row=bucket.fetchone()\n\tmysql.close()\n fwhm=row[0]\n if not fwhm > 0:\n fwhm=1.0\n\n if not plant:\n #proc_these_files.append(\"%s %f %s \\n\" % ( filename[0], fwhm/0.183, 'no'))\n\t pstr='NO'\n else:\n\t pstr='YES'\n ### since we're planting we need a psf. JMPMAKEPSF will\n ### update the proc-these-files listing\n\n ### run the make psf script .. always. This creates proc-these-files\n ### which is needed by the find.pl script.\n command='jmpmakepsf.csh ./ %s %s' % ( filename[0], pstr )\n if opt.verbose:\n sys.stderr.write( command )\n try:\n os.system(command)\n except:\n raise TaskError, \"jmpmakepsf noexec\"\n if os.access(filename[0]+'.jmpmakepsf.FAILED',os.R_OK) or not os.access(filename[0]+\".psf.fits\", os.R_OK) :\n#\t if plant:\n# raise TaskError, \"jmpmakepsf failed\"\n#\tdo without plant\n if 1==1 :\n\t plant=False\n\t\tpstr='NO'\n\t ### we're not planting so, lets keep going\n\t\t### but check that there is a line in proc_these_files\n\t add_line=True\n\t\tif not os.access('proc-these-files',os.R_OK):\n\t\t f=open('proc-these-files','w')\n\t\t for l in proc_these_files:\n\t\t f.write(l)\n\t\t f.close()\n\t \tf=open('proc-these-files','r')\n\t\tptf_lines=f.readlines()\n\t\tf.close()\n\t\tfor ptf_line in ptf_lines:\n\t\t if ptf_line[0]=='#':\n\t\t continue\n\t ptf_a=ptf_line.split()\n\t\t import re\n\t\t if re.search('%s' % (filename[0]),ptf_a[0]):\n\t\t ### there's already a line for this one\n\t\t\tadd_line=False\n\t\t break\n if add_line:\n\t\t f=open('proc-these-files','a')\n\t\t f.write(\"%s %f %s \\n\" % ( filename[0], fwhm/0.183, 'no'))\n\t\t f.close()\n\n if opt.none:\n return(-1)\n prefix=''\n if plant:\n command=\"plant.csh ./ -rmin %s -rmax %s -ang %s -width %s \" % ( opt.rmin, opt.rmax, opt.angle, opt.width)\n try: \n os.system(command)\n except:\n raise TaskError, 'plant exec. failed'\n if not os.access('plant.OK',os.R_OK):\n raise TaskError, 'plant failed'\n prefix='fk'\n #else:\n # f=open('proc-these-files','w')\n # for line in proc_these_files:\n # f.write(line)\n # f.flush()\n # f.close()\n \t\n if opt.rerun and os.access('find.OK',os.R_OK):\n os.unlink(\"find.OK\")\n\n command=\"find.pl -p \"+prefix+\" -rn %s -rx %s -a %s -aw %s -d ./ \" % ( opt.rmin, opt.rmax, opt.angle, opt.width) \n #command=\"find.pl -p \"+prefix+\" -d ./ \" \n if opt.verbose:\n sys.stderr.write( command )\n\n try:\n os.system(command)\n except:\n raise TaskErorr, \"execute find\"\n \n\n if not os.access(\"find.OK\",os.R_OK):\n raise TaskError, \"find failed\"\n\n ### check the transformation file\n command = \"checktrans -p \"+prefix\n \n try:\n os.system(command)\n except:\n raise TaskError, \"execute checktrans\"\n \n if not os.access(\"checktrans.OK\",os.R_OK):\n raise TaskError, \"checktrans failed\"\n\n if os.access(\"BAD_TRANS\"+prefix,os.R_OK):\n raise TaskError,\"BAD TRANS\"\n \n astrom=prefix+filenames[0]+\".cands.comb\"\n if opt.plant:\n astrom=prefix+filenames[0]+\".comb.found\"\n try:\n #make sure we have +10 lines in this file\n\t lines=file(astrom).readlines()\n\t if len(lines)<10:\n\t raise TaskError,\"Too few Found\"\n\texcept:\n\t raise TaskError, \"Error reading %s\" %(astrom)\n\t \n \n if os.access(astrom,os.R_OK):\n return(1)\n else:\n return(0)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_nailing(expnum,ccd):\n sql=\"\"\"\n SELECT e.expnum, (e.mjdate - f.mjdate) dt\n FROM bucket.exposure e\n JOIN bucket.exposure f\n JOIN bucket.association b ON b.expnum=f.expnum\n JOIN bucket.association a ON a.pointing=b.pointing AND a.expnum=e.expnum\n WHERE f.expnum=%d\n AND abs(e.mjdate - f.mjdate) > 0.5\n AND abs(e.mjdate - f.mjdate) < 15.0\n ORDER BY abs(e.mjdate-f.mjdate)\n \"\"\" % ( expnum )\n try:\n import MOPdbaccess\n mysql=MOPdbaccess.connect('bucket','cfhls',dbSystem='MYSQL')\n bucket=mysql.cursor()\n bucket.execute(sql)\n nailings = bucket.fetchall()\n mysql.close()\n if int(ccd) < 18:\n cutout=\"[-*,-*]\"\n else:\n cutout=None\n import MOPfits\n for nailing in nailings:\n filename=MOPfits.adGet(str(nailing[0])+opt.raw,extno=int(ccd),cutout=cutout)\n except:\n raise TaskError, \"get nailing failed\"", "response": "Get the nailing images associated with expnum."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the intersection between this frequency band and another.", "response": "def intersect(self, other):\n \"\"\"\n Return the intersection between this frequency band and another.\n\n Args:\n other (FrequencyBand): the instance to intersect with\n\n Examples::\n >>> import zounds\n >>> b1 = zounds.FrequencyBand(500, 1000)\n >>> b2 = zounds.FrequencyBand(900, 2000)\n >>> intersection = b1.intersect(b2)\n >>> intersection.start_hz, intersection.stop_hz\n (900, 1000)\n \"\"\"\n lowest_stop = min(self.stop_hz, other.stop_hz)\n highest_start = max(self.start_hz, other.start_hz)\n return FrequencyBand(highest_start, lowest_stop)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning an iterable of all bands in this scale.", "response": "def bands(self):\n \"\"\"\n An iterable of all bands in this scale\n \"\"\"\n if self._bands is None:\n self._bands = self._compute_bands()\n return self._bands"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef ensure_overlap_ratio(self, required_ratio=0.5):\n\n msg = \\\n 'band {i}: ratio must be at least {required_ratio} but was {ratio}'\n\n for i in range(0, len(self) - 1):\n b1 = self[i]\n b2 = self[i + 1]\n\n try:\n ratio = b1.intersection_ratio(b2)\n except ValueError:\n ratio = 0\n\n if ratio < required_ratio:\n raise AssertionError(msg.format(**locals()))", "response": "Ensures that every adjacent frequency band pair in the set of frequency bands meets the required overlap ratio."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the quality factor of the center frequencies and bandwidths of the logarithmic center frequencies.", "response": "def Q(self):\n \"\"\"\n The quality factor of the scale, or, the ratio of center frequencies\n to bandwidths\n \"\"\"\n return np.array(list(self.center_frequencies)) \\\n / np.array(list(self.bandwidths))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_slice(self, frequency_band):\n index = frequency_band\n\n if isinstance(index, slice):\n types = {\n index.start.__class__,\n index.stop.__class__,\n index.step.__class__\n }\n\n if Hertz not in types:\n return index\n\n try:\n start = Hertz(0) if index.start is None else index.start\n if start < Hertz(0):\n start = self.stop_hz + start\n stop = self.stop_hz if index.stop is None else index.stop\n if stop < Hertz(0):\n stop = self.stop_hz + stop\n frequency_band = FrequencyBand(start, stop)\n except (ValueError, TypeError):\n pass\n\n start_index = bisect.bisect_left(\n self.band_stops, frequency_band.start_hz)\n stop_index = bisect.bisect_left(\n self.band_starts, frequency_band.stop_hz)\n\n if self.always_even and (stop_index - start_index) % 2:\n # KLUDGE: This is simple, but it may make sense to choose move the\n # upper *or* lower bound, based on which one introduces a lower\n # error\n stop_index += 1\n return slice(start_index, stop_index)", "response": "Given a frequency band and a frequency dimension comprised of\n n_samples return a slice that may be used to extract only the frequency samples that intersect with the frequency band."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef from_sample_rate(sample_rate, n_bands, always_even=False):\n fb = FrequencyBand(0, sample_rate.nyquist)\n return LinearScale(fb, n_bands, always_even=always_even)", "response": "Returns a LinearScale instance from a sampling rate."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts an hertz into a number of semitones above or below some reference value.", "response": "def _hz_to_semitones(self, hz):\n \"\"\"\n Convert hertz into a number of semitones above or below some reference\n value, in this case, A440\n \"\"\"\n return np.log(hz / self._a440) / np.log(self._a)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nlooking up the name of a source using a resolver", "response": "def resolve(object):\n \"\"\"Look up the name of a source using a resolver\"\"\"\n\n\n import re\n sesame_cmd = 'curl -s http://cdsweb.u-strasbg.fr/viz-bin/nph-sesame/-oI?'+string.replace(object,' ','')\n f = os.popen(sesame_cmd)\n lines = f.readlines()\n f.close()\n for line in lines: \n if re.search('%J ', line):\n result2 = line.split()\n ra_deg = float(result2[1])\n dec_deg = float(result2[2])\n return (ra_deg, dec_deg)\n \n return (0,0)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nremoving unnecessary plot borders and axis ticks.", "response": "def remove_border(axes=None, keep=('left', 'bottom'), remove=('right', 'top'), labelcol=ALMOST_BLACK):\n \"\"\"\n Minimize chart junk by stripping out unnecessary plot borders and axis ticks.\n The top/right/left/bottom keywords toggle whether the corresponding plot border is drawn\n \"\"\"\n ax = axes or plt.gca()\n for spine in remove:\n ax.spines[spine].set_visible(False)\n for spine in keep:\n ax.spines[spine].set_linewidth(0.5)\n # ax.spines[spine].set_color('white')\n\n # remove all ticks, then add back the ones in keep\n # Does this also need to specify the ticks' colour, given the axes/labels are changed?\n ax.yaxis.set_ticks_position('none')\n ax.xaxis.set_ticks_position('none')\n # ax.xaxis.set_ticklabels(\"\")\n # ax.yaxis.set_ticklabels(\"\")\n\n for spine in keep:\n if spine == 'top':\n ax.xaxis.tick_top()\n if spine == 'bottom':\n ax.xaxis.tick_bottom()\n # match the label colour to that of the axes\n ax.xaxis.label.set_color(labelcol)\n ax.xaxis.set_tick_params(color=labelcol, labelcolor=labelcol)\n if spine == 'left':\n ax.yaxis.tick_left()\n ax.yaxis.label.set_color(labelcol)\n ax.yaxis.set_tick_params(color=labelcol, labelcolor=labelcol)\n if spine == 'right':\n ax.yaxis.tick_right()\n\n return"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsample function showing how to interpret the dialog exit codes. This function is not used after every call to dialog in this demo for two reasons: 1. For some boxes, unfortunately, dialog returns the code for ERROR when the user presses ESC (instead of the one chosen for ESC). As these boxes only have an OK button, and an exception is raised and correctly handled here in case of real dialog errors, there is no point in testing the dialog exit status (it can't be CANCEL as there is no CANCEL button; it can't be ESC as unfortunately, the dialog makes it appear as an error; it can't be ERROR as this is handled in dialog.py to raise an exception; therefore, it *is* OK). 2. To not clutter simple code with things that are demonstrated elsewhere.", "response": "def handle_exit_code(d, code):\n \"\"\"Sample function showing how to interpret the dialog exit codes.\n\n This function is not used after every call to dialog in this demo\n for two reasons:\n\n 1. For some boxes, unfortunately, dialog returns the code for\n ERROR when the user presses ESC (instead of the one chosen\n for ESC). As these boxes only have an OK button, and an\n exception is raised and correctly handled here in case of\n real dialog errors, there is no point in testing the dialog\n exit status (it can't be CANCEL as there is no CANCEL\n button; it can't be ESC as unfortunately, the dialog makes\n it appear as an error; it can't be ERROR as this is handled\n in dialog.py to raise an exception; therefore, it *is* OK).\n\n 2. To not clutter simple code with things that are\n demonstrated elsewhere.\n\n \"\"\"\n # d is supposed to be a Dialog instance\n if code in (d.DIALOG_CANCEL, d.DIALOG_ESC):\n if code == d.DIALOG_CANCEL:\n msg = \"You chose cancel in the last dialog box. Do you want to \" \\\n \"exit this demo?\"\n else:\n msg = \"You pressed ESC in the last dialog box. Do you want to \" \\\n \"exit this demo?\"\n # \"No\" or \"ESC\" will bring the user back to the demo.\n # DIALOG_ERROR is propagated as an exception and caught in main().\n # So we only need to handle OK here.\n if d.yesno(msg) == d.DIALOG_OK:\n sys.exit(0)\n return 0\n else:\n return 1"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngives a list of exposure numbers, find all the KBOs in that set of exposures", "response": "def searchTriples(expnums,ccd):\n \"\"\"Given a list of exposure numbers, find all the KBOs in that set of exposures\"\"\"\n import MOPfits,os \n import MOPdbaccess\n \n if len(expnums)!=3:\n return(-1)\n\n\n ### Some program Constants\n\n proc_file = open(\"proc-these-files\",\"w\")\n proc_file.write(\"# Files to be planted and searched\\n\")\n proc_file.write(\"# image fwhm plant\\n\")\n \n import string\n import os.path\n filenames=[]\n import pyfits\n for expnum in expnums:\n \ttry: \n mysql=MOPdbaccess.connect('bucket','cfhls','MYSQL')\n bucket=mysql.cursor()\n\texcept:\n raise TaskError, \"mysql failed\"\n bucket.execute(\"SELECT obs_iq_refccd FROM exposure WHERE expnum=%s\" , (expnum, ) )\n row=bucket.fetchone()\n\tmysql.close()\n fwhm=row[0]\n if not fwhm > 0:\n fwhm=1.0\n\n if int(ccd)<18:\n cutout=\"[-*,-*]\"\n else:\n cutout=None\n filename=MOPfits.adGet(str(expnum)+\"p\",extno=int(ccd),cutout=cutout)\n\n if not os.access(filename,os.R_OK):\n raise TaskError, 'adGet Failed'\n\t \n filename=os.path.splitext(filename)\n filenames.append(filename[0])\n proc_file.write(\"%s %f %s \\n\" % ( filename[0], fwhm/0.183, \"no\"))\n\n proc_file.flush()\n proc_file.close()\n \n command=\"find.pl -p '' -d ./ \"\n\n try:\n os.system(command)\n except:\n raise TaskErorr, \"execute find\"\n \n\n file_extens=[\n \"cands.comb\",\n \"measure3.cands.astrom\",\n \"measure3.WARNING\",\n \"measure3.astrom.scatter\"]\n \n if not os.access(\"find.OK\",os.R_OK):\n raise TaskError, \"find failed\"\n \n \n astrom=filenames[0]+\".measure3.cands.astrom\"\n \n if os.access(astrom,os.R_OK):\n return(1)\n else:\n return(0)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndisplays the candidates contained in a candidate record list", "response": "def discands(record):\n \"\"\"Display the candidates contained in a candidate record list\"\"\"\n import pyfits\n pyraf.iraf.images()\n pyraf.iraf.images.tv()\n display = pyraf.iraf.images.tv.display\n \n width=128\n \n cands = record['cands']\n exps= record['fileId']\n comments= record['comments']\n\n ### load some header info from the mophead file\n headers={}\n for exp in exps:\n f = pyfits.open(exp+\".fits\")\n headers[exp]={}\n for key in ['MJDATE', 'NAXIS1', 'NAXIS2', 'EXPTIME', 'FILTER']:\n headers[exp][key]=f[0].header[key]\n headers[exp]['MJD-OBSC']=headers[exp]['MJDATE']+headers[exp]['EXPTIME']/2.0/3600.0/24.0\n f.close()\n\n import math,os\n real=0\n cand_total=len(cands)\n if cand_total > 100 :\n sys.stderr.write(\"Too many candidates (%d) skipping this chip...\\n\" % ( cand_total))\n\treturn -1\n for cand in cands:\n cand_total=cand_total-1\n x2=[]\n y2=[]\n y1=[]\n x1=[]\n for i in range(len(exps)):\n\t #print exps[i]\n fileId=exps[i]\n x2.append(int(min(math.floor(cand[i]['x'])+width,headers[fileId]['NAXIS1'])))\n y2.append(int(min(math.floor(cand[i]['y'])+width,headers[fileId]['NAXIS2'])))\n x1.append(int(max(math.floor(cand[i]['x'])-width,1)))\n y1.append(int(max(math.floor(cand[i]['y'])-width,1)))\n\t#print exps\n\t#print x1,x2\n\t#print y1,y2\n x_1 = min(x1)\n y_1 = min(y1)\n x_2 = max(x2)\n y_2 = max(y2)\n\tans='f'\n\tfake=True\n\txshifts=[]\n\tyshifts=[]\n\tfirst_pass=True\n\tshow_coords=True\n\tans='j'\n\twhile ( first_pass or ans == 'f' or ans == 'c') : \n\t first_pass=False\n\t if ans=='c':\n\t if show_coords: \n\t show_coords=False\n else:\n\t show_coords=True\n\t if ans=='f':\n\t if fake:\n\t fake=False\n\t else:\n\t fake=True\n for i in range(len(exps)):\n xshift=cand[i]['x']-cand[i]['x_0']\n yshift=cand[i]['y']-cand[i]['y_0']\n\t allmark=open('all.coo','w')\n\t for j in range(len(exps)):\n\t if j==i :\n\t continue\n allmark.write('%f %f\\n' % ( cand[j]['x_0']+xshift, cand[j]['y_0']+yshift))\n\t allmark.close()\n tvmark=open('tv.coo','w')\n tvmark.write('%f %f %d\\n' % ( cand[i]['x'], cand[i]['y'], cand_total))\n x1=max(x_1 + xshift,1)\n y1=max(y_1 + yshift,1)\n x2=min(x_2 + xshift,headers[exps[i]]['NAXIS1'])\n y2=min(y_2 + yshift,headers[exps[i]]['NAXIS2'])\n cutout = \"[%d:%d,%d:%d]\" % (x1,x2,y1,y2)\n\t if not fake:\n\t if exps[i][0:2]=='fk' :\n\t fileId=exps[i][2:]\n\t else:\n fileId=exps[i]\n else:\n fileId=exps[i]\n#\t sys.stdout.write(\"---> \",fileId+cutout,xshift,yshift\n try:\n junk=display(fileId+cutout,i+1,Stdout=1)\n except:\n sys.stderr.write(\"ERROR\\n\")\n tvmark.close()\n\t if show_coords:\n pyraf.iraf.images.tv.tvmark(i+1,'all.coo',mark='circle',radii=10,color=205)\n pyraf.iraf.images.tv.tvmark(i+1,'tv.coo',mark='circle',radii=8,color=204,label='yes',nxoffset=10)\n os.unlink('tv.coo')\n\t os.unlink('all.coo')\n ### ask if this is a real candidate, take action.\n ans='j'\n while ans not in ('y', 'n', 'q', 's', 'f', 'c'):\n ans=raw_input(\"[%d] (y,n,s,q,c,f) : \" % ( cand_total))\n if ans=='y':\n\t if os.access(\"cands.REAL\",os.W_OK):\n f=open(\"cands.REAL\",\"a\")\n \t else:\n f=open(\"cands.REAL\",\"w\")\n\t for comment in comments:\n\t\t f.write(comment)\n cols=['x','y','x_0','y_0','flux','size','max_int','elon']\n #for col in cols:\n # f.write(\"%s\\t\" % col)\n f.write(\"\\n\")\n for ii in range(len(exps)):\n for col in cols:\n f.write(\"%8.2f\\t\" % cand[ii][col])\n f.write(\"\\n\")\n f.close()\n real=real+1\n if ans=='q':\n return -2\n\tif ans=='s':\n\t return -1\n return real"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef is_up(coordinate, current_time):\n cfht.date = current_time.iso.replace('-', '/')\n cfht.horizon = math.radians(-7)\n sun.compute(cfht)\n sun_rise = Time(str(sun.rise_time).replace('/', '-'))\n sun_set = Time(str(sun.set_time).replace('/', '-'))\n\n if current_time < sun_set or current_time > sun_rise:\n return False\n\n fb._ra = coordinate.ra.radian\n fb._dec = coordinate.dec.radian\n cfht.horizon = math.radians(40)\n fb.compute(cfht)\n fb_rise_time = Time(str(fb.rise_time).replace('/', '-'))\n fb_set_time = Time(str(fb.set_time).replace('/', '-'))\n\n if (current_time > fb_set_time > fb_set_time or\n fb_rise_time > current_time > fb_set_time):\n return False\n return True", "response": "Given the position and time determin if the given target is up."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef TAPQuery(query):\n\n tapURL = \"http://cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/sync\"\n\n\n ## Some default parameters for that TAP service queries.\n tapParams={'REQUEST': 'doQuery',\n 'LANG': 'ADQL',\n 'FORMAT': 'votable',\n 'QUERY': query}\n cnt=0\n while True:\t\n try:\n print \"running query\"\n r=urllib2.urlopen(tapURL,urllib.urlencode(tapParams))\n return r\n except urllib2.HTTPError, e:\n cnt+=1\n if e.code!=503:\n sys.stderr.write(\"# TAP Query got Code: %s Attempt: %d (exiting)\\n\" % (str(e.code),cnt))\n sys.exit(-1)\n sys.stderr.write(\"# TAP Query got Code: %s Attempt: %d (sleeping for 10)\\n\" % (str(e.code),cnt))\n time.sleep(10)", "response": "The main function of the TAP query."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nruns the actual step1jmp and matt codes.", "response": "def step1(expnum,\n ccd,\n prefix='',\n version='p',\n sex_thresh=_SEX_THRESHOLD,\n wave_thresh=_WAVE_THRESHOLD,\n maxcount=_MAX_COUNT,\n dry_run=False):\n \"\"\"run the actual step1jmp/matt codes.\n\n expnum: the CFHT expousre to process\n ccd: which ccd in the mosaic to process\n fwhm: the image quality, FWHM, of the image. In pixels.\n sex_thresh: the detection threhold to run sExtractor at\n wave_thresh: the detection threshold for wavelet\n maxcount: saturation level\n\n \"\"\"\n\n storage.get_file(expnum, ccd, prefix=prefix, version=version, ext='mopheader')\n filename = storage.get_image(expnum, ccd, version=version, prefix=prefix)\n fwhm = storage.get_fwhm(expnum, ccd, prefix=prefix, version=version)\n basename = os.path.splitext(filename)[0]\n\n logging.info(util.exec_prog(['step1jmp',\n '-f', basename,\n '-t', str(wave_thresh),\n '-w', str(fwhm),\n '-m', str(maxcount)]))\n\n obj_uri = storage.get_uri(expnum, ccd, version=version, ext='obj.jmp',\n prefix=prefix)\n obj_filename = basename + \".obj.jmp\"\n\n if not dry_run:\n storage.copy(obj_filename, obj_uri)\n\n ## for step1matt we need the weight image\n hdulist = fits.open(filename)\n flat_name = hdulist[0].header.get('FLAT', 'weight.fits')\n parts = os.path.splitext(flat_name)\n if parts[1] == '.fz':\n flat_name = os.path.splitext(parts[0])[0]\n else:\n flat_name = parts[0]\n try:\n flat_filename = storage.get_image(flat_name, ccd, version='', ext='fits', subdir='calibrators')\n except:\n flat_filename = storage.get_image(flat_name, ccd, version='', ext='fits', subdir='old_calibrators')\n\n if os.access('weight.fits', os.R_OK):\n os.unlink('weight.fits')\n\n if not os.access('weight.fits', os.R_OK):\n os.symlink(flat_filename, 'weight.fits')\n\n logging.info(util.exec_prog(['step1matt',\n '-f', basename,\n '-t', str(sex_thresh),\n '-w', str(fwhm),\n '-m', str(maxcount)]))\n\n if os.access('weight.fits', os.R_OK):\n os.unlink('weight.fits')\n\n obj_uri = storage.get_uri(expnum, ccd, version=version, ext='obj.matt',\n prefix=prefix)\n obj_filename = basename + \".obj.matt\"\n\n if not dry_run:\n storage.copy(obj_filename, obj_uri)\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_json(self):\n viom_table = self.get_basic_json()\n if self.slots:\n viom_table['Slots'] = {\n 'Slot': [s.get_json() for s in self.slots.values()]\n }\n if self.manage:\n viom_table['VIOMManage'] = self.manage.get_json()\n return viom_table", "response": "Create JSON data for AdapterConfig.\n "} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_json(self):\n json = self.get_basic_json()\n if self.onboard_cards:\n json['OnboardControllers'] = {\n 'OnboardController':\n [c.get_json() for c in self.onboard_cards.values()]\n }\n if self.addon_cards:\n json['AddOnCards'] = {\n 'AddOnCard': [c.get_json() for c in self.addon_cards.values()]\n }\n return json", "response": "Create JSON data for the current object."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate JSON data for LANPort.", "response": "def get_json(self):\n \"\"\"Create JSON data for LANPort.\n\n :returns: JSON data as follows:\n\n {\n \"@PortIdx\":1,\n \"PortEnable\":{\n },\n \"UseVirtualAddresses\":{\n },\n \"BootProtocol\":{\n },\n \"VirtualAddress\":{\n \"MAC\":{\n }\n },\n \"BootPriority\":{\n },\n \"ISCSIBootEnvironment\":{\n }\n }\n \"\"\"\n\n port = self.get_basic_json()\n port.update({\n 'BootProtocol': self.boot.BOOT_PROTOCOL,\n 'BootPriority': self.boot.boot_prio,\n })\n boot_env = self.boot.get_json()\n if boot_env:\n port.update(boot_env)\n if self.use_virtual_addresses and self.mac:\n port['VirtualAddress'] = {'MAC': self.mac}\n return port"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate FC port. :returns: JSON for FC port as follows: { \"@PortIdx\":1, \"PortEnable\":{ }, \"UseVirtualAddresses\":{ }, \"VirtualAddress\":{ \"WWNN\":{ }, \"WWPN\":{ }, \"MAC\":{ } }, \"BootProtocol\":{ }, \"BootPriority\":{ }, \"FCBootEnvironment\":{ } }", "response": "def get_json(self):\n \"\"\"Create FC port.\n\n :returns: JSON for FC port as follows:\n {\n \"@PortIdx\":1,\n \"PortEnable\":{\n },\n \"UseVirtualAddresses\":{\n },\n \"VirtualAddress\":{\n \"WWNN\":{\n },\n \"WWPN\":{\n },\n \"MAC\":{\n }\n },\n \"BootProtocol\":{\n },\n \"BootPriority\":{\n },\n \"FCBootEnvironment\":{\n }\n }\n\n \"\"\"\n port = self.get_basic_json()\n port.update({\n 'BootProtocol': self.boot.BOOT_PROTOCOL,\n 'BootPriority': self.boot.boot_prio,\n })\n boot_env = self.boot.get_json()\n if boot_env:\n port.update(boot_env)\n if self.use_virtual_addresses:\n addresses = {}\n if self.wwnn:\n addresses['WWNN'] = self.wwnn\n if self.wwpn:\n addresses['WWPN'] = self.wwpn\n if addresses:\n port['VirtualAddress'] = addresses\n return port"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_json(self):\n port = self.get_basic_json()\n port['Functions'] = {\n 'Function': [f.get_json() for f in self.functions.values()]\n }\n return port", "response": "Create JSON for CNA port."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating JSON for FCBootEnvironment.", "response": "def get_json(self):\n \"\"\"Create JSON for FCBootEnvironment.\n\n :returns: JSON for FCBootEnvironment as follows:\n\n {\n \"FCBootEnvironment\":{\n \"FCTargets\":{\n \"FCTarget\":[\n ]\n },\n \"FCLinkSpeed\":{\n },\n \"SANBootEnable\":{\n },\n \"FCTopology\":{\n }\n }\n }\n \"\"\"\n json = self.get_basic_json()\n for i in range(len(self.targets)):\n # @FCTargetIdx starts from 1.\n self.targets[i].set_index(i + 1)\n json['FCTargets'] = {\n 'FCTarget': [t.get_json() for t in self.targets]\n }\n return {'FCBootEnvironment': json}"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_json(self):\n if self.dhcp_usage:\n return {'DHCPUsage': self.dhcp_usage,\n 'Name': self.iqn}\n else:\n return self.get_basic_json()", "response": "Create JSON data for iSCSI initiator."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating JSON data for iSCSI target.", "response": "def get_json(self):\n \"\"\"Create JSON data for iSCSI target.\n\n :returns: JSON data for iSCSI target as follows:\n\n {\n \"DHCPUsage\":{\n },\n \"Name\":{\n },\n \"IPv4Address\":{\n },\n \"PortNumber\":{\n },\n \"BootLUN\":{\n },\n \"AuthenticationMethod\":{\n },\n \"ChapUserName\":{\n },\n \"ChapSecret\":{\n },\n \"MutualChapSecret\":{\n }\n }\n \"\"\"\n json = {\n 'DHCPUsage': self.dhcp_usage,\n 'AuthenticationMethod': self.auth_method,\n }\n if not self.dhcp_usage:\n json['Name'] = self.iqn\n json['IPv4Address'] = self.ip\n json['PortNumber'] = self.port\n json['BootLUN'] = self.lun\n if self.chap_user:\n json['ChapUserName'] = self.chap_user\n if self.chap_secret:\n json['ChapSecret'] = self.chap_secret\n if self.mutual_chap_secret:\n json['MutualChapSecret'] = self.mutual_chap_secret\n return json"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nscale the data cube into the range 0 - 255", "response": "def zscale(data,contrast,min=100,max=60000):\n\t\"\"\"Scale the data cube into the range 0-255\"\"\"\n\n\t## pic 100 random elements along each dimension\n\t## use zscale (see the IRAF display man page or\n\t## http://iraf.net/article.php/20051205162333315\n\n\n\timport random\n\tx=[]\n\tfor i in random.sample(xrange(data.shape[0]),50):\n\t for j in random.sample(xrange(data.shape[1]),50):\n\t \tx.append(data[i,j])\n\t\n\tyl=numarray.sort(numarray.clip(x,min,max))\n\tn=len(yl)\n ym=sum(yl)/float(n)\n\n xl=numarray.array(range(n))\n \txm=sum(xl)/float(n)\t\n\n\n\tss_xx=sum((xl-xm)*(xl-xm))\n\tss_yy=sum((yl-ym)*(yl-ym))\n ss_xy=sum((xl-xm)*(yl-ym))\n\tb=ss_xy/ss_xx\n\ta=ym-b*xm\n\n\tz1=yl[n/2] + (b/contrast)*(1-n/2)\n\tz2=yl[n/2] + (b/contrast)*(n-n/2)\n\n\t## Now put the data inbetween Z1 and Z2\n\thigh=data-z1\n\tz2=z2-z1\n\thigh=numarray.clip(high,0,z2)\n\n\t## and change that to 0-255\n\thigh= 256-256*high/z2\n\n\t### send back the scalled data\n\treturn high"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nupdating a row in ossuary with", "response": "def update_values(ims, image_id, iq_zeropt=True, comment=False, snr=False, commdict=None):\n \"\"\"\n Update a row in ossuary with\n :param ims: an ImageQuery, contains image table and a connector\n :param image_id: the primary key of the row to be updated\n :param iq_zeropt: Keyword set if iq and zeropoint are to be checked for updating\n :param comment: Keyword set if image is to have a comment of Stephen's added\n :param commdict: The dictionary parsed from Stephen's file of comments\n :return: No return, just updates ossuary.\n \"\"\"\n updating_params = {}\n if iq_zeropt:\n updating_params = get_iq_and_zeropoint(image_id, {})\n if comment:\n updating_params = {'comment': commdict[str(image_id)]}\n if snr:\n updating_params = get_snr(image_id, {})\n ss = ims.images.update(ims.images.c.image_id == image_id)\n ims.conn.execute(ss, updating_params)\n\n return"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nupdates the ossuary Postgres db with images observed for OSSOS. iq: Go through and check all ossuary's images for new existence of IQs/zeropoints. comment: Go through all ossuary and Then updates ossuary with new images that are at any stage of processing. Constructs full image entries, including header and info in the vtags, and inserts to ossuary. TODO: a CLUSTER after data is inserted - maybe once a week, depending how much there is CLUSTER images; need to sqlalchemy this one", "response": "def main():\n \"\"\"\n Update the ossuary Postgres db with images observed for OSSOS.\n iq: Go through and check all ossuary's images for new existence of IQs/zeropoints.\n comment: Go through all ossuary and\n Then updates ossuary with new images that are at any stage of processing.\n Constructs full image entries, including header and info in the vtags, and inserts to ossuary.\n\n TODO: a CLUSTER after data is inserted - maybe once a week, depending how much there is\n CLUSTER images; need to sqlalchemy this one\n \"\"\"\n parser = argparse.ArgumentParser()\n\n parser.add_argument(\"-iq\", \"--iq\", action=\"store_true\",\n help=\"Check existing images in ossuary that do not yet have \"\n \"IQ/zeropoint information; update where possible.\")\n parser.add_argument(\"-comment\", action=\"store_true\",\n help=\"Add comments on images provided by S. Gwyn to database.\")\n parser.add_argument(\"-snr\", action=\"store_true\",\n help=\"Update existing images in ossuary for SNR info where that exists in a vtag.\")\n args = parser.parse_args()\n\n images = web.field_obs.queries.ImagesQuery()\n processed_images, iqs = retrieve_processed_images(images) # straight list of primary keys\n commdict = parse_sgwn_comments()\n\n if args.iq:\n unmeasured_iqs = iq_unmeasured_images(images)\n sys.stdout.write('%d images in ossuary; updating %d with new IQ/zeropoint information.\\n' %\n (len(processed_images), len(unmeasured_iqs)))\n for n, image in enumerate(unmeasured_iqs): # it's in the db, so has already passed the other checks\n update_values(images, image)\n sys.stdout.write('%s %d/%d...ossuary updated.\\n' % (image, n + 1, len(unmeasured_iqs)))\n\n if args.snr:\n unmeasured = snr_unmeasured_images(images)\n sys.stdout.write('%d images in ossuary; updating %d with new SNR information.\\n' %\n (len(processed_images), len(unmeasured)))\n for n, image in enumerate(unmeasured): # it's in the db, so has already passed the other checks\n update_values(images, image, iq_zeropt=False, snr=True)\n sys.stdout.write('%s %d/%d...ossuary updated.\\n' % (image, n + 1, len(unmeasured)))\n\n if args.comment:\n sys.stdout.write('%d images in ossuary; updating with new comment information.\\n' %\n len(processed_images))\n for image in commdict.keys():\n if int(image) in processed_images:\n update_values(images, image, iq_zeropt=False, comment=True, commdict=commdict)\n sys.stdout.write('%s has comment...\\n' % image)\n\n unprocessed_images = parse_unprocessed_images(storage.list_dbimages(), processed_images)\n sys.stdout.write('%d images in ossuary; updating with %d new in VOspace.\\n' %\n (len(processed_images), len(unprocessed_images)))\n\n for n, image in enumerate(unprocessed_images):\n sys.stdout.write('%s %d/%d ' % (image, n + 1, len(unprocessed_images)))\n try:\n subheader, fullheader = get_header(image)\n if subheader is not None:\n sys.stdout.write('Header obtained. ')\n verify_ossos_image(fullheader)\n header = get_iq_and_zeropoint(image, subheader)\n header = get_snr(image, header)\n if image in commdict.keys():\n header['comment'] = commdict[image]\n put_image_in_database(header, images)\n sys.stdout.write('...added to ossuary...\\n')\n # generate_MegaCam_previews(image)\n # sys.stdout.write(' .gif preview saved.\\n')\n else:\n sys.stdout.write('Header is not available: skipping.\\n')\n except Exception, e:\n sys.stdout.write('... %s\\n' % e)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef square(self, n_coeffs, do_overlap_add=False):\n resampled_bands = [\n self._resample(band, n_coeffs)\n for band in self.iter_bands()]\n\n stacked = np.vstack(resampled_bands).T\n\n fdim = FrequencyDimension(self.scale)\n\n # TODO: This feels like it could be wrapped up nicely elsewhere\n chunk_frequency = Picoseconds(int(np.round(\n self.time_dimension.duration / Picoseconds(1) / n_coeffs)))\n\n td = TimeDimension(frequency=chunk_frequency)\n\n arr = ConstantRateTimeSeries(ArrayWithUnits(\n stacked.reshape(-1, n_coeffs, self.n_bands),\n dimensions=[self.time_dimension, td, fdim]))\n\n if not do_overlap_add:\n return arr\n\n # Begin the overlap add procedure\n overlap_ratio = self.time_dimension.overlap_ratio\n\n if overlap_ratio == 0:\n # no overlap add is necessary\n return ArrayWithUnits(stacked, [td, fdim])\n\n step_size_samples = int(n_coeffs * overlap_ratio)\n\n first_dim = int(np.round(\n (stacked.shape[0] * overlap_ratio) + (n_coeffs * overlap_ratio)))\n\n output = ArrayWithUnits(\n np.zeros((first_dim, self.n_bands)),\n dimensions=[td, fdim])\n\n for i, chunk in enumerate(arr):\n start = step_size_samples * i\n stop = start + n_coeffs\n output[start: stop] += chunk.reshape((-1, self.n_bands))\n\n return output", "response": "Compute a square view of the frequency adaptive transform by resampling each frequency band and performing an overlap - add procedure in the frequency adaptive transform by the number of samples."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef convert(self, point):\n x, y = point\n (x1, y1) = x - self.x_offset, y - self.y_offset\n logger.debug(\"converted {} {} ==> {} {}\".format(x, y, x1, y1))\n return x1, y1", "response": "Convert a point from one coordinate system to another."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef simple_settings(cls):\n\n class Settings(ff.PersistenceSettings):\n _id = cls.__name__\n id_provider = ff.StaticIdProvider(_id)\n key_builder = ff.StringDelimitedKeyBuilder()\n database = ff.FileSystemDatabase(\n path=_id, key_builder=key_builder, createdirs=True)\n\n class Model(cls, Settings):\n pass\n\n Model.__name__ = cls.__name__\n Model.__module__ = cls.__module__\n return Model", "response": "Create sane default persistence settings for learning pipelines\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\napply a pytorch network potentially in chunks", "response": "def apply_network(network, x, chunksize=None):\n \"\"\"\n Apply a pytorch network, potentially in chunks\n \"\"\"\n network_is_cuda = next(network.parameters()).is_cuda\n\n x = torch.from_numpy(x)\n\n with torch.no_grad():\n if network_is_cuda:\n x = x.cuda()\n\n if chunksize is None:\n return from_var(network(x))\n\n return np.concatenate(\n [from_var(network(x[i: i + chunksize]))\n for i in range(0, len(x), chunksize)])"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sample_norm(x):\n original = x\n # square\n x = x ** 2\n # feature-map-wise sum\n x = torch.sum(x, dim=1)\n # scale by number of feature maps\n x *= 1.0 / original.shape[1]\n x += 10e-8\n x = torch.sqrt(x)\n return original / x.view(-1, 1, x.shape[-1])", "response": "Sample the norm of a pixel."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nresetting the source location based on the init_skycoord values", "response": "def reset_coord(self):\n \"\"\"\n Reset the source location based on the init_skycoord values\n @return:\n \"\"\"\n (x, y, idx) = self.world2pix(self.init_skycoord.ra,\n self.init_skycoord.dec,\n usepv=True)\n self.update_pixel_location((x, y), idx)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef pixel_coord(self):\n return self.get_pixel_coordinates(self.reading.pix_coord, self.reading.get_ccd_num())", "response": "Return the coordinates of the source in the cutout reference frame."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef flip_flip(self, point):\n x, y = point\n if self.reading.compute_inverted():\n naxis1 = self.reading.obs.header.get(self.reading.obs.HEADER_IMG_SIZE_X,\n self.reading.obs.header.get('NAXIS1', 256))\n naxis2 = self.reading.obs.header.get(self.reading.obs.HEADER_IMG_SIZE_Y,\n self.reading.obs.header.get('NAXIS2', 256))\n logger.debug(\"Got that image has size {}, {}\".format(naxis1, naxis2))\n\n x1 = x\n y1 = y\n x = int(naxis1) - x + 1\n y = int(naxis2) - y + 1\n logger.debug(\"Inverted source location from {},{} to {},{}\".format(x1, y1, x, y))\n\n return x, y", "response": "Flips the X and Y coordinates of the image."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nupdate the x y location of the associated reading by using new_pixel_location and transforming from cutoutPixelReferenceFrame to observation refrence frame.", "response": "def update_pixel_location(self, new_pixel_location, hdu_index):\n \"\"\"\n Update the x/y location of the associated reading by using new_pixel_location and transforming from cutout\n reference frame to observation refrence frame.\n @param new_pixel_location: (x, y) location of the source in cutout pixel reference frame\n @param hdu_index: index of the associate hdu in the hdulist for this cutout\n \"\"\"\n self.reading.pix_coord = self.get_observation_coordinates(new_pixel_location[0],\n new_pixel_location[1],\n hdu_index)\n new_ccd = self.get_ccdnum(hdu_index)\n if new_ccd != self.reading.get_ccd_num():\n self._apcor = None\n self._zmag = None\n self.reading.obs.ccdnum = self.get_ccdnum(hdu_index)\n self.reading.sky_coord = self.pix2world(new_pixel_location[0],\n new_pixel_location[1],\n hdu_index)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nretrieving the pixel location of a point within the current HDUList given the location in the original FITS image. This takes into account that the image may be a cutout of a larger original. Args: point: tuple(float, float) (x, y) in original. Returns: (x, y) pixel in this image. @param extno: the extno from the original Mosaic that the x/y coordinates are from.", "response": "def get_pixel_coordinates(self, point, ccdnum):\n \"\"\"\n Retrieves the pixel location of a point within the current HDUList given the\n location in the original FITS image. This takes into account that\n the image may be a cutout of a larger original.\n\n Args:\n point: tuple(float, float)\n (x, y) in original.\n\n Returns:\n (x, y) pixel in this image.\n @param extno: the extno from the original Mosaic that the x/y coordinates are from.\n \"\"\"\n hdulist_index = self.get_hdulist_idx(ccdnum)\n if isinstance(point[0], Quantity) and isinstance(point[1], Quantity):\n pix_point = point[0].value, point[1].value\n else:\n pix_point = point\n if self.reading.inverted:\n pix_point = self.reading.obs.naxis1 - pix_point[0] +1 , self.reading.obs.naxis2 - pix_point[1] + 1\n\n (x, y) = self.hdulist[hdulist_index].converter.convert(pix_point)\n return x, y, hdulist_index"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the location of a point using the coordinate system of the original image before any cutouts were done.", "response": "def get_observation_coordinates(self, x, y, hdulist_index):\n \"\"\"\n Retrieves the location of a point using the coordinate system of\n the original observation, i.e. the original image before any\n cutouts were done.\n\n Returns:\n (x, y) in the original image coordinate system.\n @param x: x-pixel location in the cutout frame of reference\n @param y: y-pixel location in the cutout frame of reference\n @param idx: index of hdu in hdulist that the given x/y corresponds to.\n \"\"\"\n return self.hdulist[hdulist_index].converter.get_inverse_converter().convert((x, y))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef world2pix(self, ra, dec, usepv=True):\n x, y, idx = (0, 0, 0)\n for idx in range(1, len(self.hdulist)):\n hdu = self.hdulist[idx]\n x, y = hdu.wcs.sky2xy(ra, dec, usepv=usepv)\n if 0 < x < hdu.header['NAXIS1'] and 0 < y < hdu.header['NAXIS2']:\n # print \"Inside the frame.\"\n return x, y, idx\n return x, y, idx", "response": "Convert a given RA and DEC position to the X Y coordinates of the extension in the cutout reference frame."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef zmag(self,):\n if self._zmag is None:\n hdulist_index = self.get_hdulist_idx(self.reading.get_ccd_num())\n self._zmag = self.hdulist[hdulist_index].header.get('PHOTZP', 0.0)\n return self._zmag", "response": "Return the photometric zeropoint of the CCD associated with the reading."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef apcor(self):\n if self._apcor is None:\n try:\n self._apcor = Downloader().download_apcor(self.reading.get_apcor_uri())\n except:\n self._apcor = ApcorData.from_string(\"5 15 99.99 99.99\")\n return self._apcor", "response": "Returns the aperture correction of for the CCD associated with the reading."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_observed_magnitude(self, centroid=True):\n # NOTE: this import is only here so that we don't load up IRAF\n # unnecessarily (ex: for candidates processing).\n \"\"\"\n Get the magnitude at the current pixel x/y location.\n\n :return: Table\n \"\"\"\n\n max_count = float(self.astrom_header.get(\"MAXCOUNT\", 30000))\n (x, y, hdulist_index) = self.pixel_coord\n tmp_file = self._hdu_on_disk(hdulist_index)\n try:\n from ossos import daophot\n phot = daophot.phot_mag(tmp_file,\n x, y,\n aperture=self.apcor.aperture,\n sky=self.apcor.sky,\n swidth=self.apcor.swidth,\n apcor=self.apcor.apcor,\n zmag=self.zmag,\n maxcount=max_count, extno=1,\n centroid=centroid)\n if not self.apcor.valid:\n phot['PIER'][0] = 1\n return phot\n except Exception as ex:\n print ex\n raise ex\n finally:\n self.close()", "response": "Get the observed magnitude at the current pixel x y location."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef comparison_image_list(self):\n\n if self._comparison_image_list is not None:\n return self._comparison_image_list\n\n ref_ra = self.reading.ra * units.degree\n ref_dec = self.reading.dec * units.degree\n radius = self.radius is not None and self.radius or config.read('CUTOUTS.SINGLETS.RADIUS') * units.arcminute\n print(\"Querying CADC for list of possible comparison images at RA: {}, DEC: {}, raidus: {}\".format(ref_ra,\n ref_dec,\n radius))\n query_result = storage.cone_search(ref_ra, ref_dec, radius, radius) # returns an astropy.table.table.Table\n print(\"Got {} possible images\".format(len(query_result)))\n ans = raw_input(\"Do you want to lookup IQ? (y/n)\")\n print(\"Building table for presentation and selection\")\n if ans == \"y\":\n print(\"Including getting fwhm which is a bit slow.\")\n comparison_image_list = []\n if len(query_result['collectionID']) > 0: # are there any comparison images even available on that sky?\n index = -1\n for row in query_result:\n expnum = row['collectionID']\n if expnum in self._bad_comparison_images:\n continue\n date = Time(row['mjdate'], format='mjd').mpc\n if Time(row['mjdate'], format='mjd') < Time('2013-01-01 00:00:00', format='iso'):\n continue\n exptime = row['exptime']\n if float(exptime) < 250:\n continue\n filter_name = row['filter']\n if 'U' in filter_name:\n continue\n if filter_name.lower() in self.hdulist[-1].header['FILTER'].lower():\n filter_name = \"* {:8s}\".format(filter_name)\n fwhm = -1.0\n if ans == 'y':\n try:\n fwhm = \"{:5.2f}\".format(float(storage.get_fwhm_tag(expnum, 22)))\n except:\n pass\n index += 1\n comparison_image_list.append([index, expnum, date, exptime, filter_name, fwhm, None])\n self._comparison_image_list = Table(data=numpy.array(comparison_image_list),\n names=[\"ID\", \"EXPNUM\", \"DATE-OBS\", \"EXPTIME\", \"FILTER\", \"FWHM\", \"REFERENCE\"])\n return self._comparison_image_list", "response": "Returns a list of possible comparison images for the current cutout."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef retrieve_comparison_image(self):\n # selecting comparator when on a comparator should load a new one.\n collectionID = self.comparison_image_list[self.comparison_image_index]['EXPNUM']\n ref_ra = self.reading.ra * units.degree\n ref_dec = self.reading.dec * units.degree\n radius = self.radius is not None and self.radius or config.read('CUTOUTS.SINGLETS.RADIUS') * units.arcminute\n try:\n comparison = collectionID\n hdu_list = storage.ra_dec_cutout(storage.dbimages_uri(comparison),\n SkyCoord(ref_ra, ref_dec), radius)\n ccd = int(hdu_list[-1].header.get('EXTVER', 0))\n obs = Observation(str(comparison), 'p',\n ccdnum=ccd)\n x = hdu_list[-1].header.get('NAXIS1', 0) // 2.0\n y = hdu_list[-1].header.get('NAXIS2', 0) // 2.0\n ref_ra = self.reading.ra * units.degree\n ref_dec = self.reading.dec * units.degree\n reading = SourceReading(x, y, self.reading.x, self.reading.y,\n ref_ra, ref_dec, self.reading.x, self.reading.y, obs)\n self.comparison_image_list[self.comparison_image_index][\"REFERENCE\"] = SourceCutout(reading, hdu_list)\n except Exception as ex:\n print traceback.format_exc()\n print ex\n print \"Failed to load comparison image;\"\n self.comparison_image_index = None\n logger.error(\"{} {}\".format(type(ex), str(ex)))\n logger.error(traceback.format_exc())", "response": "Retrieves the comparison image for this cutout."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef check_password(self, passwd, group):\n # This provides authentication via CADC.\n \"\"\"check that the passwd provided matches the required password.\"\"\"\n return gms.isMember(self.login, passwd, group)", "response": "check that the passwd provided matches the required password"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ntransform a TimeSlice into an integer slice that numpy can work with.", "response": "def integer_based_slice(self, ts):\n \"\"\"\n Transform a :class:`TimeSlice` into integer indices that numpy can work\n with\n\n Args:\n ts (slice, TimeSlice): the time slice to translate into integer\n indices\n \"\"\"\n\n if isinstance(ts, slice):\n try:\n start = Seconds(0) if ts.start is None else ts.start\n if start < Seconds(0):\n start = self.end + start\n stop = self.end if ts.stop is None else ts.stop\n if stop < Seconds(0):\n stop = self.end + stop\n duration = stop - start\n ts = TimeSlice(start=start, duration=duration)\n except (ValueError, TypeError):\n pass\n\n if not isinstance(ts, TimeSlice):\n return ts\n\n diff = self.duration - self.frequency\n start_index = \\\n max(0, np.floor((ts.start - diff) / self.frequency))\n end = self.end if ts.duration is None else ts.end\n\n\n # KLUDGE: This is basically arbitrary, but the motivation is that we'd\n # like to differentiate between cases where the slice\n # actually/intentionally overlaps a particular sample, and cases where\n # the slice overlaps the sample by a tiny amount, due to rounding or\n # lack of precision (e.g. Seconds(1) / SR44100().frequency).\n ratio = np.round(end / self.frequency, 2)\n\n\n stop_index = np.ceil(ratio)\n return slice(int(start_index), int(stop_index))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef load_observations((observations, regex, rename), path, filenames):\n\n for filename in filenames:\n if re.search(regex, filename) is None:\n logging.error(\"Skipping {}\".format(filename))\n continue\n obs = mpc.MPCReader().read(os.path.join(path, filename))\n for ob in obs:\n if \"568\" not in ob.observatory_code:\n continue\n if not isinstance(ob.comment, mpc.OSSOSComment):\n continue\n if ob.date < Time(\"2013-01-01 00:00:00\"):\n continue\n if rename:\n new_provisional_name = os.path.basename(filename)\n new_provisional_name = new_provisional_name[0:new_provisional_name.find(\".\")]\n rename_map[ob.provisional_name] = new_provisional_name\n try:\n key1 = ob.comment.frame.split('p')[0]\n except Exception as ex:\n logger.warning(str(ex))\n logger.warning(ob.to_string())\n continue\n key2 = ob.provisional_name\n if key1 not in observations:\n observations[key1] = {}\n if key2 in observations[key1]:\n if observations[key1][key2]:\n continue\n if not observation.null_observation:\n logger.error(filename)\n logger.error(str(observations[key1][key2]))\n raise ValueError(\"conflicting observations for {} in {}\".format(key2, key1))\n observations[key1][key2] = ob", "response": "Loads the observations of the object."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef plant_kbos(filename, psf, kbos, shifts, prefix):\n\n iraf.set(uparm=\"./\")\n iraf.digiphot()\n iraf.apphot()\n iraf.daophot(_doprint=0)\n iraf.images()\n\n if shifts['nmag'] < 4:\n logging.warning(\"Mag shift based on fewer than 4 common stars.\")\n fd = open(\"plant.WARNING\", 'a')\n fd.write(\"Mag shift based on fewer than 4 common stars.\")\n fd.close()\n\n if shifts['emag'] > 0.05:\n logging.warning(\"Mag shift has large uncertainty.\")\n fd = open(\"plant.WARNING\", 'a')\n fd.write(\"Mag shift hsa large uncertainty.\")\n fd.close()\n\n addstar = tempfile.NamedTemporaryFile(suffix=\".add\")\n\n # transform KBO locations to this frame using the shifts provided.\n w = get_wcs(shifts)\n\n header = fits.open(filename)[0].header\n\n # set the rate of motion in units of pixels/hour instead of ''/hour\n scale = header['PIXSCAL1']\n rate = kbos['sky_rate']/scale\n\n # compute the location of the KBOs in the current frame.\n\n # offset magnitudes from the reference frame to the current one.\n mag = kbos['mag'] - shifts['dmag']\n angle = radians(kbos['angle'])\n\n # Move the x/y locations to account for the sky motion of the source.\n x = kbos['x'] - rate*24.0*shifts['dmjd']*cos(angle)\n y = kbos['y'] - rate*24.0*shifts['dmjd']*sin(angle)\n x, y = w.wcs_world2pix(x, y, 1)\n\n # Each source will be added as a series of PSFs so that a new PSF is added for each pixel the source moves.\n itime = float(header['EXPTIME'])/3600.0\n npsf = fabs(rint(rate * itime)) + 1\n mag += 2.5*log10(npsf)\n dt_per_psf = itime/npsf\n\n # Build an addstar file to be used in the planting of source.\n idx = 0\n for record in transpose([x, y, mag, npsf, dt_per_psf, rate, angle]):\n x = record[0]\n y = record[1]\n mag = record[2]\n npsf = record[3]\n dt = record[4]\n rate = record[5]\n angle = record[6]\n for i in range(int(npsf)):\n idx += 1\n x += dt*rate*math.cos(angle)\n y += dt*rate*math.sin(angle)\n addstar.write(\"{} {} {} {}\\n\".format(x, y, mag, idx))\n\n addstar.flush()\n fk_image = prefix+filename\n try:\n os.unlink(fk_image)\n except OSError as err:\n if err.errno == errno.ENOENT:\n pass\n else:\n raise\n\n # add the sources to the image.\n iraf.daophot.addstar(filename, addstar.name, psf, fk_image,\n simple=True, verify=False, verbose=False)\n # convert the image to short integers.\n iraf.images.chpix(fk_image, fk_image, 'ushort')", "response": "Add KBOs to an image."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef plant(expnums, ccd, rmin, rmax, ang, width, number=10, mmin=21.0, mmax=25.5, version='s', dry_run=False):\n\n # Construct a list of artificial KBOs with positions in the image\n # and rates of motion within the bounds given by the caller.\n filename = storage.get_image(expnums[0],\n ccd=ccd,\n version=version)\n header = fits.open(filename)[0].header\n bounds = util.get_pixel_bounds_from_datasec_keyword(header.get('DATASEC', '[33:2080,1:4612]'))\n\n # generate a set of artificial KBOs to add to the image.\n kbos = KBOGenerator.get_kbos(n=number,\n rate=(rmin, rmax),\n angle=(ang - width, ang + width),\n mag=(mmin, mmax),\n x=(bounds[0][0], bounds[0][1]),\n y=(bounds[1][0], bounds[1][1]),\n filename='Object.planted')\n\n for expnum in expnums:\n filename = storage.get_image(expnum, ccd, version)\n psf = storage.get_file(expnum, ccd, version, ext='psf.fits')\n plant_kbos(filename, psf, kbos, get_shifts(expnum, ccd, version), \"fk\")\n\n if dry_run:\n return\n\n uri = storage.get_uri('Object', ext='planted', version='',\n subdir=str(\n expnums[0]) + \"/ccd%s\" % (str(ccd).zfill(2)))\n\n storage.copy('Object.planted', uri)\n for expnum in expnums:\n uri = storage.get_uri(expnum,\n ccd=ccd,\n version=version,\n ext='fits', prefix='fk')\n filename = os.path.basename(uri)\n storage.copy(filename, uri)\n\n return", "response": "Plant artificial sources into a list of images provided by the caller."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _fit_radec(self):\n\n # call fitradec with mpcfile, abgfile, resfile\n self.orbfit.fitradec.restype = ctypes.c_int\n self.orbfit.fitradec.argtypes = [ ctypes.c_char_p, ctypes.c_char_p, ctypes.c_char_p ]\n\n mpc_file = tempfile.NamedTemporaryFile(suffix='.mpc')\n for observation in self.observations:\n mpc_file.write(\"{}\\n\".format(str(observation)))\n mpc_file.seek(0)\n\n abg_file = tempfile.NamedTemporaryFile()\n res_file = tempfile.NamedTemporaryFile()\n\n self.orbfit.fitradec(ctypes.c_char_p(mpc_file.name),\n ctypes.c_char_p(abg_file.name),\n ctypes.c_char_p(res_file.name))\n\n self.abg = abg_file\n self.abg.seek(0)\n self.residuals = res_file\n self.residuals.seek(0)", "response": "fit_radec of BK passing in the observations."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef predict(self, date, obs_code=568):\n time = Time(date, scale='utc', precision=6)\n jd = ctypes.c_double(time.jd)\n # call predict with agbfile, jdate, obscode\n self.orbfit.predict.restype = ctypes.POINTER(ctypes.c_double * 5)\n self.orbfit.predict.argtypes = [ ctypes.c_char_p, ctypes.c_double, ctypes.c_int ]\n predict = self.orbfit.predict(ctypes.c_char_p(self.abg.name),\n jd,\n ctypes.c_int(obs_code))\n self.coordinate = coordinates.SkyCoord(predict.contents[0],\n predict.contents[1],\n unit=(units.degree, units.degree))\n self.dra = predict.contents[2]\n self.ddec = predict.contents[3]\n self.pa = predict.contents[4]\n self.date = str(time)", "response": "Use the bk predict method to compute the location of the source on the given date."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef write(file,hdu,order=None,format=None):\n\n\n if order or format:\n warnings.warn('Use of and depricated',DeprecationWarning)\n\n ## if the order of the columns isn't constrained just use the keys in what ever order\n data='data'\n if not order:\n if not hdu.has_key('order'):\n hdu['order']=hdu[data].keys()\n else:\n hdu['order']=order\n\t\n if not format:\n if not hdu.has_key('format'):\n hdu['format']={}\n for o in hdu['order']:\n if not hdu['format'].has_key(o):\n\t hdu['format'][o]='%10s'\n else:\n hdu['format']=format\n\t\n f=open(file,'w')\n kline='## '\n vline='# '\n header='header'\n num=0\n for keyword in hdu[header]:\n kline+='%10s ' % (keyword, )\n vline+='%10s ' % str(hdu[header][keyword])\n num+=1\n if not ( num % 6 ) :\n num=0\n f.write(kline+'\\n')\n f.write(vline+'\\n')\n kline='## '\n vline='# '\n if num > 0:\n f.write(kline+'\\n')\n f.write(vline+'\\n')\n\n f.write('## ')\n for column in hdu['order']:\n f.write(' %10s' % (column))\n f.write('\\n')\n dlen=len(hdu[data][hdu['order'][0]])\n for i in range(dlen):\n f.write(' ')\n for column in hdu['order']:\n f.write(hdu['format'][column] % (hdu[data][column][i]))\n f.write(' ')\n f.write('\\n')\n f.close()\n return", "response": "Write a file in the crazy MOP format given an mop data hdu."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef store(hdu,dbase='cfeps',duser='cfhls',dtype='MYSQL',table='source'):\n\n import MOPdbaccess\n db=MOPdbaccess.connect(dbase,duser,dbSystem=dtype)\n dbc=db.cursor()\n\n ### INSERT THE 'HEADER' in the meta-data table\n file_id=hdu['header']['image']\n for key in hdu['header'].keys():\n if key == 'image':\n continue\n value=hdu['header'][key]\n ### require that the file_id+keyword is unique\n sql=\"DELETE FROM metadata WHERE file_id='%s' and keyword='%s' \" % ( file_id, key)\n dbc.execute(sql)\n sql=\"INSERT INTO metadata (file_id, keyword, value) values ('%s','%s','%s' ) \"% ( file_id, key, value)\n dbc.execute(sql)\n db.commit()\n\n sql=\"DELETE FROM source WHERE file_id LIKE '%s' \" % ( file_id )\n dbc.execute(sql)\n\n sql=\"INSERT INTO %s ( \" % ( table)\n sep=\" \"\n values=\"(\"\n cols=hdu['hdu2sql'].keys()\n for col in cols:\n sql+=sep+hdu['hdu2sql'][col]\n values+=sep+\" %s \" \n sep=', '\n values+=\", '%s', %d, %d )\" % ( file_id, int(hdu['header']['EXPNUM']), int(hdu['header']['CHIP'] )-1 )\n sql+=\", file_id, expnum, ccd ) VALUES \"+values\n #print sql\n #sys.exit()\n #values=[]\n for row in range(len(hdu['data'][cols[0]])):\n value=[]\n for col in cols:\n value.append(hdu['data'][col][row])\n dbc.execute(sql,value)\n \n sql=\"\"\"insert into source_idx ( sourceID, x, y, z ) SELECT sourceID, \n (360.0*COS(RADIANS(raDeg))*COS(RADIANS(decDeg))),\n (360.0*SIN(RADIANS(raDeg))*COS(RADIANS(decDeg))),\n (360.0*SIN(RADIANS(decDeg))) FROM source WHERE file_id='%s'\"\"\" % (file_id)\n\n dbc.execute(sql)\n db.commit()\n\n return", "response": "Store the contents of a MOP data structure into a SQL table"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef read(file):\n \n f = open(file,'r')\n lines=f.readlines()\n f.close()\n\n import re, string\n keywords=[]\n values=[]\n formats={}\n header={}\n cdata={}\n\n for line in lines:\n if ( re.match(r'^##',line)):\n ## header line\n m=string.split(string.lstrip(line[2:]))\n if not m:\n sys.stderr.write( \"Ill formed header line in %s \\ n \" % ( file, ))\n sys.stderr.write( line)\n continue\n keywords=m\n add_id=False\n continue\n if ( re.match(r'^# ',line)):\n ## this is a keyword value line, must match the previous keyword line\n m=string.split(string.lstrip(line[1:]))\n values=m\n if len(values)!= len(keywords):\n sys.stderr.write( \"Ill formed header, keyword/value missmatach\\n\")\n for index in range(len(values)):\n header[keywords[index]]=values[index]\n\t ## blank the arrays, we've already stuck this into the header\n keywords=[]\n values=[]\n continue\n if ( re.match(r'^#F',line)):\n ## this is a format line for the columns needs to be a keyword line first\n if not keywords:\n sys.stderr.write(\"Cann't have formats without column names...\\n\")\n continue\n f=string.split(string.lstrip(line))\n if add_id:\n f.append('%8d')\n for col in keywords:\n formats[col]=f.pop(0)\n ### must be the actual data\n ### expect a previous header line to define column headers\n if not keywords:\n sys.stderr.write( \"No keywords for data columns, assuming x,y,mag,msky\\n\")\n keywords=('X','Y','MAG','MSKY')\n values = string.split(string.lstrip(line))\n ### add the id value to the values array if needed\n if not 'ID' in keywords:\n keywords.append('ID')\n add_id=True\n if not cdata:\n for keyword in keywords:\n cdata[keyword]=[]\n if add_id:\n values.append(len(cdata[keywords[0]])+1) \n if len(values)!=len(keywords):\n sys.stderr.write(\"Keyword and value index have different length?\\n\")\n\t continue\n for index in range(len(values)):\n cdata[keywords[index]].append(values[index])\n\n hdu={'data': cdata, 'header': header, 'format': formats, 'order': keywords}\n ### the hdu is a dictionary with the header and data\n return hdu", "response": "Read in a file and create a data strucuture that is a hash with the header keywords and data."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef xymatch(outfile, filenames, tol=2):\n import math\n import sys\n output={}\n files=[]\n for filename in filenames:\n this_file=read(filename)\n ## match files based on the 'X' and 'Y' column.\n ## if those don't exist then skip this file\n if not this_file['data'].has_key('X') or not this_file['data'].has_key('Y'):\n continue\n if not output.has_key('data'):\n output=this_file\n continue\n delete_list=[]\n for keyword in this_file['header']:\n if not keyword in output['header']:\n output['header'][keyword]=this_file['header'][keyword]\n for col in this_file['data'].keys():\n if not output['data'].has_key(col):\n output['order'].append(col)\n output['data'][col]=[]\n output['order'].append(col)\n if this_file['format'].has_key(col):\n output['format'][col]=this_file['format'][col]\n else:\n output['format'][col]='%10s'\n ### pad previously values with empties.. \n for i in range(len(output['data']['X'])):\n output['data'][col].append(None)\n for i in xrange(len(output['data']['X'])):\n x1=float(output['data']['X'][i])\n y1=float(output['data']['Y'][i])\n matched=False\n for j in xrange(len(this_file['data']['X'])):\n x2=float(this_file['data']['X'][j])\n y2=float(this_file['data']['Y'][j])\n if ( ((x1-x2)**2+(y1-y2)**2) < tol**2):\n for col in this_file['data'].keys():\n if output['data'][col][i] is None:\n output['data'][col][i]=this_file['data'][col][j]\n matched=True\n break\n if not matched:\n delete_list.append(i)\n\n delete_list.sort()\n delete_list.reverse()\n for i in delete_list:\n for col in output['data'].keys():\n del output['data'][col][i]\n\n\n write(outfile,output)", "response": "Given a list of MOPfiles merge them based on x and y coordinates matching."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef default_logging_dict(*loggers: str, **kwargs: Any) -> DictStrAny:\n kwargs.setdefault('level', 'INFO')\n return {\n 'version': 1,\n 'disable_existing_loggers': True,\n 'filters': {\n 'ignore_errors': {\n '()': IgnoreErrorsFilter,\n },\n },\n 'formatters': {\n 'default': {\n 'format': '%(asctime)s [%(levelname)s:%(name)s] %(message)s',\n },\n 'naked': {\n 'format': u'%(message)s',\n },\n },\n 'handlers': {\n 'stdout': {\n 'class': 'logging.StreamHandler',\n 'filters': ['ignore_errors'],\n 'formatter': 'default',\n 'level': 'DEBUG',\n 'stream': sys.stdout,\n },\n 'stderr': {\n 'class': 'logging.StreamHandler',\n 'formatter': 'default',\n 'level': 'WARNING',\n 'stream': sys.stderr,\n },\n },\n 'loggers': {\n logger: dict(handlers=['stdout', 'stderr'], **kwargs)\n for logger in loggers\n },\n }", "response": "Prepare logging dict suitable with logging. config. dictConfig."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef update_sentry_logging(logging_dict: DictStrAny,\n sentry_dsn: Optional[str],\n *loggers: str,\n level: Union[str, int] = None,\n **kwargs: Any) -> None:\n r\"\"\"Enable Sentry logging if Sentry DSN passed.\n\n .. note::\n Sentry logging requires `raven `_\n library to be installed.\n\n **Usage**::\n\n from logging.config import dictConfig\n\n LOGGING = default_logging_dict()\n SENTRY_DSN = '...'\n\n update_sentry_logging(LOGGING, SENTRY_DSN)\n dictConfig(LOGGING)\n\n **Using AioHttpTransport for SentryHandler**\n\n This will allow to use ``aiohttp.client`` for pushing data to Sentry in\n your ``aiohttp.web`` app, which means elimination of sync calls to Sentry.\n\n ::\n\n from raven_aiohttp import AioHttpTransport\n update_sentry_logging(LOGGING, SENTRY_DSN, transport=AioHttpTransport)\n\n :param logging_dict: Logging dict.\n :param sentry_dsn:\n Sentry DSN value. If ``None`` do not update logging dict at all.\n :param \\*loggers:\n Use Sentry logging for each logger in the sequence. If the sequence is\n empty use Sentry logging to each available logger.\n :param \\*\\*kwargs: Additional kwargs to be passed to ``SentryHandler``.\n \"\"\"\n # No Sentry DSN, nothing to do\n if not sentry_dsn:\n return\n\n # Add Sentry handler\n kwargs['class'] = 'raven.handlers.logging.SentryHandler'\n kwargs['dsn'] = sentry_dsn\n logging_dict['handlers']['sentry'] = dict(\n level=level or 'WARNING',\n **kwargs)\n\n loggers = tuple(logging_dict['loggers']) if not loggers else loggers\n for logger in loggers:\n # Ignore missing loggers\n logger_dict = logging_dict['loggers'].get(logger)\n if not logger_dict:\n continue\n\n # Ignore logger from logger config\n if logger_dict.pop('ignore_sentry', False):\n continue\n\n # Handlers list should exist\n handlers = list(logger_dict.setdefault('handlers', []))\n handlers.append('sentry')\n logger_dict['handlers'] = tuple(handlers)", "response": "r Update Sentry logging dictionary."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef trim(hdu, datasec='DATASEC'):\n datasec = re.findall(r'(\\d+)',\n hdu.header.get(datasec))\n l = int(datasec[0]) - 1\n r = int(datasec[1])\n b = int(datasec[2]) - 1\n t = int(datasec[3])\n logger.info(\"Trimming [%d:%d,%d:%d]\" % ( l, r, b, t))\n hdu.data = hdu.data[b:t, l:r]\n hdu.header.update('DATASEC',\n \"[%d:%d,%d:%d]\" % (1, r - l + 1, 1, t - b + 1),\n comment=\"Image was trimmed\")\n hdu.header.update('ODATASEC',\n \"[%d:%d,%d:%d]\" % (l + 1, r, b + 1, t),\n comment=\"previous DATASEC\")\n return", "response": "TRIM a CFHT MEGAPRIME frame using the DATASEC keyword"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nparse the OSSOS release file into a list of objects.", "response": "def ossos_release_parser(table=False, data_release=parameters.RELEASE_VERSION):\n \"\"\"\n extra fun as this is space-separated so using CSV parsers is not an option\n \"\"\"\n names = ['cl', 'p', 'j', 'k', 'sh', 'object', 'mag', 'e_mag', 'Filt', 'Hsur', 'dist', 'e_dist', 'Nobs',\n 'time', 'av_xres', 'av_yres', 'max_x', 'max_y', 'a', 'e_a', 'e', 'e_e', 'i', 'e_i', 'Omega', 'e_Omega',\n 'omega', 'e_omega', 'tperi', 'e_tperi', 'RAdeg', 'DEdeg', 'JD', 'rate']#, 'eff', 'm_lim']\n\n if table:\n retval = Table.read(parameters.RELEASE_DETECTIONS[data_release], format='ascii', guess=False,\n delimiter=' ', data_start=0, comment='#', names=names, header_start=None)\n else:\n retval = []\n with open(data_release, 'r') as detectionsfile:\n for line in detectionsfile.readlines()[1:]: # first line is column definitions\n obj = TNO.from_string(line, version=parameters.RELEASE_DETECTIONS[data_release])\n retval.append(obj)\n\n return retval"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef ossos_discoveries(directory=parameters.REAL_KBO_AST_DIR,\n suffix='ast',\n no_nt_and_u=False,\n single_object=None,\n all_objects=True,\n data_release=None,\n ):\n \"\"\"\n Returns a list of objects holding orbfit.Orbfit objects with the observations in the Orbfit.observations field.\n Default is to return only the objects corresponding to the current Data Release.\n \"\"\"\n retval = []\n # working_context = context.get_context(directory)\n # files = working_context.get_listing(suffix)\n files = [f for f in os.listdir(directory) if (f.endswith('mpc') or f.endswith('ast') or f.endswith('DONE'))]\n\n if single_object is not None:\n files = filter(lambda name: name.startswith(single_object), files)\n elif all_objects and data_release is not None:\n # only return the objects corresponding to a particular Data Release\n data_release = ossos_release_parser(table=True, data_release=data_release)\n objects = data_release['object']\n files = filter(lambda name: name.partition(suffix)[0].rstrip('.') in objects, files)\n\n for filename in files:\n # keep out the not-tracked and uncharacteried.\n if no_nt_and_u and (filename.__contains__('nt') or filename.startswith('u')):\n continue\n # observations = mpc.MPCReader(directory + filename)\n mpc_filename = directory + filename\n abg_filename = os.path.abspath(directory + '/../abg/') + \"/\" + os.path.splitext(filename)[0] + \".abg\"\n obj = TNO(None, ast_filename=mpc_filename, abg_filename=abg_filename)\n retval.append(obj)\n\n return retval", "response": "Returns a list of orbfit. Orbfit objects with the observations in the Orbfit. observations field."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef ossos_release_with_metadata():\n # discoveries = ossos_release_parser()\n discoveries = []\n observations = ossos_discoveries()\n for obj in observations:\n discov = [n for n in obj[0].mpc_observations if n.discovery.is_discovery][0]\n tno = parameters.tno()\n tno.dist = obj[1].distance\n tno.ra_discov = discov.coordinate.ra.degrees\n tno.mag = discov.mag\n tno.name = discov.provisional_name\n discoveries.append(tno)\n\n # for obj in discoveries:\n # observation = [n for n in observations if n.observations[-1].provisional_name == obj.name][0]\n # for obs in observation.observations:\n # if obs.discovery.is_discovery:\n # if obj.mag is not None:\n # H = obj.mag + 2.5 * math.log10(1. / ((obj.dist ** 2) * ((obj.dist - 1.) ** 2)))\n # else:\n # H = None\n # obj.H = H\n # obj.T = observation.T\n # obj.discovery_date = obs.date\n # obj.observations = observation\n\n return discoveries", "response": "Wrap the objects from the Version Releases together with the objects instantiated from fitting their mpc lines"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nloads a Survey Simulator model file as an array of ephem EllipticalBody objects.", "response": "def _kbos_from_survey_sym_model_input_file(model_file):\n \"\"\"\n Load a Survey Simulator model file as an array of ephem EllipticalBody objects.\n @param model_file:\n @return:\n \"\"\"\n lines = storage.open_vos_or_local(model_file).read().split('\\n')\n kbos = []\n for line in lines:\n if len(line) == 0 or line[0] == '#': # skip initial column descriptors and the final blank line\n continue\n kbo = ephem.EllipticalBody()\n values = line.split()\n kbo.name = values[8]\n kbo.j = values[9]\n kbo.k = values[10]\n kbo._a = float(values[0])\n kbo._e = float(values[1])\n kbo._inc = float(values[2])\n kbo._Om = float(values[3])\n kbo._om = float(values[4])\n kbo._M = float(values[5])\n kbo._H = float(values[6])\n epoch = ephem.date(2453157.50000 - ephem.julian_date(0))\n kbo._epoch_M = epoch\n kbo._epoch = epoch\n kbos.append(kbo)\n return kbos"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nround a number to the right sig figs.", "response": "def round_sig_error(num, uncert, pm=False):\n \"\"\"\n Return a string of the number and its uncertainty to the right sig figs via uncertainty's print methods.\n The uncertainty determines the sig fig rounding of the number.\n https://pythonhosted.org/uncertainties/user_guide.html\n \"\"\"\n u = ufloat(num, uncert)\n if pm:\n return '{:.1uL}'.format(u)\n else:\n return '{:.1uLS}'.format(u)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _parse_elcm_response_body_as_json(response):\n try:\n body = response.text\n body_parts = body.split('\\r\\n')\n if len(body_parts) > 0:\n return jsonutils.loads(body_parts[-1])\n else:\n return None\n except (TypeError, ValueError):\n raise ELCMInvalidResponse('eLCM response does not contain valid json '\n 'data. Response is \"%s\".' % body)", "response": "parse eLCM response body as json"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef elcm_request(irmc_info, method, path, **kwargs):\n host = irmc_info['irmc_address']\n port = irmc_info.get('irmc_port', 443)\n auth_method = irmc_info.get('irmc_auth_method', 'basic')\n userid = irmc_info['irmc_username']\n password = irmc_info['irmc_password']\n client_timeout = irmc_info.get('irmc_client_timeout', 60)\n\n # Request headers, params, and data\n headers = kwargs.get('headers', {'Accept': 'application/json'})\n params = kwargs.get('params')\n data = kwargs.get('data')\n\n auth_obj = None\n try:\n protocol = {80: 'http', 443: 'https'}[port]\n auth_obj = {\n 'basic': requests.auth.HTTPBasicAuth(userid, password),\n 'digest': requests.auth.HTTPDigestAuth(userid, password)\n }[auth_method.lower()]\n\n except KeyError:\n raise scci.SCCIInvalidInputError(\n (\"Invalid port %(port)d or \" +\n \"auth_method for method %(auth_method)s\") %\n {'port': port, 'auth_method': auth_method})\n\n try:\n r = requests.request(method,\n protocol + '://' + host + path,\n headers=headers,\n params=params,\n data=data,\n verify=False,\n timeout=client_timeout,\n allow_redirects=False,\n auth=auth_obj)\n except requests.exceptions.RequestException as requests_exception:\n raise scci.SCCIClientError(requests_exception)\n\n # Process status_code 401\n if r.status_code == 401:\n raise scci.SCCIClientError('UNAUTHORIZED')\n\n return r", "response": "send an eLCM request to the server"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsend an eLCM request to get profile versions of the current node", "response": "def elcm_profile_get_versions(irmc_info):\n \"\"\"send an eLCM request to get profile versions\n\n :param irmc_info: node info\n :returns: dict object of profiles if succeed\n {\n \"Server\":{\n \"@Version\": \"1.01\",\n \"AdapterConfigIrmc\":{\n \"@Version\": \"1.00\"\n },\n \"HWConfigurationIrmc\":{\n \"@Version\": \"1.00\"\n },\n \"SystemConfig\":{\n \"IrmcConfig\":{\n \"@Version\": \"1.02\"\n },\n \"BiosConfig\":{\n \"@Version\": \"1.02\"\n }\n }\n }\n }\n :raises: SCCIClientError if SCCI failed\n \"\"\"\n # Send GET request to the server\n resp = elcm_request(irmc_info,\n method='GET',\n path=URL_PATH_PROFILE_MGMT + 'version')\n\n if resp.status_code == 200:\n return _parse_elcm_response_body_as_json(resp)\n else:\n raise scci.SCCIClientError(('Failed to get profile versions with '\n 'error code %s' % resp.status_code))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef elcm_profile_get(irmc_info, profile_name):\n # Send GET request to the server\n resp = elcm_request(irmc_info,\n method='GET',\n path=URL_PATH_PROFILE_MGMT + profile_name)\n\n if resp.status_code == 200:\n return _parse_elcm_response_body_as_json(resp)\n elif resp.status_code == 404:\n raise ELCMProfileNotFound('Profile \"%s\" not found '\n 'in the profile store.' % profile_name)\n else:\n raise scci.SCCIClientError(('Failed to get profile \"%(profile)s\" with '\n 'error code %(error)s' %\n {'profile': profile_name,\n 'error': resp.status_code}))", "response": "send an eLCM request to get a profile data"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef elcm_profile_create(irmc_info, param_path):\n # Send POST request to the server\n # NOTE: This task may take time, so set a timeout\n _irmc_info = dict(irmc_info)\n _irmc_info['irmc_client_timeout'] = PROFILE_CREATE_TIMEOUT\n\n resp = elcm_request(_irmc_info,\n method='POST',\n path=URL_PATH_PROFILE_MGMT + 'get',\n params={'PARAM_PATH': param_path})\n\n if resp.status_code == 202:\n return _parse_elcm_response_body_as_json(resp)\n else:\n raise scci.SCCIClientError(('Failed to create profile for path '\n '\"%(param_path)s\" with error code '\n '%(error)s' %\n {'param_path': param_path,\n 'error': resp.status_code}))", "response": "Send an eLCM request to create a profile"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef elcm_profile_set(irmc_info, input_data):\n # Prepare the data to apply\n if isinstance(input_data, dict):\n data = jsonutils.dumps(input_data)\n else:\n data = input_data\n\n # Send POST request to the server\n # NOTE: This task may take time, so set a timeout\n _irmc_info = dict(irmc_info)\n _irmc_info['irmc_client_timeout'] = PROFILE_SET_TIMEOUT\n\n content_type = 'application/x-www-form-urlencoded'\n if input_data['Server'].get('HWConfigurationIrmc'):\n content_type = 'application/json'\n resp = elcm_request(_irmc_info,\n method='POST',\n path=URL_PATH_PROFILE_MGMT + 'set',\n headers={'Content-type': content_type},\n data=data)\n\n if resp.status_code == 202:\n return _parse_elcm_response_body_as_json(resp)\n else:\n raise scci.SCCIClientError(('Failed to apply param values with '\n 'error code %(error)s' %\n {'error': resp.status_code}))", "response": "send an eLCM request to set param values to the node"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef elcm_profile_delete(irmc_info, profile_name):\n # Send DELETE request to the server\n resp = elcm_request(irmc_info,\n method='DELETE',\n path=URL_PATH_PROFILE_MGMT + profile_name)\n\n if resp.status_code == 200:\n # Profile deleted\n return\n elif resp.status_code == 404:\n # Profile not found\n raise ELCMProfileNotFound('Profile \"%s\" not found '\n 'in the profile store.' % profile_name)\n else:\n raise scci.SCCIClientError(('Failed to delete profile \"%(profile)s\" '\n 'with error code %(error)s' %\n {'profile': profile_name,\n 'error': resp.status_code}))", "response": "send an eLCM request to delete a profile"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef elcm_session_list(irmc_info):\n # Send GET request to the server\n resp = elcm_request(irmc_info,\n method='GET',\n path='/sessionInformation/')\n\n if resp.status_code == 200:\n return _parse_elcm_response_body_as_json(resp)\n else:\n raise scci.SCCIClientError(('Failed to list sessions with '\n 'error code %s' % resp.status_code))", "response": "send an eLCM request to list all sessions in the node"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef elcm_session_get_status(irmc_info, session_id):\n # Send GET request to the server\n resp = elcm_request(irmc_info,\n method='GET',\n path='/sessionInformation/%s/status' % session_id)\n\n if resp.status_code == 200:\n return _parse_elcm_response_body_as_json(resp)\n elif resp.status_code == 404:\n raise ELCMSessionNotFound('Session \"%s\" does not exist' % session_id)\n else:\n raise scci.SCCIClientError(('Failed to get status of session '\n '\"%(session)s\" with error code %(error)s' %\n {'session': session_id,\n 'error': resp.status_code}))", "response": "send an eLCM request to get the status of the specified session"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef elcm_session_terminate(irmc_info, session_id):\n # Send DELETE request to the server\n resp = elcm_request(irmc_info,\n method='DELETE',\n path='/sessionInformation/%s/terminate' % session_id)\n\n if resp.status_code == 200:\n return\n elif resp.status_code == 404:\n raise ELCMSessionNotFound('Session \"%s\" does not exist' % session_id)\n else:\n raise scci.SCCIClientError(('Failed to terminate session '\n '\"%(session)s\" with error code %(error)s' %\n {'session': session_id,\n 'error': resp.status_code}))", "response": "send an eLCM request to terminate a session"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsending an eLCM request to remove a session from the session list", "response": "def elcm_session_delete(irmc_info, session_id, terminate=False):\n \"\"\"send an eLCM request to remove a session from the session list\n\n :param irmc_info: node info\n :param session_id: session id\n :param terminate: a running session must be terminated before removing\n :raises: ELCMSessionNotFound if the session does not exist\n :raises: SCCIClientError if SCCI failed\n \"\"\"\n # Terminate the session first if needs to\n if terminate:\n # Get session status to check\n session = elcm_session_get_status(irmc_info, session_id)\n status = session['Session']['Status']\n\n # Terminate session if it is activated or running\n if status == 'running' or status == 'activated':\n elcm_session_terminate(irmc_info, session_id)\n\n # Send DELETE request to the server\n resp = elcm_request(irmc_info,\n method='DELETE',\n path='/sessionInformation/%s/remove' % session_id)\n\n if resp.status_code == 200:\n return\n elif resp.status_code == 404:\n raise ELCMSessionNotFound('Session \"%s\" does not exist' % session_id)\n else:\n raise scci.SCCIClientError(('Failed to remove session '\n '\"%(session)s\" with error code %(error)s' %\n {'session': session_id,\n 'error': resp.status_code}))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _process_session_data(irmc_info, operation, session_id,\n session_timeout=BIOS_CONFIG_SESSION_TIMEOUT):\n \"\"\"process session for Bios config backup/restore or RAID config operation\n\n :param irmc_info: node info\n :param operation: one of 'BACKUP_BIOS', 'RESTORE_BIOS' or 'CONFIG_RAID'\n :param session_id: session id\n :param session_timeout: session timeout\n :return: a dict with following values:\n {\n 'bios_config': ,\n 'warning': \n }\n or\n {\n 'raid_config': ,\n 'warning': \n }\n \"\"\"\n session_expiration = time.time() + session_timeout\n\n while time.time() < session_expiration:\n # Get session status to check\n session = elcm_session_get_status(irmc_info=irmc_info,\n session_id=session_id)\n\n status = session['Session']['Status']\n if status == 'running' or status == 'activated':\n # Sleep a bit\n time.sleep(5)\n elif status == 'terminated regularly':\n result = {}\n\n if operation == 'BACKUP_BIOS':\n # Bios profile is created, get the data now\n result['bios_config'] = elcm_profile_get(\n irmc_info=irmc_info,\n profile_name=PROFILE_BIOS_CONFIG)\n elif operation == 'RESTORE_BIOS':\n # Bios config applied successfully\n pass\n elif operation == 'CONFIG_RAID':\n # Getting raid config\n result['raid_config'] = elcm_profile_get(irmc_info,\n PROFILE_RAID_CONFIG)\n\n # Cleanup operation by deleting related session and profile.\n # In case of error, report it as warning instead of error.\n try:\n elcm_session_delete(irmc_info=irmc_info,\n session_id=session_id,\n terminate=True)\n if operation == 'CONFIG_RAID':\n return result\n\n elcm_profile_delete(irmc_info=irmc_info,\n profile_name=PROFILE_BIOS_CONFIG)\n\n except scci.SCCIError as e:\n result['warning'] = e\n\n return result\n else:\n # Error occurred, get session log to see what happened\n session_log = elcm_session_get_log(irmc_info=irmc_info,\n session_id=session_id)\n\n raise scci.SCCIClientError(\n ('Failed to %(operation)s config. '\n 'Session log is \"%(session_log)s\".' %\n {'operation': operation,\n 'session_log': jsonutils.dumps(session_log)}))\n\n else:\n raise ELCMSessionTimeout(\n ('Failed to %(operation)s config. '\n 'Session %(session_id)s log is timeout.' %\n {'operation': operation,\n 'session_id': session_id}))", "response": "Process the data from the session for BIOS config backup or restore or RAID config."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef restore_bios_config(irmc_info, bios_config):\n\n def _process_bios_config():\n try:\n if isinstance(bios_config, dict):\n input_data = bios_config\n else:\n input_data = jsonutils.loads(bios_config)\n\n # The input data must contain flag \"@Processing\":\"execute\" in the\n # equivalent section.\n bios_cfg = input_data['Server']['SystemConfig']['BiosConfig']\n bios_cfg['@Processing'] = 'execute'\n\n return input_data\n except (TypeError, ValueError, KeyError):\n raise scci.SCCIInvalidInputError(\n ('Invalid input bios config \"%s\".' % bios_config))\n\n # 1. Parse the bios config and create the input data\n input_data = _process_bios_config()\n\n # 2. Make sure there is no BiosConfig profile in the store\n try:\n # Get the profile first, if not found, then an exception\n # will be raised.\n elcm_profile_get(irmc_info=irmc_info,\n profile_name=PROFILE_BIOS_CONFIG)\n # Profile found, delete it\n elcm_profile_delete(irmc_info=irmc_info,\n profile_name=PROFILE_BIOS_CONFIG)\n except ELCMProfileNotFound:\n # Ignore this error as it's not an error in this case\n pass\n\n # 3. Send a request to apply the param values\n session = elcm_profile_set(irmc_info=irmc_info,\n input_data=input_data)\n\n # 4. Param values applying is in progress, we monitor the session\n session_timeout = irmc_info.get('irmc_bios_session_timeout',\n BIOS_CONFIG_SESSION_TIMEOUT)\n _process_session_data(irmc_info=irmc_info,\n operation='RESTORE_BIOS',\n session_id=session['Session']['Id'],\n session_timeout=session_timeout)", "response": "Restores the bios configuration to the server."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the status of secure boot mode on the node.", "response": "def get_secure_boot_mode(irmc_info):\n \"\"\"Get the status if secure boot is enabled or not.\n\n :param irmc_info: node info.\n :raises: SecureBootConfigNotFound, if there is no configuration for secure\n boot mode in the bios.\n :return: True if secure boot mode is enabled on the node, False otherwise.\n \"\"\"\n\n result = backup_bios_config(irmc_info=irmc_info)\n\n try:\n bioscfg = result['bios_config']['Server']['SystemConfig']['BiosConfig']\n return bioscfg['SecurityConfig']['SecureBootControlEnabled']\n\n except KeyError:\n msg = (\"Failed to get secure boot mode from server %s. Upgrading iRMC \"\n \"firmware may resolve this issue.\" % irmc_info['irmc_address'])\n raise SecureBootConfigNotFound(msg)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nenable or disable secure boot on the server.", "response": "def set_secure_boot_mode(irmc_info, enable):\n \"\"\"Enable/Disable secure boot on the server.\n\n :param irmc_info: node info\n :param enable: True, if secure boot needs to be\n enabled for next boot, else False.\n \"\"\"\n\n bios_config_data = {\n 'Server': {\n '@Version': '1.01',\n 'SystemConfig': {\n 'BiosConfig': {\n '@Version': '1.01',\n 'SecurityConfig': {\n 'SecureBootControlEnabled': enable\n }\n }\n }\n }\n }\n restore_bios_config(irmc_info=irmc_info, bios_config=bios_config_data)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nupdating the RAID input data with the new data.", "response": "def _update_raid_input_data(target_raid_config, raid_input):\n \"\"\"Process raid input data.\n\n :param target_raid_config: node raid info\n :param raid_input: raid information for creating via eLCM\n :raises ELCMValueError: raise msg if wrong input\n :return: raid_input: raid input data which create raid configuration\n {\n \"Server\":{\n \"HWConfigurationIrmc\":{\n \"@Processing\":\"execute\",\n \"Adapters\":{\n \"RAIDAdapter\":[\n {\n \"@AdapterId\":\"RAIDAdapter0\",\n \"@ConfigurationType\":\"Addressing\",\n \"LogicalDrives\":{\n \"LogicalDrive\":[\n {\n \"@Number\":0,\n \"@Action\":\"Create\",\n \"RaidLevel\":\"1\"\n }\n ]\n }\n }\n ]\n },\n \"@Version\":\"1.00\"\n },\n \"@Version\":\"1.01\"\n }\n }\n \"\"\"\n\n logical_disk_list = target_raid_config['logical_disks']\n\n raid_input['Server']['HWConfigurationIrmc'].update({'@Processing':\n 'execute'})\n array_info = raid_input['Server']['HWConfigurationIrmc']['Adapters'][\n 'RAIDAdapter'][0]\n array_info['LogicalDrives'] = {'LogicalDrive': []}\n array_info['Arrays'] = {'Array': []}\n\n for i, logical_disk in enumerate(logical_disk_list):\n physical_disks = logical_disk.get('physical_disks')\n\n # Auto create logical drive along with random physical disks.\n # Allow auto create along with raid 10 and raid 50\n # with specific physical drive.\n if not physical_disks or logical_disk['raid_level'] \\\n in ('10', '50'):\n array_info['LogicalDrives']['LogicalDrive'].append(\n {'@Action': 'Create',\n 'RaidLevel': logical_disk['raid_level'],\n 'InitMode': 'slow'})\n array_info['LogicalDrives']['LogicalDrive'][i].update({\n \"@Number\": i})\n\n else:\n # Create array disks with specific physical servers\n arrays = {\n \"@Number\": i,\n \"@ConfigurationType\": \"Setting\",\n \"PhysicalDiskRefs\": {\n \"PhysicalDiskRef\": []\n }\n }\n\n lo_drive = {\n \"@Number\": i,\n \"@Action\": \"Create\",\n \"RaidLevel\": \"\",\n \"ArrayRefs\": {\n \"ArrayRef\": [\n ]\n },\n \"InitMode\": \"slow\"\n }\n\n array_info['Arrays']['Array'].append(arrays)\n array_info['LogicalDrives']['LogicalDrive'].append(lo_drive)\n\n lo_drive.update({'RaidLevel': logical_disk['raid_level']})\n lo_drive['ArrayRefs']['ArrayRef'].append({\"@Number\": i})\n\n for element in logical_disk['physical_disks']:\n arrays['PhysicalDiskRefs']['PhysicalDiskRef'].append({\n '@Number': element})\n if logical_disk['size_gb'] != \"MAX\":\n # Ensure correctly order these items in dict\n size = collections.OrderedDict()\n size['@Unit'] = 'GB'\n size['#text'] = logical_disk['size_gb']\n array_info['LogicalDrives']['LogicalDrive'][i]['Size'] = size\n\n return raid_input"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_existing_logical_drives(raid_adapter):\n existing_logical_drives = []\n logical_drives = raid_adapter['Server']['HWConfigurationIrmc'][\n 'Adapters']['RAIDAdapter'][0].get('LogicalDrives')\n if logical_drives is not None:\n for drive in logical_drives['LogicalDrive']:\n existing_logical_drives.append(drive['@Number'])\n\n return existing_logical_drives", "response": "Collect existing logical drives on the server."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _create_raid_adapter_profile(irmc_info):\n\n try:\n # Attempt erase exist adapter on BM Server\n elcm_profile_delete(irmc_info, PROFILE_RAID_CONFIG)\n except ELCMProfileNotFound:\n # Ignore this error as it's not an error in this case\n pass\n\n session = elcm_profile_create(irmc_info, PARAM_PATH_RAID_CONFIG)\n\n # Monitoring currently session until done.\n session_timeout = irmc_info.get('irmc_raid_session_timeout',\n RAID_CONFIG_SESSION_TIMEOUT)\n\n return _process_session_data(irmc_info, 'CONFIG_RAID',\n session['Session']['Id'],\n session_timeout)", "response": "Create raid adapter profile on the server."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef create_raid_configuration(irmc_info, target_raid_config):\n\n if len(target_raid_config['logical_disks']) < 1:\n raise ELCMValueError(message=\"logical_disks must not be empty\")\n\n # Check RAID config in the new RAID adapter. Must be erased before\n # create new RAID config.\n raid_adapter = get_raid_adapter(irmc_info)\n logical_drives = raid_adapter['Server']['HWConfigurationIrmc'][\n 'Adapters']['RAIDAdapter'][0].get('LogicalDrives')\n session_timeout = irmc_info.get('irmc_raid_session_timeout',\n RAID_CONFIG_SESSION_TIMEOUT)\n if logical_drives is not None:\n # Delete exist logical drives in server.\n # NOTE(trungnv): Wait session complete and raise error if\n # delete raid config during FGI(Foreground Initialization) in-progress\n # in previous mechanism.\n delete_raid_configuration(irmc_info)\n # Updating raid adapter profile after deleted profile.\n raid_adapter = get_raid_adapter(irmc_info)\n\n # Create raid configuration based on target_raid_config of node\n raid_input = _update_raid_input_data(target_raid_config, raid_adapter)\n session = elcm_profile_set(irmc_info, raid_input)\n # Monitoring raid creation session until done.\n _process_session_data(irmc_info, 'CONFIG_RAID',\n session['Session']['Id'],\n session_timeout)", "response": "Create a RAID config based on target_raid_config."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndelete whole raid configuration or one of logical drive on the server.", "response": "def delete_raid_configuration(irmc_info):\n \"\"\"Delete whole raid configuration or one of logical drive on the server.\n\n :param irmc_info: node info\n \"\"\"\n # Attempt to get raid configuration on BM Server\n raid_adapter = get_raid_adapter(irmc_info)\n existing_logical_drives = _get_existing_logical_drives(raid_adapter)\n # Ironic requires delete_configuration first. Will pass if blank raid\n # configuration in server.\n if not existing_logical_drives:\n return\n raid_adapter['Server']['HWConfigurationIrmc'].update({\n '@Processing': 'execute'})\n logical_drive = raid_adapter['Server']['HWConfigurationIrmc'][\n 'Adapters']['RAIDAdapter'][0]['LogicalDrives']['LogicalDrive']\n\n for drive in logical_drive:\n drive['@Action'] = 'Delete'\n\n # Attempt to delete logical drive in the raid config\n session = elcm_profile_set(irmc_info, raid_adapter)\n # Monitoring raid config delete session until done.\n session_timeout = irmc_info.get('irmc_raid_session_timeout',\n RAID_CONFIG_SESSION_TIMEOUT)\n _process_session_data(irmc_info, 'CONFIG_RAID', session['Session']['Id'],\n session_timeout)\n\n # Attempt to delete raid adapter\n elcm_profile_delete(irmc_info, PROFILE_RAID_CONFIG)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef set_bios_configuration(irmc_info, settings):\n\n bios_config_data = {\n 'Server': {\n 'SystemConfig': {\n 'BiosConfig': {}\n }\n }\n }\n\n versions = elcm_profile_get_versions(irmc_info)\n server_version = versions['Server'].get('@Version')\n bios_version = \\\n versions['Server']['SystemConfig']['BiosConfig'].get('@Version')\n\n if server_version:\n bios_config_data['Server']['@Version'] = server_version\n if bios_version:\n bios_config_data['Server']['SystemConfig']['BiosConfig']['@Version'] = \\\n bios_version\n\n configs = {}\n for setting_param in settings:\n setting_name = setting_param.get(\"name\")\n setting_value = setting_param.get(\"value\")\n # Revert-conversion from a string of True/False to boolean.\n # It will be raise failed if put \"True\" or \"False\" string value.\n if isinstance(setting_value, six.string_types):\n if setting_value.lower() == \"true\":\n setting_value = True\n elif setting_value.lower() == \"false\":\n setting_value = False\n try:\n type_config, config = BIOS_CONFIGURATION_DICTIONARY[\n setting_name].split(\"_\")\n if type_config in configs.keys():\n configs[type_config][config] = setting_value\n else:\n configs.update({type_config: {config: setting_value}})\n except KeyError:\n raise BiosConfigNotFound(\"Invalid BIOS setting: %s\"\n % setting_param)\n bios_config_data['Server']['SystemConfig']['BiosConfig'].update(configs)\n restore_bios_config(irmc_info, bios_config_data)", "response": "Set the BIOS configurations on the server."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_bios_settings(irmc_info):\n\n bios_config = backup_bios_config(irmc_info)['bios_config']\n bios_config_data = bios_config['Server']['SystemConfig']['BiosConfig']\n settings = []\n\n # TODO(trungnv): Allow working with multi levels of BIOS dictionary.\n for setting_param in BIOS_CONFIGURATION_DICTIONARY:\n type_config, config = BIOS_CONFIGURATION_DICTIONARY[\n setting_param].split(\"_\")\n if config in bios_config_data.get(type_config, {}):\n value = six.text_type(bios_config_data[type_config][config])\n settings.append({'name': setting_param, 'value': value})\n return settings", "response": "Get the current BIOS settings on the server"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_resource_context(router: web.AbstractRouter,\n url_prefix: str = None,\n name_prefix: str = None) -> Iterator[Any]:\n \"\"\"Context manager for adding resources for given router.\n\n Main goal of context manager to easify process of adding resources with\n routes to the router. This also allow to reduce amount of repeats, when\n supplying new resources by reusing URL & name prefixes for all routes\n inside context manager.\n\n Behind the scene, context manager returns a function which calls::\n\n resource = router.add_resource(url, name)\n resource.add_route(method, handler)\n\n **Usage**::\n\n with add_resource_context(app.router, '/api', 'api') as add_resource:\n add_resource('/', get=views.index)\n add_resource('/news', get=views.list_news, post=views.create_news)\n\n :param router: Route to add resources to.\n :param url_prefix: If supplied prepend this prefix to each resource URL.\n :param name_prefix: If supplied prepend this prefix to each resource name.\n \"\"\"\n def add_resource(url: str,\n get: View = None,\n *,\n name: str = None,\n **kwargs: Any) -> web.Resource:\n \"\"\"Inner function to create resource and add necessary routes to it.\n\n Support adding routes of all methods, supported by aiohttp, as\n GET/POST/PUT/PATCH/DELETE/HEAD/OPTIONS/*, e.g.,\n\n ::\n\n with add_resource_context(app.router) as add_resource:\n add_resource('/', get=views.get, post=views.post)\n add_resource('/wildcard', **{'*': views.wildcard})\n\n :param url:\n Resource URL. If ``url_prefix`` setup in context it will be\n prepended to URL with ``/``.\n :param get:\n GET handler. Only handler to be setup without explicit call.\n :param name: Resource name.\n :type name: str\n :rtype: aiohttp.web.Resource\n \"\"\"\n kwargs['get'] = get\n\n if url_prefix:\n url = '/'.join((url_prefix.rstrip('/'), url.lstrip('/')))\n\n if not name and get:\n name = get.__name__\n if name_prefix and name:\n name = '.'.join((name_prefix.rstrip('.'), name.lstrip('.')))\n\n resource = router.add_resource(url, name=name)\n for method, handler in kwargs.items():\n if handler is None:\n continue\n resource.add_route(method.upper(), handler)\n\n return resource\n\n yield add_resource", "response": "A context manager for adding resources to a given router."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nparses an AIoredis URL into a dict suitable to pass to aioredis. create_redis.", "response": "def parse_aioredis_url(url: str) -> DictStrAny:\n \"\"\"\n Convert Redis URL string to dict suitable to pass to\n ``aioredis.create_redis(...)`` call.\n\n **Usage**::\n\n async def connect_redis(url=None):\n url = url or 'redis://localhost:6379/0'\n return await create_redis(**get_aioredis_parts(url))\n\n :param url: URL to access Redis instance, started with ``redis://``.\n \"\"\"\n parts = urlparse(url)\n\n db = parts.path[1:] or None # type: Optional[Union[str, int]]\n if db:\n db = int(db)\n\n return {\n 'address': (parts.hostname, parts.port or 6379),\n 'db': db,\n 'password': parts.password}"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef str_kwargs(self):\n iys, ims, ids, ihmsfs = sofa_time.jd_dtf(self.scale.upper()\n .encode('utf8'),\n 6,\n self.jd1, self.jd2)\n\n # Get the str_fmt element of the first allowed output subformat \n _, _, str_fmt = self._select_subfmts(self.out_subfmt)[0]\n\n yday = None\n has_yday = '{yday:' in str_fmt or False\n\n for iy, im, iday, ihmsf in itertools.izip(iys, ims, ids, ihmsfs):\n ihr, imin, isec, ifracsec = ihmsf\n if has_yday:\n yday = datetime(iy, im, iday).timetuple().tm_yday\n\n # MPC uses day fraction as time part of datetime\n fracday = (((((ifracsec / 1000000.0 + isec) / 60.0 + imin) / 60.0) + ihr) / 24.0) * (10 ** 6)\n fracday = '{0:06g}'.format(fracday)[0:self.precision]\n #format_str = '{{0:g}}'.format(self.precision)\n #fracday = int(format_str.format(fracday))\n yield dict(year=int(iy), mon=int(im), day=int(iday), hour=int(ihr), min=int(imin), sec=int(isec),\n fracsec=int(ifracsec), yday=yday, fracday=fracday)", "response": "Generator that yields a dict of values corresponding to the internal JD values."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef oggvorbis(s):\n try:\n s = np.arange(s)\n except TypeError:\n s = np.arange(s[0])\n\n i = np.sin((s + .5) / len(s) * np.pi) ** 2\n f = np.sin(.5 * np.pi * i)\n return f * (1. / f.max())", "response": "This function is taken from the ogg vorbis spec to get the total length of the window in samples"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting the source location from reading into the coordinate system of reference_reading.", "response": "def convert_source_location(self, source_reading, reference_reading):\n \"\"\"\n Converts the source (x, y) location from reading into the coordinate\n system of reference_reading.\n \"\"\"\n offset_x, offset_y = reference_reading.get_coordinate_offset(source_reading)\n focus = source_reading.x + offset_x, source_reading.y + offset_y\n return focus"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalculating what the focal point of the downloaded image should be.", "response": "def calculate_focus(self, reading):\n \"\"\"\n Determines what the focal point of the downloaded image should be.\n\n Returns:\n focal_point: (x, y)\n The location of the source in the middle observation, in the\n coordinate system of the current source reading.\n \"\"\"\n middle_index = len(self.source.get_readings()) // 2\n middle_reading = self.source.get_reading(middle_index)\n return self.convert_source_location(middle_reading, reading)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef top_down_SolarSystem(discoveries,\n inner_limit=6, # truncate at 8 AU to show that we don't have sensitivity in close\n extent=83, # extend to 83 AU as indicative only, sensitivity is to ~300 AU\n plot_discoveries=None,\n plot_colossos=False,\n plot_blocks=None,\n plot_galaxy=False,\n feature_blocks=None,\n plot_Ijiraq=False, # detected Ijiraq at 9.80 AU in the 13AE block\n label_blocks=True,\n savefilename=None):\n \"\"\"\n Plot the OSSOS discoveries on a top-down Solar System showing the position of Neptune and model TNOs.\n Discoveries are plotted each at their time of discovery according to the value in the Version Release.\n Coordinates should be polar to account for RA hours, radial axis is in AU.\n :return: a saved plot\n\n This is not a 'projection' of the particles into any common plane. Each wedge is a different latitude above\n the plane, but inside each wedge it is heliocentric distance vs RA.\n That is fine. It's just important to know that if someone asks we know this is NOT (for example) a projection of each\n object down into the ecliptic.\n A very minor future long-term improvement is that the galactic\n plane wedge is not (I don't think) symmetric around 18h and 6h.\n I think that lines of constant (say 15 degree galactic lat) are\n centered a little off (I think the wedge would 'twist' a little\n bit counter-clockwise). I am not absolutely sure along the\n ecliptic where the b= +/- 15 degree lines are but I don't\n think they are symmetric?\n \"\"\"\n fig = plt.figure(figsize=(6, 6))\n rect = [0.01, 0.01, 0.95, .95] # the plot occupies not all the figspace\n\n ax1 = fig.add_axes(rect, polar=True, frameon=False) # theta (RA) is zero at E, increases anticlockwise\n ax1.set_aspect('equal')\n\n ax1.set_rlim(0, extent)\n ax1.set_rgrids([20, 40, 60, 80], labels=[\"\", \"\", '20 au', '40 au', '60 au', '80 au'], angle=190, alpha=0.45) # angle = 308\n ax1.yaxis.set_major_locator(MultipleLocator(20))\n ax1.xaxis.set_major_locator(MultipleLocator(math.radians(15))) # every 2 hours\n ax1.grid(axis='x', color='k', linestyle='--', alpha=0.2)\n ax1.set_xticklabels(['', '0h', \"\", '2h', \"\", '4h', \"\", '6h', \"\", '8h', \"\", '10h', \"\", '12h', \"\", '14h', \"\", '16h',\n \"\", '18h', \"\", '20h', \"\", '22h', \"\", ],\n # \"\"]) # str(r)+'h' for r in range(-1,24)],\n # ['', '0h', '2h', '4h', '6h', '8h', '10h', '12h', '14h', '16h', '18h', '20h', '22h'],\n color='b', alpha=0.6) # otherwise they get in the way\n\n\n if plot_galaxy:\n # plot exclusion zones due to Galactic plane: RAs indicate where bar starts, rather than its centre angle\n width = math.radians(3 * 15)\n plt.bar(math.radians(4.5 * 15), extent, width=width, color=plot_fanciness.ALMOST_BLACK, linewidth=0, alpha=0.2)\n plt.bar(math.radians(16.5 * 15), extent, width=width, color=plot_fanciness.ALMOST_BLACK, linewidth=0, alpha=0.2)\n ax1.annotate('galactic plane', (math.radians(6.9 * 15), extent - 15), size=10, color='k', alpha=0.45)\n ax1.annotate(' galactic plane\\navoidance zone', (math.radians(16.9 * 15), extent - 12), size=10, color='k', alpha=0.45)\n\n # again RAs indicate where bar starts, so subtract half the block width from the block centrepoints\n # and approximate blocks as math.radians(7) degrees wide.\n\n for blockname, block in parameters.BLOCKS.items():\n if plot_blocks:\n if blockname in plot_blocks:\n if feature_blocks is not None and blockname in feature_blocks:\n colour = 'm' #'#E47833'\n alpha = 0.25\n else:\n colour = 'b'\n alpha = 0.1\n # if blockname.startswith('13') or blockname.startswith('14'):\n width = math.radians(7) # the 3 x 7 field layout\n # else:\n # width = math.radians(5) # the 4 x 5 field layout\n\n # cmap = plt.get_cmap('Blues')\n # if blockname.endswith('AO'):\n # colour = '#E47833'\n # alpha = 0.17\n # cmap = plt.get_cmap('Oranges') # colorbrewer.sequential.Oranges_3.mpl_colors\n\n ax1.bar(ephem.hours(block[\"RA\"]) - math.radians(3.5), extent, linewidth=0.1,\n width=width, bottom=inner_limit, zorder=0, color=colour, alpha=alpha)\n if label_blocks:\n ax1.annotate(blockname[3], (ephem.hours(block[\"RA\"]) + math.radians(0.36), extent - 3.), size=15,\n color='b')\n\n # No optional on these just yet\n plot_planets_plus_Pluto(ax1)\n ra, dist, hlat, Hmag = parsers.synthetic_model_kbos(kbotype='resonant', arrays=True, maglimit=24.7)\n # can't plot Hmag as marker size in current setup.\n ax1.scatter(ra, dist, marker='o', s=2, facecolor=plot_fanciness.ALMOST_BLACK,\n edgecolor=plot_fanciness.ALMOST_BLACK, linewidth=0.1, alpha=0.12, zorder=1)\n\n if plot_discoveries is not None:\n plot_ossos_discoveries(ax1, discoveries, plot_discoveries, plot_colossos=plot_colossos)\n\n # special detection in 13AE: Saturn's moon Ijiraq at 2013-04-09 shows inner limit of sensitivity.\n if plot_Ijiraq:\n # Position from Horizons as it's excluded from the list of detections\n ax1.scatter(ephem.hours('14 29 46.57'), 9.805,\n marker='o', s=4, facecolor='b', edgecolor=plot_fanciness.ALMOST_BLACK, linewidth=0.15, alpha=0.8)\n\n plt.draw()\n if savefilename is not None:\n outfile = '{}.pdf'.format(savefilename)\n else:\n outfile = 'topdown_RA_d_OSSOS_v{}.pdf'.format(parameters.RELEASE_VERSION)\n plt.savefig(outfile, transparent=True, bbox_inches='tight')\n\n return", "response": "Plots the top - down Solar System on a given set of discoveries."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nplot the discovered objects in the OSSOS format.", "response": "def plot_ossos_discoveries(ax, discoveries, plot_discoveries,\n plot_colossos=False, split_plutinos=False):\n \"\"\"\n plotted at their discovery locations, provided by the Version Releases in decimal degrees.\n \"\"\"\n fc = ['b', '#E47833', 'k']\n alpha = [0.85, 0.6, 1.]\n marker = ['o', 'd']\n size = [7, 25]\n\n plottable = [] # Which blocks' discoveries to include?\n for d in discoveries:\n for n in plot_discoveries:\n if d['object'].startswith(n): # can for sure be better, but this hack works. Need to get where going\n plottable.append(d)\n\n # Hack to get in the O15BD objects\n # directory_name = '/Users/bannisterm/Dropbox/OSSOS/measure3/ossin/D_tmp/'\n # kbos = parsers.ossos_discoveries(directory_name, all_objects=False, data_release=None)\n # for kbo in kbos:\n # plottable_kbo = {'RAdeg': kbo.discovery.coordinate.ra.to_string(unit=units.degree, sep=':'),\n # 'dist': kbo.orbit.distance.value}\n # plottable.append(plottable_kbo)\n\n if plot_colossos:\n fainter = []\n colossos = []\n for n in plottable:\n if n['object'] in parameters.COLOSSOS:\n colossos.append(n)\n else:\n fainter.append(n)\n plot_ossos_points(fainter, ax, marker[0], size[0], fc[0], alpha[1], 1)\n plot_ossos_points(colossos, ax, marker[1], size[1], fc[2], alpha[2], 2)\n elif split_plutinos:\n # plutino_index = np.where((plottable['cl'] == 'res') & (plottable['j'] == 3) & (plottable['k'] == 2))\n raise NotImplementedError\n else:\n plot_ossos_points(plottable, ax, marker[0], size[0], fc[0], alpha[0], 2)\n\n return"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef orbit_fit_residuals(discoveries, blockID='O13AE'):\n ra_residuals = []\n dec_residuals = []\n ressum = []\n for i, orbit in enumerate(discoveries):\n if (orbit.da / orbit.a) < 0.01:\n print i, len(discoveries), orbit.observations[0].provisional_name, orbit.da / orbit.a\n res = orbit.residuals.split('\\n')\n for r in res:\n rr = r.split(' ')\n if (len(rr) > 1) and not (rr[0].startswith('!')):\n ra_residuals.append(float(r.split(' ')[4]))\n dec_residuals.append(float(r.split(' ')[5]))\n ressum.append(1000 * (float(r.split(' ')[4]) ** 2 + float(r.split(' ')[5]) ** 2))\n\n plt.figure(figsize=(6, 6))\n bins = [r for r in range(-25, 150, 25)]\n n, bins, patches = plt.hist(ressum, histtype='step', color='b', bins=bins, label='$dra^{2}+ddec^{2}$')\n # midpoints = [r/1000. for r in range(-350,350,50)]\n # popt, pcov = curve_fit(gaussian, midpoints, n)\n # print 'a', popt[0], 'mu', popt[1], 'sigma', popt[2]\n # xm = np.linspace(-.375, .375, 100) # 100 evenly spaced points\n # plt.plot(xm, gaussian(xm, popt[0], popt[1], popt[2]), ls='--', color='#E47833', linewidth=2)\n # sigma = r'$\\sigma = %.2f \\pm %.2f$' % (popt[2], np.sqrt(pcov[2, 2]))\n # plt.annotate(sigma, (0.12, 50), color='#E47833')\n plt.xlim((-25, 150)) # sufficient for 13AE data\n plt.ylabel('observations of orbits with d(a)/a < 1% (i.e 3-5 month arcs)')\n plt.xlabel('orbit fit residuals (milliarcsec)')\n plt.legend()\n # clean_legend()\n plt.draw()\n outfile = 'orbit_fit_residuals_13AE_corrected'\n plt.savefig('/ossos_sequence/'+ outfile + '.pdf', transparent=True)\n\n return", "response": "Brett is the actual orbit fit residuals\n for orbits with d ( a ) < 1%."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_irmc_firmware_version(snmp_client):\n\n try:\n bmc_name = snmp_client.get(BMC_NAME_OID)\n irmc_firm_ver = snmp_client.get(IRMC_FW_VERSION_OID)\n return ('%(bmc)s%(sep)s%(firm_ver)s' %\n {'bmc': bmc_name if bmc_name else '',\n 'firm_ver': irmc_firm_ver if irmc_firm_ver else '',\n 'sep': '-' if bmc_name and irmc_firm_ver else ''})\n except SNMPFailure as e:\n raise SNMPIRMCFirmwareFailure(\n SNMP_FAILURE_MSG % (\"GET IRMC FIRMWARE VERSION\", e))", "response": "Get the irmc firmware version of the node."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets the bios firmware version of the node.", "response": "def get_bios_firmware_version(snmp_client):\n \"\"\"Get bios firmware version of the node.\n\n :param snmp_client: an SNMP client object.\n :raises: SNMPFailure if SNMP operation failed.\n :returns: a string of bios firmware version.\n \"\"\"\n\n try:\n bios_firmware_version = snmp_client.get(BIOS_FW_VERSION_OID)\n return six.text_type(bios_firmware_version)\n except SNMPFailure as e:\n raise SNMPBIOSFirmwareFailure(\n SNMP_FAILURE_MSG % (\"GET BIOS FIRMWARE VERSION\", e))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget the server model of the node.", "response": "def get_server_model(snmp_client):\n \"\"\"Get server model of the node.\n\n :param snmp_client: an SNMP client object.\n :raises: SNMPFailure if SNMP operation failed.\n :returns: a string of server model.\n \"\"\"\n\n try:\n server_model = snmp_client.get(SERVER_MODEL_OID)\n return six.text_type(server_model)\n except SNMPFailure as e:\n raise SNMPServerModelFailure(\n SNMP_FAILURE_MSG % (\"GET SERVER MODEL\", e))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the authorization data for an SNMP request.", "response": "def _get_auth(self):\n \"\"\"Return the authorization data for an SNMP request.\n\n :returns: A\n :class:`pysnmp.entity.rfc3413.oneliner.cmdgen.CommunityData`\n object.\n \"\"\"\n if self.version == SNMP_V3:\n # Handling auth/encryption credentials is not (yet) supported.\n # This version supports a security name analogous to community.\n return cmdgen.UsmUserData(self.security)\n else:\n mp_model = 1 if self.version == SNMP_V2C else 0\n return cmdgen.CommunityData(self.community, mpModel=mp_model)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nusing PySNMP to perform an SNMP GET operation on a single object.", "response": "def get(self, oid):\n \"\"\"Use PySNMP to perform an SNMP GET operation on a single object.\n\n :param oid: The OID of the object to get.\n :raises: SNMPFailure if an SNMP request fails.\n :returns: The value of the requested object.\n \"\"\"\n try:\n results = self.cmd_gen.getCmd(self._get_auth(),\n self._get_transport(),\n oid)\n except snmp_error.PySnmpError as e:\n raise SNMPFailure(SNMP_FAILURE_MSG % (\"GET\", e))\n\n error_indication, error_status, error_index, var_binds = results\n\n if error_indication:\n # SNMP engine-level error.\n raise SNMPFailure(SNMP_FAILURE_MSG % (\"GET\", error_indication))\n\n if error_status:\n # SNMP PDU error.\n raise SNMPFailure(\n \"SNMP operation '%(operation)s' failed: %(error)s at\"\n \" %(index)s\" %\n {'operation': \"GET\", 'error': error_status.prettyPrint(),\n 'index':\n error_index and var_binds[int(error_index) - 1]\n or '?'})\n\n # We only expect a single value back\n name, val = var_binds[0]\n return val"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_next(self, oid):\n try:\n results = self.cmd_gen.nextCmd(self._get_auth(),\n self._get_transport(),\n oid)\n except snmp_error.PySnmpError as e:\n raise SNMPFailure(SNMP_FAILURE_MSG % (\"GET_NEXT\", e))\n\n error_indication, error_status, error_index, var_binds = results\n\n if error_indication:\n # SNMP engine-level error.\n raise SNMPFailure(\n SNMP_FAILURE_MSG % (\"GET_NEXT\", error_indication))\n\n if error_status:\n # SNMP PDU error.\n raise SNMPFailure(\n \"SNMP operation '%(operation)s' failed: %(error)s at\"\n \" %(index)s\" %\n {'operation': \"GET_NEXT\", 'error': error_status.prettyPrint(),\n 'index':\n error_index and var_binds[int(error_index) - 1]\n or '?'})\n\n return [val for row in var_binds for name, val in row]", "response": "Use PySNMP to perform an SNMP GET NEXT operation on a table object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set(self, oid, value):\n try:\n results = self.cmd_gen.setCmd(self._get_auth(),\n self._get_transport(),\n (oid, value))\n except snmp_error.PySnmpError as e:\n raise SNMPFailure(SNMP_FAILURE_MSG % (\"SET\", e))\n\n error_indication, error_status, error_index, var_binds = results\n\n if error_indication:\n # SNMP engine-level error.\n raise SNMPFailure(SNMP_FAILURE_MSG % (\"SET\", error_indication))\n\n if error_status:\n # SNMP PDU error.\n raise SNMPFailure(\n \"SNMP operation '%(operation)s' failed: %(error)s at\"\n \" %(index)s\" %\n {'operation': \"SET\", 'error': error_status.prettyPrint(),\n 'index':\n error_index and var_binds[int(error_index) - 1]\n or '?'})", "response": "Use PySNMP to perform an SNMP SET operation on a single object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the name of the MOP formatted file to parse.", "response": "def filename(self):\n \"\"\"\n Name if the MOP formatted file to parse.\n @rtype: basestring\n @return: filename\n \"\"\"\n if self._filename is None:\n self._filename = storage.get_file(self.basename,\n self.ccd,\n ext=self.extension,\n version=self.type,\n prefix=self.prefix)\n return self._filename"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _parse(self):\n with open(self.filename, 'r') as fobj:\n lines = fobj.read().split('\\n')\n\n # Create a header object with content at start of file\n self.header = MOPHeader(self.subfmt).parser(lines)\n\n # Create a data attribute to hold an astropy.table.Table object\n self.data = MOPDataParser(self.header).parse(lines)", "response": "read in a file and return a MOPFile object."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef table(self):\n if self._table is None:\n column_names = []\n for fileid in self.header.file_ids:\n for column_name in self.header.column_names:\n column_names.append(\"{}_{}\".format(column_name, fileid))\n column_names.append(\"ZP_{}\".format(fileid))\n if len(column_names) > 0:\n self._table = Table(names=column_names)\n else:\n self._table = Table()\n return self._table", "response": "Returns the astropy. table. Table object that will contain the data result\n "} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngive a set of lines parse the into a MOP Header", "response": "def parser(self, lines):\n \"\"\"Given a set of lines parse the into a MOP Header\"\"\"\n while len(lines) > 0:\n if lines[0].startswith('##') and lines[1].startswith('# '):\n # A two-line keyword/value line starts here.\n self._header_append(lines.pop(0), lines.pop(0))\n elif lines[0].startswith('# '):\n # Lines with single comments are exposure numbers unless preceeded by double comment line\n self._append_file_id(lines.pop(0))\n elif lines[0].startswith('##'):\n # Double comment lines without a single comment following are column headers for dataset.\n self._set_column_names(lines.pop(0)[2:])\n else:\n # Last line of the header reached.\n return self\n raise IOError(\"Failed trying to read header\")"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncomputes the MJD from the MJD - OBS - CENTER keyword and the given list of values.", "response": "def _compute_mjd(self, kw, val):\n \"\"\"\n Sometimes that MJD-OBS-CENTER keyword maps to a three component string, instead of a single value.\n @param kw: a list of keywords\n @param val: a list of matching values\n @return: the MJD-OBS-CENTER as a julian date.\n \"\"\"\n try:\n idx = kw.index('MJD-OBS-CENTER')\n except ValueError:\n return\n if len(val) == len(kw):\n return\n if len(val) - 2 != len(kw):\n raise ValueError(\"convert: keyword/value lengths don't match: {}/{}\".format(kw, val))\n val.insert(idx, Time(\"{} {} {}\".format(val.pop(idx), val.pop(idx), val.pop(idx)),\n format='mpc',\n scale='utc').mjd)\n logging.debug(\"Computed MJD: {}\".format(val[idx]))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a dictionary of information about a particular exposure", "response": "def getExpInfo(expnum):\n \"\"\"Return a dictionary of information about a particular exposure\"\"\"\n\n col_names=['object',\n 'e.expnum',\n 'mjdate',\n 'uttime',\n 'filter',\n 'elongation',\n 'obs_iq_refccd',\n 'triple', 'qso_status']\n\n sql=\"SELECT \"\n sep=\" \"\n for col_name in col_names:\n sql=sql+sep+col_name\n sep=\",\"\n sql=sql+\" FROM bucket.exposure e \"\n sql=sql+\" JOIN bucket.circumstance c ON e.expnum=c.expnum \"\n sql=sql+\" LEFT JOIN triple_members t ON e.expnum=t.expnum \"\n sql=sql+\" WHERE e.expnum=%d \" % ( expnum ) \n #sys.stderr.write(sql);\n cfeps.execute(sql)\n rows=cfeps.fetchall()\n #print rows\n result={}\n #import datetime\n for idx in range(len(rows[0])):\n result[col_names[idx]]=rows[0][idx]\n \n return(result)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef getTripInfo(triple):\n\n col_names=['mjdate', 'filter', 'elongation', 'discovery','checkup', 'recovery', 'iq','block' ]\n sql=\"SELECT mjdate md,\"\n sql=sql+\" filter, avg(elongation), d.id, checkup.checkup, recovery.recovery , avg(obs_iq_refccd), b.qname \"\n sql=sql+\"FROM triple_members t JOIN bucket.exposure e ON t.expnum=e.expnum \"\n sql=sql+\"JOIN bucket.blocks b ON b.expnum=e.expnum \"\n sql=sql+\"JOIN bucket.circumstance c on e.expnum=c.expnum \"\n sql=sql+\"LEFT JOIN discovery d ON t.triple=d.triple \"\n sql=sql+\"LEFT JOIN checkup ON t.triple=checkup.triple \"\n sql=sql+\"LEFT JOIN recovery ON t.triple=recovery.triple \"\n sql=sql+\"WHERE t.triple=%s \"\n sql=sql+\"GROUP BY t.triple ORDER BY t.triple \"\n cfeps.execute(sql,(triple, ) )\n rows=cfeps.fetchall()\n result={}\n #import datetime\n for idx in range(len(rows[0])):\n result[col_names[idx]]=rows[0][idx]\n\n return result", "response": "Return a dictionary of information about a particular triple"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets all exposures of specified pointing ID.", "response": "def getExpnums(pointing,night=None):\n \"\"\"Get all exposures of specified pointing ID.\n\n Default is to return a list of all exposure numbers\"\"\"\n\n if night:\n night=\" floor(e.mjdate-0.0833)=%d \" % ( night ) \n else:\t\n night=''\n \n sql=\"SELECT e.expnum \"\n sql=sql+\"FROM bucket.exposure e \"\n sql=sql+\"JOIN bucket.association a on e.expnum=a.expnum \"\n sql=sql+\"WHERE a.pointing=\"+str(pointing)+\" AND \"+night\n sql=sql+\" ORDER BY mjdate, uttime DESC \"\n cfeps.execute(sql)\n\n return(cfeps.fetchall())"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef getTriples(pointing):\n\n sql=\"SELECT id FROM triples t join triple_members m ON t.id=m.triple\"\n sql+=\" join bucket.exposure e on e.expnum=m.expnum \"\n sql+=\" WHERE pointing=%s group by id order by e.expnum \"\n cfeps.execute(sql, ( pointing, ) )\n return(cfeps.fetchall())", "response": "Get all triples of a specified pointing ID."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a new list of triples based on new images in the db.", "response": "def createNewTriples(Win):\n \"\"\"Add entries to the triples tables based on new images in the db\"\"\"\n\n win.help(\"Building list of exposures to look for triples\")\n\n cols=('e.expnum', 'object',\n 'mjdate',\n 'uttime',\n 'elongation',\n 'filter',\n 'obs_iq_refccd','qso_status' )\n header='%6s %-10s%-12s%10s%10s%10s%8s%10s' % cols \n\n\n pointings=getNewTriples()\n num_p=len(pointings)\n\n for pointing in pointings:\n pid=pointing[0]\n\tmjd=pointing[1]\n expnums=getExpnums(pointing=pid,night=mjd)\n num_p=num_p-1\n while (1):\n ### Loop over this pointing until keystroke gets us out\n\n win.help(\"Select (space) members of triplets - %d remaining\" % num_p )\n\n ## start with an empty list\n explist=[]\n choices=[]\n\n current_date=''\n for expnum in expnums:\n info=getExpInfo(expnum[0])\n row=()\n if not str(info['triple'])=='None' :\n continue\n if str(info['obs_iq_refccd'])=='None':\n info['obs_iq_refccd']=-1.0\n\n choices.append('%6d %10s %15s %10s %8.2f %10s %8.2f %10s' % (\n int(info['e.expnum']),\n str(info['object']),\n str(info['mjdate']),\n str(info['uttime']),\n float(str(info['elongation'])),\n str(info['filter']),\n float(str(info['obs_iq_refccd'])),\n str(info['qso_status'])\n ))\n explist.append(expnum[0])\n\n if len(choices)<3:\n ### we need to provide at least 3 choices,\n ### otherwise this isn't a triple (is it)\n break\n\n ### win.list returns the user's choices as a list.\n choice_list=win.list(header,choices)\n\n ### zero length list implies were done.\n if choice_list==None:\n break\n\n ### is this actually a triple?\n if len(choice_list)!=3:\n win.help(\"Must have 3 members to make a tripple\")\n continue\n\n ### Create a new line in the triple table\n sql = \"INSERT INTO triples (id, pointing ) VALUES ( NULL, %s ) \"\n cfeps.execute(sql, ( pid, ) )\n\t sql = \"SELECT id FROM triples WHERE pointing=%s order by id desc\" \n\t cfeps.execute(sql, ( pid, ) )\n\t ttt=cfeps.fetchall()\n triple= ttt[0][0]\n\t win.help(str(triple))\n\n ### record the members of this new triple.\n sql = \"INSERT INTO triple_members (triple, expnum) VALUES ( %s, %s)\";\n\t win.help(sql)\n\n for exp in choice_list:\n cfeps.execute(sql,(triple,explist[exp]))\n\n return(0)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nprovide user with a list of triples that could be discovery triples", "response": "def setDiscoveryTriples(win,table=\"discovery\"):\n \"\"\"Provide user with a list of triples that could be discovery triples\"\"\"\n\n win.help(\"Getting a list of pointings with triples from the CFEPS db\")\n\n pointings=getPointingsWithTriples()\n win.help(\"Select the \"+table+\" triple form the list...\")\n import time\n for pointing in pointings:\n header=\"%10s %10s %8s %10s %8s\" % (pointing[1],'mjdate','Elongation','Filter', 'IQ')\n triples=getTriples(pointing=pointing[0])\n choices=[]\n triplist=[]\n\tno_type=0\n\tprevious_list=[]\n for triple in triples:\n\t #win.help(str(triple))\n tripinfo=getTripInfo(triple[0])\n if not tripinfo[table]==None:\n\t previous_list.append(triple[0])\n\t #if not abs(180-tripinfo['elongation'])< 20:\n\t #\tcontinue\n triplist.append(triple)\n if str(tripinfo['iq'])=='None':\n tripinfo['iq']=-1.0 \n obs_type=' '\n\t if tripinfo['discovery']:\n\t \tobs_type='D'\n\t elif tripinfo['checkup']:\n\t obs_type='C'\n elif tripinfo['recovery']:\n\t obs_type='R'\n\t if obs_type==' ':\n\t no_type+=1\n line=(obs_type,tripinfo['mjdate'], tripinfo['elongation'],\n tripinfo['filter'], tripinfo['iq'], tripinfo['block'] ) \n choices.append('%10s %10s %8.2f %10s %8.2f %8s' % line)\n if len(choices)==0 or no_type==0:\n continue\n\t#if len(previous_list)==1:\n\t# continue\n win.help(\"Choose a \"+table+\" triple (space) [no choice means skip] then press enter\\n (q) to exit\")\n choice=win.list(header,choices)\n if choice==None:\n win.help(\"Loading next triple\")\n break\n ### Record which triplet is a discovery triplet\n if len(choice)!=1:\n win.help(\"Loading next triple\\n\")\n continue\n discovery_triple=triplist[choice[0]]\n\tfor triple in previous_list:\n\t sql=\"DELETE FROM \"+table+\" WHERE triple=%s \"\n\t cfeps.execute(sql,triple)\n sql=\"INSERT INTO \"+table+\" ( triple ) VALUES ( %s ) \"\n cfeps.execute(sql,discovery_triple)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nruns the actual step1jmp code for a given CFHT entry.", "response": "def run(expnum,\n ccd,\n prefix='',\n version='p',\n sex_thresh=_SEX_THRESHOLD,\n wave_thresh=_WAVE_THRESHOLD,\n maxcount=_MAX_COUNT,\n dry_run=False, \n force=True):\n \"\"\"run the actual step1jmp/matt codes.\n\n expnum: the CFHT expousre to process\n ccd: which ccd in the mosaic to process\n fwhm: the image quality, FWHM, of the image. In pixels.\n sex_thresh: the detection threhold to run sExtractor at\n wave_thresh: the detection threshold for wavelet\n maxcount: saturation level\n\n \"\"\"\n message = storage.SUCCESS\n\n if storage.get_status(task, prefix, expnum, version, ccd) and not force:\n logging.info(\"{} completed successfully for {} {} {} {}\".format(task, prefix, expnum, version, ccd))\n return\n \n with storage.LoggingManager(task, prefix, expnum, ccd, version, dry_run):\n try: \n if not storage.get_status(dependency, prefix, expnum, version, ccd):\n raise IOError(35, \"Cannot start {} as {} not yet completed for {}{}{}{:02d}\".format(\n task, dependency, prefix, expnum, version, ccd))\n logging.info(\"Retrieving imaging and input parameters from VOSpace\") \n storage.get_file(expnum, ccd, prefix=prefix, version=version, ext='mopheader')\n filename = storage.get_image(expnum, ccd, version=version, prefix=prefix)\n fwhm = storage.get_fwhm(expnum, ccd, prefix=prefix, version=version)\n basename = os.path.splitext(filename)[0]\n\n _get_weight_map(filename, ccd)\n \n logging.info(\"Launching step1jmp\")\n logging.info(util.exec_prog(['step1jmp',\n '-f', basename,\n '-t', str(wave_thresh),\n '-w', str(fwhm),\n '-m', str(maxcount)]))\n\n logging.info(util.exec_prog(['step1matt',\n '-f', basename,\n '-t', str(sex_thresh),\n '-w', str(fwhm),\n '-m', str(maxcount)]))\n\n if os.access('weight.fits', os.R_OK):\n os.unlink('weight.fits')\n \n if not dry_run:\n for ext in ['obj.jmp', 'obj.matt']:\n obj_uri = storage.get_uri(expnum, ccd, version=version, ext=ext,\n prefix=prefix)\n obj_filename = basename + \".\" + ext\n count = 0\n with open(obj_filename, 'r'):\n while True:\n try:\n count += 1\n logging.info(\"Attempt {} to copy {} -> {}\".format(count, obj_filename, obj_uri))\n storage.copy(obj_filename, obj_uri)\n break\n except Exception as ex:\n if count > 10:\n raise ex\n logging.info(message)\n except Exception as ex:\n message = str(ex)\n logging.error(message)\n\n if not dry_run:\n storage.set_status(task, prefix, expnum, version, ccd, status=message)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nquery the CADC TAP service to determine the list of images that overlap with the given reference image.", "response": "def query(ra ,dec, rad=0.1, query=None):\n \"\"\"Query the CADC TAP service to determine the list of images for the\nNewHorizons Search. Things to determine:\n \n\n a- Images to have the reference subtracted from.\n b- Image to use as the 'REFERENCE' image.\n c- Images to be used for input into the reference image\n\nLogic: Given a particular Image/CCD find all the CCDs of the same field that\noverlap that CCD but are taken more than 7 days later or earlier than\nthat image.\n\n \"\"\"\n if query is None:\n query=( \"\"\" SELECT \"\"\"\n \"\"\" \"II/246/out\".raj2000 as ra, \"II/246/out\".dej2000 as dec, \"II/246/out\".jmag as jmag \"\"\"\n \"\"\" FROM \"II/246/out\" \"\"\"\n \"\"\" WHERE \"\"\"\n \"\"\" CONTAINS(POINT('ICRS', raj2000, dej2000), CIRCLE('ICRS', {}, {}, {})) = 1 \"\"\".format(ra,dec,rad) )\n\n tapURL = \"http://TAPVizieR.u-strasbg.fr/TAPVizieR/tap/sync\"\n\n\n ## Some default parameters for that TAP service queries.\n tapParams={'REQUEST': 'doQuery',\n 'LANG': 'ADQL',\n 'FORMAT': 'votable',\n 'QUERY': query}\n\n response = requests.get(tapURL, params=tapParams)\n data = StringIO(response.text)\n data.seek(0)\n data.seek(0)\n T = votable.parse_single_table(data).to_table()\n\n return T"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef objIngest(obj_file):\n import sys,os,math,re\n \n import pyfits\n \n import MOPfiles\n\n obj=MOPfiles.read(obj_file)\n\n \"\"\"\n The SQL description of the source table\n +-------------+---------+------+-----+---------+----------------+\n | Field | Type | Null | Key | Default | Extra |\n +-------------+---------+------+-----+---------+----------------+\n | sourceID | int(11) | | PRI | NULL | auto_increment |\n | x_pix | float | YES | MUL | NULL | |\n | y_pix | float | YES | | NULL | |\n | iso_flux | float | YES | | NULL | |\n | iso_err | float | YES | | NULL | |\n | aper_flux | float | YES | | NULL | |\n | aper_err | float | YES | | NULL | |\n | iso_area | float | YES | | NULL | |\n | kron_radius | float | YES | MUL | NULL | |\n | elongation | float | YES | | NULL | |\n | cxx | float | YES | | NULL | |\n | cyy | float | YES | | NULL | |\n | cxy | float | YES | | NULL | |\n | max_flux | float | YES | | NULL | |\n | max_int | float | YES | | NULL | |\n | mag_dao | float | YES | MUL | NULL | |\n | merr_dao | float | YES | | NULL | |\n | sky_cts | float | YES | | NULL | |\n | chi2 | float | YES | | NULL | |\n | npix | float | YES | | NULL | |\n | sharp | float | YES | | NULL | |\n | ra_deg | float | YES | MUL | NULL | |\n | dec_deg | float | YES | | NULL | |\n +-------------+---------+------+-----+---------+----------------+\n \"\"\"\n\n\n \"\"\"\n Columns in the SOURCE table...\n ## X Y FLUX_ISO FLUXERR_ISO FLUX_APER FLUXERR_APER ISOAREA_IMAGE KRON_RADIUS ELONGATION CXX_IMAGE CYY_IMAGE CXY_IMAGE FLUX_MAX ID MAX_INT FLUX MERR SKY ELON X^2 N_PIX MAG SHARP SIZE\n\n \"\"\"\n\n ### The mapping \n obj['hdu2sql']={'MAX_INT': 'peak',\n 'FLUX': 'flux',\n\t\t 'MAG': 'mag',\n\t\t 'MERR': 'merr',\n 'SKY': 'sky',\n 'ELON': 'elongation',\n 'X^2': 'chi2',\n 'N_PIX': 'npix',\n 'SHARP': 'sharpness',\n 'Y': 'yPix',\n 'X': 'xPix',\n 'SIZE': 'size',\n 'RA': 'raDeg',\n 'DEC': 'decDeg',\n }\n\n\n MOPfiles.store(obj)\n\n return", "response": "Ingests the source table of the object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nloading the targets from an EDB file", "response": "def load_edbfile(file=None):\n \"\"\"Load the targets from a file\"\"\"\n import ephem,string,math\n if file is None:\n import tkFileDialog\n\ttry:\n file=tkFileDialog.askopenfilename()\n except:\n return \n if file is None or file == '':\n return\n f=open(file)\n lines=f.readlines()\n f.close()\n for line in lines:\n p=line.split(',')\n\tname=p[0].strip().upper()\n\tmpc_objs[name]=ephem.readdb(line)\n\tmpc_objs[name].compute()\n objInfoDict[name]=\"%6s %6s %6s\\n\" % ( string.center(\"a\",6),\n\t\t\t\t string.center(\"e\",6),\n\t \t\t\t\t string.center(\"i\",6) ) \n objInfoDict[name]+=\"%6.2f %6.3f %6.2f\\n\" % (mpc_objs[name]._a,mpc_objs[name]._e,math.degrees(mpc_objs[name]._inc))\n objInfoDict[name]+=\"%7.2f %7.2f\\n\" % ( mpc_objs[name].earth_distance, mpc_objs[name].mag)\n doplot(mpc_objs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load_abgfiles(dir=None):\n import ephem,string\n if dir is None:\n import tkFileDialog\n\ttry: \n dir=tkFileDialog.askdirectory()\n except:\n return\n if dir is None:\n return None\n from glob import glob\n files=glob(dir+\"/*.abg\")\n import os\n for f in files:\n (name,ext)=os.path.splitext(os.path.basename(f))\n\tNAME=name.strip().upper()\n\tkbos[name]=f\n\t#print name\n aei=file(dir+\"/\"+name+\".aei\")\n lines=aei.readlines()\n aei.close()\n objInfoDict[name]=\"%6s %6s %6s\\n\" % ( string.center(\"a\",6),\n\t\t\t\t string.center(\"e\",6),\n\t \t\t\t\t string.center(\"i\",6) ) \n try:\n (a,e,i,N,w,T) = lines[4].split()\n except:\n print lines[4]\n (a,e,i,N,w,T) = (0,0,0,0,0,0)\n objInfoDict[name]+=\"%6.2f %6.3f %6.2f\\n\" % (float(a),float(e),float(i))\n s=lines[5][0:2]\n (a,e,i,N,w,T) = lines[5][2:-1].split()\n objInfoDict[name]+=\" %6.3f %6.3f %6.3f\\n\" % (float(a),float(e),float(i))\n abg = file(dir+\"/\"+name+\".abg\")\n lines = abg.readlines()\n abg.close()\n\tfor line in lines:\n\t if not line[0:5] == \"# Bar\":\n\t continue\n objInfoDict[name]+=line[2:-1]+\"\\n\"\n break\n res=file(dir+\"/\"+name+\".res\")\n lines=res.readlines()\n res.close()\n line=lines.pop()\n values=line.split()\n s = \"[nobs: \"+values[0]+\" dt: \"+values[1]+\"]\\n\"\n objInfoDict[name]+=s\n mpc = file(dir+\"/../mpc/\"+name+\".mpc\")\n lines=mpc.readlines()\n mpc.close()\n\tlast_date=0\n\tfor line in lines:\n\t if len(line)==0:\n\t\tcontinue\n\t if line[0]==\"#\":\n\t\tcontinue\n this_year=int(line[15:19])\n this_month=int(line[20:22])\n this_day = int(line[23:25])\n\t this_date = this_year+this_month/12.0 +this_day/365.25\n\t if last_date < this_date:\n\t\tlast_date=this_date\n\t\tyear=this_year\n\t\tday = this_day\n\t\tmonth=this_month\t\n mag={}\n for line in lines:\n try:\n mags=line[65:69].strip()\n\t\tif len(mags)==0:\n\t\t continue\n filter=line[70]\n\t if filter not in mag:\n mag[filter]=[]\n mag[filter].append(float(mags)) \n\t except:\n\t continue\n mags=''\n for filter in mag:\n\t magv=0\n \t for m in mag[filter]:\n\t magv = magv+m/len(mag[filter]) \n mags = mags+ \"%4.1f-%s \" % ( magv , filter)\n if len(mags)==0:\n\t mags= \"N/A\"\n objInfoDict[name]+=\"MAG: \"+mags+\"\\n\"\n objInfoDict[name]+=\"Last obs: %s %s %s \\n\" % (year,month, day)\n\n doplot(kbos)", "response": "Load the targets from a file"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef fits_list(filter,root,fnames):\n import re, pyfits, wcsutil\n for file in fnames:\n if re.match(filter,file):\n fh=pyfits.open(file)\n for ext in fh:\n obj=ext.header.get('OBJECT',file)\n dx=ext.header.get('NAXIS1',None)\n dy=ext.header.get('NAXIS2',None)\n wcs=wcsutil.WCSObject(ext)\n (x1,y1)=wcs.xy2rd((1,1,))\n (x2,y2)=wcs.xy2rd((dx,dy))\n ccds=[x1,y1,x2,y2]\n pointing={'label': obj, 'camera': ccds}\n return files", "response": "Get a list of files matching filter in directory root"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nloads fits images in a directory", "response": "def load_fis(dir=None):\n \"\"\"Load fits images in a directory\"\"\"\n if dir is None:\n import tkFileDialog\n try:\n dir=tkFileDialog.askdirectory()\n except:\n return\n if dir is None:\n return None\n from os.path import walk\n walk(dir,fits_list,\"*.fits\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndraw the actual plot of the objects in the kbos.", "response": "def do_objs(kbos):\n \"\"\"Draw the actual plot\"\"\"\n\n import orbfit, ephem, math\n import re\n re_string=w.FilterVar.get()\n vlist=[]\n for name in kbos:\n if not re.search(re_string,name):\n continue\n vlist.append(name)\n if type(kbos[name])==type(ephem.EllipticalBody()):\n\t kbos[name].compute(w.date.get())\n\t ra=kbos[name].ra\n\t dec=kbos[name].dec\n\t a=math.radians(10.0/3600.0)\n\t b=a\n\t ang=0.0\n\t color='blue'\n yoffset=+10\n\t xoffset=+10\n else:\n yoffset=-10\n xoffset=-10\n file=kbos[name]\n\t jdate=ephem.julian_date(w.date.get())\n\t obs=568\n \t try:\n\t position=orbfit.predict(file,jdate,obs)\n\t except:\n\t\tcontinue\n\t ra=math.radians(position[0])\n\t dec=math.radians(position[1])\n\t a=math.radians(position[2]/3600.0)\n\t b=math.radians(position[3]/3600.0)\n\t ang=math.radians(position[4])\n\t if ( a> math.radians(1.0) ):\n\t color='green'\n else:\n\t color='black'\n\tif w.show_ellipse.get()==1 :\n if ( a < math.radians(5.0) ): \n\t w.create_ellipse(ra,dec,a,b,ang)\n if ( a < math.radians(1.0) ): \n w.create_point(ra,dec,size=2,color=color)\n\tif w.show_labels.get()==1:\n\t w.label(ra,dec,name,offset=[xoffset,yoffset])\n vlist.sort()\n for v in vlist:\n w.objList.insert(END,v)\n w.plot_pointings()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nprints the canvas to a postscript file", "response": "def eps(self):\n\t\"\"\"Print the canvas to a postscript file\"\"\"\n\n import tkFileDialog,tkMessageBox\n filename=tkFileDialog.asksaveasfilename(message=\"save postscript to file\",filetypes=['eps','ps'])\n\tif filename is None:\n\t return \n\n\tself.postscript(file=filename)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting from canvas to screen coordinates", "response": "def c2s(self,p=[0,0]):\n \"\"\"Convert from canvas to screen coordinates\"\"\"\n\n return((p[0]-self.canvasx(self.cx1),p[1]-self.canvasy(self.cy1)))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef coord_grid(self):\n\n import math,ephem\n ra2=math.pi*2\n ra1=0\n dec1=-1*math.pi/2.0\n dec2=math.pi/2.0\n\n \n ## grid space choices\n ## ra space in hours\n ra_grids=[\"06:00:00\",\n \"03:00:00\",\n \"01:00:00\",\n \"00:30:00\",\n \"00:15:00\",\n \"00:05:00\",\n \"00:01:00\",\n \"00:00:30\",\n \"00:00:15\",\n \"00:00:05\"]\n dec_grids=[\"45:00:00\",\n \"30:00:00\",\n \"15:00:00\",\n \"05:00:00\",\n \"01:00:00\",\n \"00:15:00\",\n \"00:05:00\",\n \"00:01:00\",\n \"00:00:30\"]\n dra=(self.x2-self.x1)/3.0\n ddec=(self.y2-self.y1)/3.0\n for ra_grid in ra_grids:\n if self.hours(ra_grid) 0 ) :\n\t\t points.append((d[0],d[1],d[2]))\n\t else:\n\t\t points.append(('',math.radians(float(d[1])),math.radians(float(d[2]))))\n\n self.plot_points_list(points)\n\treturn", "response": "Load some pointings from a file."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create_pointing(self,event):\n\n import math\n (ra,dec)=self.c2p((self.canvasx(event.x),\n self.canvasy(event.y)))\n this_camera=camera(camera=self.camera.get())\n ccds=this_camera.getGeometry(ra,dec)\n items=[]\n for ccd in ccds:\n (x1,y1)=self.p2c((ccd[0],ccd[1]))\n (x2,y2)=self.p2c((ccd[2],ccd[3]))\n item=self.create_rectangle(x1,y1,x2,y2)\n items.append(item)\n\tlabel={}\n\tlabel['text']=w.plabel.get()\n\tlabel['id']=self.label(this_camera.ra,this_camera.dec,label['text'])\n self.pointings.append({\n\t\t\"label\": label,\n\t\t\"items\": items,\n\t\t\"camera\": this_camera} )\n self.current_pointing(len(self.pointings)-1)", "response": "Create the sky coverage of pointing at event. x event. y on the canavas."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef plot_pointings(self,pointings=None):\n\n if pointings is None:\n pointings=self.pointings\n\n\ti=0\n for pointing in pointings:\n items=[]\n\t i=i+1\n\t label={}\n\t label['text']=pointing['label']['text']\n for ccd in pointing[\"camera\"].getGeometry():\n (x1,y1)=self.p2c((ccd[0],ccd[1]))\n (x2,y2)=self.p2c((ccd[2],ccd[3]))\n item=self.create_rectangle(x1,y1,x2,y2,stipple='gray25',fill=pointing.get('color',''))\n items.append(item)\n\t if w.show_labels.get()==1:\n\t label['id']=self.label(pointing[\"camera\"].ra,pointing[\"camera\"].dec,label['text'])\n pointing[\"items\"]=items\n\t pointing[\"label\"]=label", "response": "Plot pointings on canavs"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nlets the label of the current pointing to the value in the plabel box", "response": "def set_pointing_label(self):\n\t \"\"\"Let the label of the current pointing to the value in the plabel box\"\"\"\n\n\t self.pointings[self.current]['label']['text']=w.plabel.get()\n\t self.reset()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef save_pointings(self):\n\n import tkFileDialog\n f=tkFileDialog.asksaveasfile()\n i=0\n if self.pointing_format.get()=='CFHT PH':\n\t f.write(\"\"\"\n\n\n\nFixed Targets\nFixed Targets for CFHT QSO\n\n\n Name of target\n\n\n Right ascension of target\n \n\n Declination of target\n \n\n Epoch of coordinates\n \n\nPointing name\n \n \n=0:\n\t\t dec[0]='+'+dec[0]\n\t if self.pointing_format.get()=='Palomar':\n\t\t f.write( \"%5d %16s %2s %2s %4s %3s %2s %4s 2000\\n\" % (i, name,\n\t\t\t\t\t\t\t\t\t ra[0].zfill(2),\n\t\t\t\t\t\t\t\t\t ra[1].zfill(2),\n\t\t\t\t\t\t\t\t\t ra[2].zfill(2),\n\t\t\t\t\t\t\t\t\t dec[0].zfill(3),\n\t\t\t\t\t\t\t\t\t dec[1].zfill(2),\n\t\t\t\t\t\t\t\t\t dec[2].zfill(2)))\n\t elif self.pointing_format.get()=='CFHT PH':\n #f.write(\"%f %f\\n\" % (pointing[\"camera\"].ra,pointing[\"camera\"].dec))\n f.write(\"%-20s|%11s|%11s|%6.1f|%-5d|\\n\" % (name,sra,sdec,2000.0,1))\n elif self.pointing_format.get()=='KPNO/CTIO':\n str1 = sra.replace(\":\",\" \")\n\t str2 = sdec.replace(\":\",\" \")\n f.write(\"%16s %16s %16s 2000\\n\" % ( name, str1, str2) )\n elif self.pointing_format.get()=='SSim':\n ra = []\n \t\tdec= []\n for ccd in pointing[\"camera\"].getGeometry():\n\t\t ra.append(ccd[0])\n\t\t ra.append(ccd[2])\n\t\t dec.append(ccd[1])\n\t\t dec.append(ccd[3])\n \t import math\n \t\tdra=math.degrees(math.fabs(max(ra)-min(ra)))\n \t\tddec=math.degrees(math.fabs(max(dec)-min(dec)))\n f.write(\"%f %f %16s %16s DATE 1.00 1.00 500 FILE\\n\" % (dra, ddec, sra, sdec ) ) \n if self.pointing_format.get()=='CFHT PH':\n\t f.write(\"\"\"]]\n
\n\n\"\"\")\n\tf.close()", "response": "Print the currently defined POINTINGS."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef getGeometry(self,ra=None,dec=None):\n\n import math,ephem\n ccds=[]\n\n if ra is None:\n ra=self.ra\n if dec is None:\n dec=self.dec\n self.ra=ephem.hours(ra)\n self.dec=ephem.degrees(dec)\n for geo in self.geometry[self.camera]:\n ycen=math.radians(geo[\"dec\"])+dec\n xcen=math.radians(geo[\"ra\"])/math.cos(ycen)+ra\n dy=math.radians(geo[\"ddec\"])\n dx=math.radians(geo[\"dra\"]/math.cos(ycen))\n ccds.append([xcen-dx/2.0,ycen-dy/2.0,xcen+dx/2.0,ycen+dy/2.0])\n\n return ccds", "response": "Return an array of rectangles that represent the ra dec corners of the FOV"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncomputes the separation between self and ( ra dec )", "response": "def separation(self,ra,dec):\n \"\"\"Compute the separation between self and (ra,dec)\"\"\"\n\n import ephem\n return ephem.separation((self.ra,self.dec),(ra,dec))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nproducing random binary features totally irrespective of the content of x but in the same shape as x.", "response": "def produce_fake_hash(x):\n \"\"\"\n Produce random, binary features, totally irrespective of the content of\n x, but in the same shape as x.\n \"\"\"\n h = np.random.binomial(1, 0.5, (x.shape[0], 1024))\n packed = np.packbits(h, axis=-1).view(np.uint64)\n return zounds.ArrayWithUnits(\n packed, [x.dimensions[0], zounds.IdentityDimension()])"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconvert a string of hexadecimal values to decimal values parameters", "response": "def _parse_raw_bytes(raw_bytes):\n \"\"\"Convert a string of hexadecimal values to decimal values parameters\n\n Example: '0x2E 0xF1 0x80 0x28 0x00 0x1A 0x01 0x00' is converted to:\n 46, 241, [128, 40, 0, 26, 1, 0]\n\n :param raw_bytes: string of hexadecimal values\n :returns: 3 decimal values\n \"\"\"\n bytes_list = [int(x, base=16) for x in raw_bytes.split()]\n return bytes_list[0], bytes_list[1], bytes_list[2:]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsend raw IPMI command to BMC using raw command object.", "response": "def _send_raw_command(ipmicmd, raw_bytes):\n \"\"\"Use IPMI command object to send raw ipmi command to BMC\n\n :param ipmicmd: IPMI command object\n :param raw_bytes: string of hexadecimal values. This is commonly used\n for certain vendor specific commands.\n :returns: dict -- The response from IPMI device\n \"\"\"\n\n netfn, command, data = _parse_raw_bytes(raw_bytes)\n response = ipmicmd.raw_command(netfn, command, data=data)\n\n return response"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_tpm_status(d_info):\n\n # note:\n # Get TPM support status : ipmi cmd '0xF5', valid flags '0xC0'\n #\n # $ ipmitool raw 0x2E 0xF5 0x80 0x28 0x00 0x81 0xC0\n #\n # Raw response:\n # 80 28 00 C0 C0: True\n # 80 28 00 -- --: False (other values than 'C0 C0')\n\n ipmicmd = ipmi_command.Command(bmc=d_info['irmc_address'],\n userid=d_info['irmc_username'],\n password=d_info['irmc_password'])\n try:\n response = _send_raw_command(ipmicmd, GET_TPM_STATUS)\n if response['code'] != 0:\n raise IPMIFailure(\n \"IPMI operation '%(operation)s' failed: %(error)s\" %\n {'operation': \"GET TMP status\",\n 'error': response.get('error')})\n out = ' '.join('{:02X}'.format(x) for x in response['data'])\n return out is not None and out[-5:] == 'C0 C0'\n\n except ipmi_exception.IpmiException as e:\n raise IPMIFailure(\n \"IPMI operation '%(operation)s' failed: %(error)s\" %\n {'operation': \"GET TMP status\", 'error': e})", "response": "Get the TPM support status of the node."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting output of ipmiraw command and the ordinal numbers.", "response": "def _pci_seq(ipmicmd):\n \"\"\"Get output of ipmiraw command and the ordinal numbers.\n\n :param ipmicmd: IPMI command object.\n :returns: List of tuple contain ordinal number and output of ipmiraw\n command.\n \"\"\"\n for i in range(1, 0xff + 1):\n try:\n res = _send_raw_command(ipmicmd, GET_PCI % hex(i))\n yield i, res\n except ipmi_exception.IpmiException as e:\n raise IPMIFailure(\n \"IPMI operation '%(operation)s' failed: %(error)s\" %\n {'operation': \"GET PCI device quantity\", 'error': e})"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_pci_device(d_info, pci_device_ids):\n\n # note:\n # Get quantity of PCI devices:\n # ipmi cmd '0xF1'\n #\n # $ ipmitool raw 0x2E 0xF1 0x80 0x28 0x00 0x1A 0x01 0x00\n #\n # Raw response:\n # 80 28 00 00 00 05 data1 data2 34 17 76 11 00 04\n # 01\n\n # data1: 2 octet of VendorID\n # data2: 2 octet of DeviceID\n\n ipmicmd = ipmi_command.Command(bmc=d_info['irmc_address'],\n userid=d_info['irmc_username'],\n password=d_info['irmc_password'])\n\n response = itertools.takewhile(\n lambda y: (y[1]['code'] != 0xC9 and y[1].get('error') is None),\n _pci_seq(ipmicmd))\n\n def _pci_count(accm, v):\n out = v[1]['data']\n # if system returns value, record id will be increased.\n pci_id = \"0x{:02x}{:02x}/0x{:02x}{:02x}\".format(\n out[7], out[6], out[9], out[8])\n return accm + 1 if pci_id in pci_device_ids else accm\n\n device_count = functools.reduce(_pci_count, response, 0)\n\n return device_count", "response": "Get quantity of PCI devices of the node."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef defaults(current: dict, *args: AnyMapping) -> dict:\n for data in args:\n for key, value in data.items():\n current.setdefault(key, value)\n return current", "response": "Override current dict with defaults values."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nprovide default function for Schema validation.", "response": "def validate_func_factory(validator_class: Any) -> ValidateFunc:\n \"\"\"Provide default function for Schema validation.\n\n :param validator_class: JSONSchema suitable validator class.\n \"\"\"\n def validate_func(schema: AnyMapping, pure_data: AnyMapping) -> AnyMapping:\n \"\"\"Validate schema with given data.\n\n :param schema: Schema representation to use.\n :param pure_data: Pure data to validate.\n \"\"\"\n return validator_class(schema).validate(pure_data)\n return validate_func"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef moreData(ra,dec,box):\n\n\timport cfhtCutout\n\tcdata={'ra_deg': ra, 'dec_deg': dec, 'radius_deg': 0.2}\n\tinter=cfhtCutout.find_images(cdata,0.2)", "response": "Search the CFHT archive for more images of this location"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nchecks if xpa is running", "response": "def xpacheck():\n\t\"\"\"Check if xpa is running\"\"\"\n\timport os\n\tf=os.popen('xpaaccess ds9')\n\tl=f.readline()\n\tf.close()\n\tif l.strip()!='yes':\n \t\tlogger.debug(\"\\t Can't get ds9 access, xpaccess said: %s\" % ( l.strip()))\n\t\treturn (False)\n\treturn(True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nmarking a circle on the current image", "response": "def mark(x,y,label=None):\n\t\"\"\"Mark a circle on the current image\"\"\"\n\tif label is not None: \n\t os.system(\"xpaset -p ds9 regions color red \")\n\t cmd=\"echo 'image; text %d %d # text={%s}' | xpaset ds9 regions \" % ( x,y,label) \n\telse:\n\t os.system(\"xpaset -p ds9 regions color blue\")\n\t cmd=\"echo 'image; circle %d %d 10 ' | xpaset ds9 regions \" % (x,y)\n\tos.system(cmd)\n\treturn"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef display(url):\n\timport os\n\toscmd=\"curl --silent -g --fail --max-time 1800 --user jkavelaars '%s'\" % (url)\n\tlogger.debug(oscmd)\n \tos.system(oscmd+' | xpaset ds9 fits')\n\treturn", "response": "Display a file in ds9"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ninverting the transform. After calling this method, calling the instance will do the inverse transform. Calling this twice return the instance to the original transform.", "response": "def inv(self):\n \"\"\"Invert the transform.\n\n After calling this method, calling the instance will do the inverse\n transform. Calling this twice return the instance to the original\n transform.\n \"\"\"\n\n self.x, self.y = self.y, self.x\n self._x_, self._y_ = self._y_, self._x_\n self.xfac, self.yfac = 1 / self.yfac, 1 / self.xfac\n self._xfac_, self._yfac_ = 1 / self._yfac_, 1 / self._xfac_\n self._u = 1 / self._u.conj()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef matrix(self, full=False, keeppads=True):\n\n v = np.fft.hfft(self._u, n=self.N) / self.N\n idx = sum(np.ogrid[0:self.N, -self.N:0])\n C = v[idx] # follow scipy.linalg.{circulant,toeplitz,hankel}\n\n if keeppads:\n a = self._yfac_.copy()\n b = self._xfac_.copy()\n else:\n a = self.yfac.copy()\n b = self.xfac.copy()\n C = self._unpad(C, 0, True)\n C = self._unpad(C, 1, False)\n a = a.reshape(-1, 1)\n\n if not full:\n return a, b, C\n else:\n return a * C * b", "response": "Return the matrix form of the integral transform."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nadding padding to an array.", "response": "def _pad(self, a, axis, extrap, out):\n \"\"\"Add padding to an array.\n\n Parameters\n ----------\n a : (..., Nin, ...) ndarray\n array to be padded to size `N`\n axis : int\n axis along which to pad\n extrap : {bool, 'const'} or 2-tuple\n Method to extrapolate `a`.\n For a 2-tuple, the two elements are for the left and right pads,\n whereas a single value applies to both ends.\n Options are:\n * True: power-law extrapolation using the end segment\n * False: zero padding\n * 'const': constant padding with the end point value\n out : bool\n pad the output if True, otherwise the input; the two cases have\n their left and right pad sizes reversed\n \"\"\"\n\n assert a.shape[axis] == self.Nin\n\n axis %= a.ndim # to fix the indexing below with axis+1\n\n to_axis = [1] * a.ndim\n to_axis[axis] = -1\n\n Npad = self.N - self.Nin\n if out:\n _Npad, Npad_ = Npad - Npad//2, Npad//2\n else:\n _Npad, Npad_ = Npad//2, Npad - Npad//2\n\n try:\n _extrap, extrap_ = extrap\n except (TypeError, ValueError):\n _extrap = extrap_ = extrap\n\n if isinstance(_extrap, bool):\n if _extrap:\n end = np.take(a, [0], axis=axis)\n ratio = np.take(a, [1], axis=axis) / end\n exp = np.arange(-_Npad, 0).reshape(to_axis)\n _a = end * ratio ** exp\n else:\n _a = np.zeros(a.shape[:axis] + (_Npad,) + a.shape[axis+1:])\n elif _extrap == 'const':\n end = np.take(a, [0], axis=axis)\n _a = np.repeat(end, _Npad, axis=axis)\n else:\n raise ValueError(\"left extrap not supported\")\n if isinstance(extrap_, bool):\n if extrap_:\n end = np.take(a, [-1], axis=axis)\n ratio = end / np.take(a, [-2], axis=axis)\n exp = np.arange(1, Npad_ + 1).reshape(to_axis)\n a_ = end * ratio ** exp\n else:\n a_ = np.zeros(a.shape[:axis] + (Npad_,) + a.shape[axis+1:])\n elif extrap_ == 'const':\n end = np.take(a, [-1], axis=axis)\n a_ = np.repeat(end, Npad_, axis=axis)\n else:\n raise ValueError(\"right extrap not supported\")\n\n return np.concatenate((_a, a, a_), axis=axis)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nundo padding in an array.", "response": "def _unpad(self, a, axis, out):\n \"\"\"Undo padding in an array.\n\n Parameters\n ----------\n a : (..., N, ...) ndarray\n array to be trimmed to size `Nin`\n axis : int\n axis along which to unpad\n out : bool\n trim the output if True, otherwise the input; the two cases have\n their left and right pad sizes reversed\n \"\"\"\n\n assert a.shape[axis] == self.N\n\n Npad = self.N - self.Nin\n if out:\n _Npad, Npad_ = Npad - Npad//2, Npad//2\n else:\n _Npad, Npad_ = Npad//2, Npad - Npad//2\n\n return np.take(a, range(_Npad, self.N - Npad_), axis=axis)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nroughing sanity checks on the input function.", "response": "def check(self, F):\n \"\"\"Rough sanity checks on the input function.\n \"\"\"\n\n assert F.ndim == 1, \"checker only supports 1D\"\n\n f = self.xfac * F\n fabs = np.abs(f)\n\n iQ1, iQ3 = np.searchsorted(fabs.cumsum(), np.array([0.25, 0.75]) * fabs.sum())\n assert 0 != iQ1 != iQ3 != self.Nin, \"checker giving up\"\n fabs_l = fabs[:iQ1].mean()\n fabs_m = fabs[iQ1:iQ3].mean()\n fabs_r = fabs[iQ3:].mean()\n\n if fabs_l > fabs_m:\n warnings.warn(\"left wing seems heavy: {:.2g} vs {:.2g}, \"\n \"change tilt and mind convergence\".format(fabs_l, fabs_m), RuntimeWarning)\n if fabs_m < fabs_r:\n warnings.warn(\"right wing seems heavy: {:.2g} vs {:.2g}, \"\n \"change tilt and mind convergence\".format(fabs_m, fabs_r), RuntimeWarning)\n\n if fabs[0] > fabs[1]:\n warnings.warn(\"left tail may blow up: {:.2g} vs {:.2g}, \"\n \"change tilt or avoid extrapolation\".format(f[0], f[1]), RuntimeWarning)\n if fabs[-2] < fabs[-1]:\n warnings.warn(\"right tail may blow up: {:.2g} vs {:.2g}, \"\n \"change tilt or avoid extrapolation\".format(f[-2], f[-1]), RuntimeWarning)\n\n if f[0]*f[1] <= 0:\n warnings.warn(\"left tail looks wiggly: {:.2g} vs {:.2g}, \"\n \"avoid extrapolation\".format(f[0], f[1]), RuntimeWarning)\n if f[-2]*f[-1] <= 0:\n warnings.warn(\"right tail looks wiggly: {:.2g} vs {:.2g}, \"\n \"avoid extrapolation\".format(f[-2], f[-1]), RuntimeWarning)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fft(x, axis=-1, padding_samples=0):\n if padding_samples > 0:\n padded = np.concatenate(\n [x, np.zeros((len(x), padding_samples), dtype=x.dtype)],\n axis=axis)\n else:\n padded = x\n\n transformed = np.fft.rfft(padded, axis=axis, norm='ortho')\n\n sr = audio_sample_rate(int(Seconds(1) / x.dimensions[axis].frequency))\n scale = LinearScale.from_sample_rate(sr, transformed.shape[-1])\n new_dimensions = list(x.dimensions)\n new_dimensions[axis] = FrequencyDimension(scale)\n return ArrayWithUnits(transformed, new_dimensions)", "response": "Apply an FFT along the given axis and return a new array with the specified amount of zero - padding"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef morlet_filter_bank(\n samplerate,\n kernel_size,\n scale,\n scaling_factor,\n normalize=True):\n \"\"\"\n Create a :class:`~zounds.core.ArrayWithUnits` instance with a\n :class:`~zounds.timeseries.TimeDimension` and a\n :class:`~zounds.spectral.FrequencyDimension` representing a bank of morlet\n wavelets centered on the sub-bands of the scale.\n\n Args:\n samplerate (SampleRate): the samplerate of the input signal\n kernel_size (int): the length in samples of each filter\n scale (FrequencyScale): a scale whose center frequencies determine the\n fundamental frequency of each filer\n scaling_factor (int or list of int): Scaling factors for each band,\n which determine the time-frequency resolution tradeoff.\n The number(s) should fall between 0 and 1, with smaller numbers\n achieving better frequency resolution, and larget numbers better\n time resolution\n normalize (bool): When true, ensure that each filter in the bank\n has unit norm\n\n See Also:\n :class:`~zounds.spectral.FrequencyScale`\n :class:`~zounds.timeseries.SampleRate`\n \"\"\"\n basis_size = len(scale)\n basis = np.zeros((basis_size, kernel_size), dtype=np.complex128)\n\n try:\n if len(scaling_factor) != len(scale):\n raise ValueError('scaling factor must have same length as scale')\n except TypeError:\n scaling_factor = np.repeat(float(scaling_factor), len(scale))\n\n sr = int(samplerate)\n\n for i, band in enumerate(scale):\n scaling = scaling_factor[i]\n w = band.center_frequency / (scaling * 2 * sr / kernel_size)\n basis[i] = morlet(\n M=kernel_size,\n w=w,\n s=scaling)\n basis = basis.real\n\n if normalize:\n basis /= np.linalg.norm(basis, axis=-1, keepdims=True) + 1e-8\n\n basis = ArrayWithUnits(\n basis, [FrequencyDimension(scale), TimeDimension(*samplerate)])\n\n return basis", "response": "Create a Morlet filter bank of the log - likelihood wavelets of the specified frequency scale and the signal of the specified frequency scale."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nadjust the image bias.", "response": "def set_bias(self, bias):\n \"\"\"\n Adjusts the image bias.\n\n Bias determines where the color changes start. At low bias, low\n intensities (i.e., low pixel values) will have non-zero color\n differences, while at high bias only high pixel values will have\n non-zero differences\n\n Args:\n bias: float\n A number between 0 and 1. Note that upon initialization the\n colormap has a default bias of 0.5.\n\n Returns: void\n \"\"\"\n self.x_offset += (bias - self._bias)\n self._bias = bias\n\n self._build_cdict()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nadjust the image contrast.", "response": "def set_contrast(self, contrast):\n \"\"\"\n Adjusts the image contrast.\n\n Contrast refers to the rate of change of color with color level.\n At low contrast, color changes gradually over many intensity\n levels, while at high contrast it can change rapidly within a\n few levels\n\n Args:\n contrast: float\n A number between 0 and 1. Note that upon initialization the\n colormap has a default contrast value of 0.5.\n\n Returns: void\n \"\"\"\n self._contrast = contrast\n\n self.x_spread = 2 * (1.0 - contrast)\n self.y_spread = 2.0 - 2 * (1.0 - contrast)\n\n self._build_cdict()"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncopy file from source to destination.", "response": "def _copy(src, dst, src_is_storage, dst_is_storage):\n \"\"\"\n Copies file from source to destination\n\n Args:\n src (str or file-like object): Source file.\n dst (str or file-like object): Destination file.\n src_is_storage (bool): Source is storage.\n dst_is_storage (bool): Destination is storage.\n \"\"\"\n # If both storage: Tries to perform same storage direct copy\n if src_is_storage and dst_is_storage:\n system_src = get_instance(src)\n system_dst = get_instance(dst)\n\n # Same storage copy\n if system_src is system_dst:\n\n # Checks if same file\n if system_src.relpath(src) == system_dst.relpath(dst):\n raise same_file_error(\n \"'%s' and '%s' are the same file\" % (src, dst))\n\n # Tries to copy\n try:\n return system_dst.copy(src, dst)\n except (UnsupportedOperation, ObjectException):\n pass\n\n # Copy from compatible storage using \"copy_from_\" or\n # \"copy_to_\" method if any\n for caller, called, method in (\n (system_dst, system_src, 'copy_from_%s'),\n (system_src, system_dst, 'copy_to_%s')):\n if hasattr(caller, method % called.storage):\n try:\n return getattr(caller, method % called.storage)(\n src, dst, called)\n except (UnsupportedOperation, ObjectException):\n continue\n\n # At least one storage object: copies streams\n with cos_open(src, 'rb') as fsrc:\n with cos_open(dst, 'wb') as fdst:\n\n # Get stream buffer size\n for stream in (fsrc, fdst):\n try:\n buffer_size = getattr(stream, '_buffer_size')\n break\n except AttributeError:\n continue\n else:\n buffer_size = COPY_BUFSIZE\n\n # Read and write\n copyfileobj(fsrc, fdst, buffer_size)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef copy(src, dst):\n # Handles path-like objects and checks if storage\n src, src_is_storage = format_and_is_storage(src)\n dst, dst_is_storage = format_and_is_storage(dst)\n\n # Local files: Redirects to \"shutil.copy\"\n if not src_is_storage and not dst_is_storage:\n return shutil_copy(src, dst)\n\n with handle_os_exceptions():\n # Checks destination\n if not hasattr(dst, 'read'):\n try:\n # If destination is directory: defines an output file inside it\n if isdir(dst):\n dst = join(dst, basename(src))\n\n # Checks if destination dir exists\n elif not isdir(dirname(dst)):\n raise IOError(\"No such file or directory: '%s'\" % dst)\n\n except ObjectPermissionError:\n # Unable to check target directory due to missing read access,\n # but do not raise to allow to write if possible\n pass\n\n # Performs copy\n _copy(src, dst, src_is_storage, dst_is_storage)", "response": "Copies a source file to a destination file or directory."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef copyfile(src, dst, follow_symlinks=True):\n # Handles path-like objects and checks if storage\n src, src_is_storage = format_and_is_storage(src)\n dst, dst_is_storage = format_and_is_storage(dst)\n\n # Local files: Redirects to \"shutil.copyfile\"\n if not src_is_storage and not dst_is_storage:\n return shutil_copyfile(src, dst, follow_symlinks=follow_symlinks)\n\n with handle_os_exceptions():\n # Checks destination\n try:\n if not hasattr(dst, 'read') and not isdir(dirname(dst)):\n raise IOError(\"No such file or directory: '%s'\" % dst)\n\n except ObjectPermissionError:\n # Unable to check target directory due to missing read access, but\n # do not raise to allow to write if possible\n pass\n\n # Performs copy\n _copy(src, dst, src_is_storage, dst_is_storage)", "response": "Copies a source file to a destination file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _handle_client_error():\n try:\n yield\n\n except _ClientError as exception:\n error = exception.response['Error']\n if error['Code'] in _ERROR_CODES:\n raise _ERROR_CODES[error['Code']](error['Message'])\n raise", "response": "Handle boto exception and convert to class\n IO exceptions"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncopying object of the same storage.", "response": "def copy(self, src, dst, other_system=None):\n \"\"\"\n Copy object of the same storage.\n\n Args:\n src (str): Path or URL.\n dst (str): Path or URL.\n other_system (pycosio._core.io_system.SystemBase subclass): Unused.\n \"\"\"\n copy_source = self.get_client_kwargs(src)\n copy_destination = self.get_client_kwargs(dst)\n with _handle_client_error():\n self.client.copy_object(CopySource=copy_source, **copy_destination)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets base keyword arguments for a specific path.", "response": "def get_client_kwargs(self, path):\n \"\"\"\n Get base keyword arguments for client for a\n specific path.\n\n Args:\n path (str): Absolute path or URL.\n\n Returns:\n dict: client args\n \"\"\"\n bucket_name, key = self.split_locator(path)\n kwargs = dict(Bucket=bucket_name)\n if key:\n kwargs['Key'] = key\n return kwargs"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _get_session(self):\n if self._session is None:\n self._session = _boto3.session.Session(\n **self._storage_parameters.get('session', dict()))\n return self._session", "response": "Returns a boto3. session. Session object."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_client(self):\n client_kwargs = self._storage_parameters.get('client', dict())\n\n # Handles unsecure mode\n if self._unsecure:\n client_kwargs = client_kwargs.copy()\n client_kwargs['use_ssl'] = False\n\n return self._get_session().client(\"s3\", **client_kwargs)", "response": "Get the client object for the current object."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the URL roots for this storage.", "response": "def _get_roots(self):\n \"\"\"\n Return URL roots for this storage.\n\n Returns:\n tuple of str or re.Pattern: URL roots\n \"\"\"\n region = self._get_session().region_name or r'[\\w-]+'\n return (\n # S3 scheme\n # - s3:///\n 's3://',\n\n # Virtual-hosted\u2013style URL\n # - http://.s3.amazonaws.com/\n # - https://.s3.amazonaws.com/\n # - http://.s3-.amazonaws.com/\n # - https://.s3-.amazonaws.com/\n _re.compile(r'https?://[\\w.-]+\\.s3\\.amazonaws\\.com'),\n _re.compile(\n r'https?://[\\w.-]+\\.s3-%s\\.amazonaws\\.com' % region),\n\n # Path-hosted\u2013style URL\n # - http://s3.amazonaws.com//\n # - https://s3.amazonaws.com//\n # - http://s3-.amazonaws.com//\n # - https://s3-.amazonaws.com//\n _re.compile(r'https?://s3\\.amazonaws\\.com'),\n _re.compile(r'https?://s3-%s\\.amazonaws\\.com' % region),\n\n # Transfer acceleration URL\n # - http://.s3-accelerate.amazonaws.com\n # - https://.s3-accelerate.amazonaws.com\n # - http://.s3-accelerate.dualstack.amazonaws.com\n # - https://.s3-accelerate.dualstack.amazonaws.com\n _re.compile(\n r'https?://[\\w.-]+\\.s3-accelerate\\.amazonaws\\.com'),\n _re.compile(\n r'https?://[\\w.-]+\\.s3-accelerate\\.dualstack'\n r'\\.amazonaws\\.com'))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_time(header, keys, name):\n for key in keys:\n try:\n return _to_timestamp(header.pop(key))\n except KeyError:\n continue\n raise _UnsupportedOperation(name)", "response": "Get time from header\n "} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn object or bucket HTTP header.", "response": "def _head(self, client_kwargs):\n \"\"\"\n Returns object or bucket HTTP header.\n\n Args:\n client_kwargs (dict): Client arguments.\n\n Returns:\n dict: HTTP header.\n \"\"\"\n with _handle_client_error():\n # Object\n if 'Key' in client_kwargs:\n header = self.client.head_object(**client_kwargs)\n\n # Bucket\n else:\n header = self.client.head_bucket(**client_kwargs)\n\n # Clean up HTTP request information\n for key in ('AcceptRanges', 'ResponseMetadata'):\n header.pop(key, None)\n return header"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nmakes a directory. args: client_kwargs (dict): Client arguments.", "response": "def _make_dir(self, client_kwargs):\n \"\"\"\n Make a directory.\n\n args:\n client_kwargs (dict): Client arguments.\n \"\"\"\n with _handle_client_error():\n # Object\n if 'Key' in client_kwargs:\n return self.client.put_object(Body=b'', **client_kwargs)\n\n # Bucket\n return self.client.create_bucket(\n Bucket=client_kwargs['Bucket'],\n CreateBucketConfiguration=dict(\n LocationConstraint=self._get_session().region_name))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nremove an object from the cache.", "response": "def _remove(self, client_kwargs):\n \"\"\"\n Remove an object.\n\n args:\n client_kwargs (dict): Client arguments.\n \"\"\"\n with _handle_client_error():\n # Object\n if 'Key' in client_kwargs:\n return self.client.delete_object(**client_kwargs)\n\n # Bucket\n return self.client.delete_bucket(Bucket=client_kwargs['Bucket'])"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _list_locators(self):\n with _handle_client_error():\n response = self.client.list_buckets()\n\n for bucket in response['Buckets']:\n yield bucket.pop('Name'), bucket", "response": "Lists locators.\n\n Returns:\n generator of tuple: locator name str, locator header dict"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _list_objects(self, client_kwargs, path, max_request_entries):\n client_kwargs = client_kwargs.copy()\n if max_request_entries:\n client_kwargs['MaxKeys'] = max_request_entries\n\n while True:\n with _handle_client_error():\n response = self.client.list_objects_v2(\n Prefix=path, **client_kwargs)\n\n try:\n for obj in response['Contents']:\n yield obj.pop('Key'), obj\n except KeyError:\n raise _ObjectNotFoundError('Not found: %s' % path)\n\n # Handles results on more than one page\n try:\n client_kwargs['ContinuationToken'] = response[\n 'NextContinuationToken']\n except KeyError:\n # End of results\n break", "response": "A generator function that returns objects from the object store."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _read_range(self, start, end=0):\n # Get object part from S3\n try:\n with _handle_client_error():\n response = self._client.get_object(\n Range=self._http_range(start, end), **self._client_kwargs)\n\n # Check for end of file\n except _ClientError as exception:\n if exception.response['Error']['Code'] == 'InvalidRange':\n # EOF\n return bytes()\n raise\n\n # Get object content\n return response['Body'].read()", "response": "Read a range of bytes in stream."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nflush the write buffers of the given buffer.", "response": "def _flush(self, buffer):\n \"\"\"\n Flush the write buffers of the stream if applicable.\n\n Args:\n buffer (memoryview): Buffer content.\n \"\"\"\n with _handle_client_error():\n self._client.put_object(\n Body=buffer.tobytes(), **self._client_kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _flush(self):\n # Initialize multi-part upload\n if 'UploadId' not in self._upload_args:\n with _handle_client_error():\n self._upload_args[\n 'UploadId'] = self._client.create_multipart_upload(\n **self._client_kwargs)['UploadId']\n\n # Upload part with workers\n response = self._workers.submit(\n self._client.upload_part, Body=self._get_buffer().tobytes(),\n PartNumber=self._seek, **self._upload_args)\n\n # Save part information\n self._write_futures.append(\n dict(response=response, PartNumber=self._seek))", "response": "Flush the write buffers of the stream."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _close_writable(self):\n # Wait parts upload completion\n for part in self._write_futures:\n part['ETag'] = part.pop('response').result()['ETag']\n\n # Complete multipart upload\n with _handle_client_error():\n try:\n self._client.complete_multipart_upload(\n MultipartUpload={'Parts': self._write_futures},\n UploadId=self._upload_args['UploadId'],\n **self._client_kwargs)\n except _ClientError:\n # Clean up if failure\n self._client.abort_multipart_upload(\n UploadId=self._upload_args['UploadId'],\n **self._client_kwargs)\n raise", "response": "Close the object in write mode."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nhandling Azure exception and convert to class IO exceptions", "response": "def _handle_azure_exception():\n \"\"\"\n Handles Azure exception and convert to class IO exceptions\n\n Raises:\n OSError subclasses: IO error.\n \"\"\"\n try:\n yield\n\n except _AzureHttpError as exception:\n if exception.status_code in _ERROR_CODES:\n raise _ERROR_CODES[exception.status_code](str(exception))\n raise"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _properties_model_to_dict(properties):\n result = {}\n for attr in properties.__dict__:\n value = getattr(properties, attr)\n\n if hasattr(value, '__module__') and 'models' in value.__module__:\n value = _properties_model_to_dict(value)\n\n if not (value is None or (isinstance(value, dict) and not value)):\n result[attr] = value\n\n return result", "response": "Convert properties model to dict."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_endpoint(self, sub_domain):\n storage_parameters = self._storage_parameters or dict()\n account_name = storage_parameters.get('account_name')\n\n if not account_name:\n raise ValueError('\"account_name\" is required for Azure storage')\n\n suffix = storage_parameters.get(\n 'endpoint_suffix', 'core.windows.net')\n\n self._endpoint = 'http%s://%s.%s.%s' % (\n '' if self._unsecure else 's', account_name, sub_domain, suffix)\n\n return account_name, suffix.replace('.', r'\\.')", "response": "Get endpoint information from storage parameters and update self. _endpoint"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _secured_storage_parameters(self):\n parameters = self._storage_parameters or dict()\n\n # Handles unsecure mode\n if self._unsecure:\n parameters = parameters.copy()\n parameters['protocol'] = 'http'\n\n return parameters", "response": "Updates storage parameters with unsecure mode."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nformats path for cross Azure storage account copy function.", "response": "def _format_src_url(self, path, caller_system):\n \"\"\"\n Ensure path is absolute and use the correct URL format for use with\n cross Azure storage account copy function.\n\n Args:\n path (str): Path or URL.\n caller_system (pycosio.storage.azure._AzureBaseSystem subclass):\n System calling this method (Can be another Azure system).\n\n Returns:\n str: URL.\n \"\"\"\n path = '%s/%s' % (self._endpoint, self.relpath(path))\n\n # If SAS token available, use it to give cross account copy access.\n if caller_system is not self:\n try:\n path = '%s?%s' % (path, self._storage_parameters['sas_token'])\n except KeyError:\n pass\n\n return path"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _update_listing_client_kwargs(client_kwargs, max_request_entries):\n client_kwargs = client_kwargs.copy()\n if max_request_entries:\n client_kwargs['num_results'] = max_request_entries\n return client_kwargs", "response": "Updates the client kwargs for listing functions."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts object model to dict.", "response": "def _model_to_dict(obj):\n \"\"\"\n Convert object model to dict.\n\n Args:\n obj: Object model.\n\n Returns:\n dict: Converted model.\n \"\"\"\n result = _properties_model_to_dict(obj.properties)\n for attribute in ('metadata', 'snapshot'):\n try:\n value = getattr(obj, attribute)\n except AttributeError:\n continue\n if value:\n result[attribute] = value\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _read_range(self, start, end=0):\n stream = _BytesIO()\n try:\n with _handle_azure_exception():\n self._get_to_stream(\n stream=stream, start_range=start,\n end_range=(end - 1) if end else None, **self._client_kwargs)\n\n # Check for end of file\n except _AzureHttpError as exception:\n if exception.status_code == 416:\n # EOF\n return bytes()\n raise\n\n return stream.getvalue()", "response": "Reads a range of bytes in stream."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreading and return all the bytes from the stream until EOF.", "response": "def _readall(self):\n \"\"\"\n Read and return all the bytes from the stream until EOF.\n\n Returns:\n bytes: Object content\n \"\"\"\n stream = _BytesIO()\n with _handle_azure_exception():\n self._get_to_stream(stream=stream, **self._client_kwargs)\n return stream.getvalue()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _init_append(self):\n if self._content_length:\n # Adjust size if content length specified\n with _handle_azure_exception():\n self._resize(\n content_length=self._content_length, **self._client_kwargs)\n self._reset_head()\n\n # Make initial seek position to current end of file\n self._seek = self._size", "response": "Initializes file on a mode."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates the file in Azure.", "response": "def _create(self):\n \"\"\"\n Create the file if not exists.\n \"\"\"\n # Create new file\n with _handle_azure_exception():\n self._create_from_size(\n content_length=self._content_length, **self._client_kwargs)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nflush the buffer of the stream if applicable.", "response": "def _flush(self, buffer, start, end):\n \"\"\"\n Flush the write buffer of the stream if applicable.\n\n Args:\n buffer (memoryview): Buffer content.\n start (int): Start of buffer position to flush.\n Supported only with page blobs.\n end (int): End of buffer position to flush.\n Supported only with page blobs.\n \"\"\"\n buffer_size = len(buffer)\n if not buffer_size:\n return\n\n # Write range normally\n with self._size_lock:\n if end > self._size:\n # Require to resize the blob if note enough space\n with _handle_azure_exception():\n self._resize(content_length=end, **self._client_kwargs)\n self._reset_head()\n\n if buffer_size > self.MAX_FLUSH_SIZE:\n # Too large buffer, needs to split in multiples requests\n futures = []\n for part_start in range(0, buffer_size, self.MAX_FLUSH_SIZE):\n\n # Split buffer\n buffer_part = buffer[\n part_start:part_start + self.MAX_FLUSH_SIZE]\n if not len(buffer_part):\n # No more data\n break\n\n # Upload split buffer in parallel\n start_range = start + part_start\n futures.append(self._workers.submit(\n self._update_range, data=buffer_part.tobytes(),\n start_range=start_range,\n end_range=start_range + len(buffer_part) - 1,\n **self._client_kwargs))\n\n with _handle_azure_exception():\n # Wait for upload completion\n for future in _as_completed(futures):\n future.result()\n\n else:\n # Buffer lower than limit, do one requests.\n with _handle_azure_exception():\n self._update_range(\n data=buffer.tobytes(), start_range=start,\n end_range=end - 1, **self._client_kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _generate_async(self, generator):\n first_value_future = self._workers.submit(next, generator)\n\n def get_first_element(future=first_value_future):\n \"\"\"\n Get first element value from future.\n\n Args:\n future (concurrent.futures._base.Future): First value future.\n\n Returns:\n Evaluated value\n \"\"\"\n try:\n yield future.result()\n except StopIteration:\n return\n\n return chain(get_first_element(), generator)", "response": "A generator function that returns the previous generator object after having run the first element evaluated\n in background task."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nhandles Swift exception and convert to class IO exceptions", "response": "def _handle_client_exception():\n \"\"\"\n Handle Swift exception and convert to class\n IO exceptions\n\n Raises:\n OSError subclasses: IO error.\n \"\"\"\n try:\n yield\n\n except _ClientException as exception:\n if exception.http_status in _ERROR_CODES:\n raise _ERROR_CODES[exception.http_status](\n exception.http_reason)\n raise"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef copy(self, src, dst, other_system=None):\n container, obj = self.split_locator(src)\n with _handle_client_exception():\n self.client.copy_object(\n container=container, obj=obj, destination=self.relpath(dst))", "response": "Copy object of the same storage."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_client_kwargs(self, path):\n container, obj = self.split_locator(path)\n kwargs = dict(container=container)\n if obj:\n kwargs['obj'] = obj\n return kwargs", "response": "Get base keyword arguments for a specific path."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a new instance of the Swift client.", "response": "def _get_client(self):\n \"\"\"\n Swift client\n\n Returns:\n swiftclient.client.Connection: client\n \"\"\"\n kwargs = self._storage_parameters\n\n # Handles unsecure mode\n if self._unsecure:\n kwargs = kwargs.copy()\n kwargs['ssl_compression'] = False\n\n return _swift.client.Connection(**kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _head(self, client_kwargs):\n with _handle_client_exception():\n # Object\n if 'obj' in client_kwargs:\n return self.client.head_object(**client_kwargs)\n\n # Container\n return self.client.head_container(**client_kwargs)", "response": "Returns object HTTP header."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _make_dir(self, client_kwargs):\n with _handle_client_exception():\n # Object\n if 'obj' in client_kwargs:\n return self.client.put_object(\n client_kwargs['container'], client_kwargs['obj'], b'')\n\n # Container\n return self.client.put_container(client_kwargs['container'])", "response": "Make a directory.\n\n args:\n client_kwargs (dict): Client arguments."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _remove(self, client_kwargs):\n with _handle_client_exception():\n # Object\n if 'obj' in client_kwargs:\n return self.client.delete_object(\n client_kwargs['container'], client_kwargs['obj'])\n\n # Container\n return self.client.delete_container(client_kwargs['container'])", "response": "Remove an object from the database."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _list_locators(self):\n with _handle_client_exception():\n response = self.client.get_account()\n\n for container in response[1]:\n yield container.pop('name'), container", "response": "Lists locators.\n\n Returns:\n generator of tuple: locator name str, locator header dict"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _list_objects(self, client_kwargs, path, max_request_entries):\n kwargs = dict(prefix=path)\n if max_request_entries:\n kwargs['limit'] = max_request_entries\n else:\n kwargs['full_listing'] = True\n\n with _handle_client_exception():\n response = self.client.get_container(\n client_kwargs['container'], **kwargs)\n\n for obj in response[1]:\n yield obj.pop('name'), obj", "response": "List objects in the current locator."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nread a range of bytes in stream.", "response": "def _read_range(self, start, end=0):\n \"\"\"\n Read a range of bytes in stream.\n\n Args:\n start (int): Start stream position.\n end (int): End stream position.\n 0 To not specify end.\n\n Returns:\n bytes: number of bytes read\n \"\"\"\n try:\n with _handle_client_exception():\n return self._client.get_object(*self._client_args, headers=dict(\n Range=self._http_range(start, end)))[1]\n\n except _ClientException as exception:\n if exception.http_status == 416:\n # EOF\n return b''\n raise"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nflushing the write buffers of the stream if applicable.", "response": "def _flush(self, buffer):\n \"\"\"\n Flush the write buffers of the stream if applicable.\n\n Args:\n buffer (memoryview): Buffer content.\n \"\"\"\n container, obj = self._client_args\n with _handle_client_exception():\n self._client.put_object(container, obj, buffer)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _flush(self):\n # Upload segment with workers\n name = self._segment_name % self._seek\n response = self._workers.submit(\n self._client.put_object, self._container, name,\n self._get_buffer())\n\n # Save segment information in manifest\n self._write_futures.append(dict(\n etag=response, path='/'.join((self._container, name))))", "response": "Flush the write buffers of the stream."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nclosing the object in write mode.", "response": "def _close_writable(self):\n \"\"\"\n Close the object in write mode.\n \"\"\"\n # Wait segments upload completion\n for segment in self._write_futures:\n segment['etag'] = segment['etag'].result()\n\n # Upload manifest file\n with _handle_client_exception():\n self._client.put_object(self._container, self._object_name, _dumps(\n self._write_futures), query_string='multipart-manifest=put')"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ninitialize file on a mode.", "response": "def _init_append(self):\n \"\"\"\n Initializes file on 'a' mode.\n \"\"\"\n # Require to load the full file content in buffer\n self._write_buffer[:] = self._readall()\n\n # Make initial seek position to current end of file\n self._seek = self._size"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nflushing the write buffers of the object if applicable and close the object.", "response": "def close(self):\n \"\"\"\n Flush the write buffers of the stream if applicable and\n close the object.\n \"\"\"\n if self._writable and not self._is_raw_of_buffered and not self._closed:\n self._closed = True\n if self._write_buffer:\n self.flush()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreads the next size bytes from the stream without advancing the position.", "response": "def _peek(self, size=-1):\n \"\"\"\n Return bytes from the stream without advancing the position.\n\n Args:\n size (int): Number of bytes to read. -1 to read the full\n stream.\n\n Returns:\n bytes: bytes read\n \"\"\"\n with self._seek_lock:\n seek = self._seek\n with handle_os_exceptions():\n return self._read_range(seek, seek + size)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef readall(self):\n if not self._readable:\n raise UnsupportedOperation('read')\n\n with self._seek_lock:\n # Get data starting from seek\n with handle_os_exceptions():\n if self._seek and self._seekable:\n data = self._read_range(self._seek)\n\n # Get all data\n else:\n data = self._readall()\n\n # Update seek\n self._seek += len(data)\n return data", "response": "Read and return all the bytes from the stream until EOF."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef readinto(self, b):\n if not self._readable:\n raise UnsupportedOperation('read')\n\n # Get and update stream positions\n size = len(b)\n with self._seek_lock:\n start = self._seek\n end = start + size\n self._seek = end\n\n # Read data range\n with handle_os_exceptions():\n read_data = self._read_range(start, end)\n\n # Copy to bytes-like object\n read_size = len(read_data)\n if read_size:\n memoryview(b)[:read_size] = read_data\n\n # Update stream position if end of file\n if read_size != size:\n with self._seek_lock:\n self._seek = start + read_size\n\n # Return read size\n return read_size", "response": "Reads bytes into a pre - allocated writable bytes - like object b and returns the number of bytes read."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef seek(self, offset, whence=SEEK_SET):\n if not self._seekable:\n raise UnsupportedOperation('seek')\n\n seek = self._update_seek(offset, whence)\n\n # If seek move out of file, add padding until new seek position.\n if self._writable:\n size = len(self._write_buffer)\n if seek > size:\n self._write_buffer[seek:size] = b'\\0' * (seek - size)\n\n return seek", "response": "Change the stream position to the given byte offset."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _update_seek(self, offset, whence):\n with self._seek_lock:\n if whence == SEEK_SET:\n self._seek = offset\n elif whence == SEEK_CUR:\n self._seek += offset\n elif whence == SEEK_END:\n self._seek = offset + self._size\n else:\n raise ValueError('whence value %s unsupported' % whence)\n return self._seek", "response": "Update seek value.\n\n Args:\n offset (int): Offset.\n whence (int): Whence.\n\n Returns:\n int: Seek position."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef write(self, b):\n if not self._writable:\n raise UnsupportedOperation('write')\n\n # This function write data in a buffer\n # \"flush()\" need to be called to really write content on\n # Cloud Storage\n size = len(b)\n with self._seek_lock:\n start = self._seek\n end = start + size\n self._seek = end\n\n buffer = self._write_buffer\n if end <= len(buffer):\n buffer = memoryview(buffer)\n buffer[start:end] = b\n return size", "response": "Writes the given bytes - like object b to the underlying raw stream and returns the number of bytes written."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncheck for HTTP errors and raise OSError if relevant.", "response": "def _handle_http_errors(response):\n \"\"\"\n Check for HTTP errors and raise\n OSError if relevant.\n\n Args:\n response (requests.Response):\n\n Returns:\n requests.Response: response\n \"\"\"\n code = response.status_code\n if 200 <= code < 400:\n return response\n elif code in (403, 404):\n raise {403: _ObjectPermissionError,\n 404: _ObjectNotFoundError}[code](response.reason)\n response.raise_for_status()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _head(self, client_kwargs):\n return _handle_http_errors(\n self.client.request(\n 'HEAD', timeout=self._TIMEOUT, **client_kwargs)).headers", "response": "Returns object HTTP header."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreading a range of bytes in stream.", "response": "def _read_range(self, start, end=0):\n \"\"\"\n Read a range of bytes in stream.\n\n Args:\n start (int): Start stream position.\n end (int): End stream position.\n 0 To not specify end.\n\n Returns:\n bytes: number of bytes read\n \"\"\"\n # Get object part\n response = self._client.request(\n 'GET', self.name, headers=dict(Range=self._http_range(start, end)),\n timeout=self._TIMEOUT)\n\n if response.status_code == 416:\n # EOF\n return b''\n\n # Get object content\n return _handle_http_errors(response).content"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _readall(self):\n return _handle_http_errors(\n self._client.request(\n 'GET', self.name, timeout=self._TIMEOUT)).content", "response": "Read and return all the bytes from the stream until EOF."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nflush the write buffers of the stream if applicable and the object on the cloud is saved.", "response": "def flush(self):\n \"\"\"\n Flush the write buffers of the stream if applicable and\n save the object on the cloud.\n \"\"\"\n if self._writable:\n with self._seek_lock:\n buffer = self._get_buffer()\n\n # Flush that part of the file\n end = self._seek\n start = end - len(buffer)\n\n # Clear buffer\n self._write_buffer = bytearray()\n\n # Flush content\n with handle_os_exceptions():\n self._flush(buffer, start, end)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef seek(self, offset, whence=SEEK_SET):\n if not self._seekable:\n raise UnsupportedOperation('seek')\n\n # Flush before moving position\n self.flush()\n\n return self._update_seek(offset, whence)", "response": "Change the stream position to the given byte offset."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nflushes the buffer of the stream if applicable.", "response": "def _flush(self):\n \"\"\"\n Flush the write buffers of the stream if applicable.\n\n In write mode, send the buffer content to the cloud object.\n \"\"\"\n # Flush buffer to specified range\n buffer = self._get_buffer()\n start = self._buffer_size * (self._seek - 1)\n end = start + len(buffer)\n\n future = self._workers.submit(\n self._flush_range, buffer=buffer, start=start, end=end)\n self._write_futures.append(future)\n future.add_done_callback(partial(self._update_size, end))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _update_size(self, size, future):\n with self._size_lock:\n # Update value\n if size > self._size and future.done:\n # Size can be lower if seek down on an 'a' mode open file.\n self._size = size", "response": "Update the size of the current entry in the cache."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nflush a buffer to a range of the file.", "response": "def _flush_range(self, buffer, start, end):\n \"\"\"\n Flush a buffer to a range of the file.\n\n Meant to be used asynchronously, used to provides parallel flushing of\n file parts when applicable.\n\n Args:\n buffer (memoryview): Buffer content.\n start (int): Start of buffer position to flush.\n end (int): End of buffer position to flush.\n \"\"\"\n # On first call, Get file size if exists\n with self._size_lock:\n if not self._size_synched:\n self._size_synched = True\n try:\n self._size = self.raw._size\n except (ObjectNotFoundError, UnsupportedOperation):\n self._size = 0\n\n # It is not possible to flush a part if start > size:\n # If it is the case, wait that previous parts are flushed before\n # flushing this one\n while start > self._size:\n sleep(self._FLUSH_WAIT)\n\n # Flush buffer using RAW IO\n self._raw_flush(buffer, start, end)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nexecute a shell command.", "response": "def run(cmd):\n \"\"\" Execute a shell command.\n\n Both envoy and python-sh failed me...\n commands, although deprecated, feels like the easiest tool to use.\n \"\"\"\n\n status, output = commands.getstatusoutput(cmd)\n if status:\n print(output)\n return status == 0"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nread in all datanommer entries from a file created with datanommer - dump > myfile. json", "response": "def read_datanommer_entries_from_filedump():\n \"\"\" Read in all datanommer entries from a file created with:\n\n $ datanommer-dump > myfile.json\n \"\"\"\n\n # TODO -- un-hardcode this filename when I need to run this next time.\n #filename = \"../myfile.json\"\n filename = \"../datanommer-dump-2012-11-22.json\"\n print(\"Reading %s\" % filename)\n progress = progressbar.ProgressBar(widgets=[\n progressbar.widgets.Percentage(),\n progressbar.widgets.Bar(),\n progressbar.widgets.ETA(),\n ])\n\n def _entries():\n failed = 0\n with open(filename, 'r') as f:\n raw = f.read()\n lines = raw.split('\\n}\\n')\n for line in progress(lines):\n try:\n yield json.loads(line + \"\\n}\")\n except:\n failed += 1\n print(\" * Failed to parse %i json objects\" % failed)\n\n def comp(a, b):\n return cmp(a['timestamp'], b['timestamp'])\n\n result = sorted(list(_entries()), cmp=comp)\n print(\" * Read and sorted %i messages\" % len(result))\n print()\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates or update a model instance.", "response": "def create_or_update(self,\n identifier: Identifier,\n data: Dict[str, Any],\n ) -> Tuple[object, bool]:\n \"\"\"\n Create or update a model.\n\n :param identifier: An object with :attr:`class_name` and :attr:`key`\n attributes\n :param data: A dictionary keyed by column name, with values being the\n converted values to set on the model instance\n :return: A two-tuple of model instance and whether or not it was created.\n \"\"\"\n raise NotImplementedError"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a dictionary of raw values for a specific identifier and data.", "response": "def maybe_convert_values(self,\n identifier: Identifier,\n data: Dict[str, Any],\n ) -> Dict[str, Any]:\n \"\"\"\n Takes a dictionary of raw values for a specific identifier, as parsed\n from the YAML file, and depending upon the type of db column the data\n is meant for, decides what to do with the value (eg leave it alone,\n convert a string to a date/time instance, or convert identifiers to\n model instances by calling :meth:`self.loader.convert_identifiers`)\n\n :param identifier: An object with :attr:`class_name` and :attr:`key`\n attributes\n :param data: A dictionary keyed by column name, with values being the\n raw values as parsed from the YAML\n :return: A dictionary keyed by column name, with values being the\n converted values meant to be set on the model instance\n \"\"\"\n raise NotImplementedError"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncopy a file from src to dst.", "response": "def copy(self, src, dst, other_system=None):\n \"\"\"\n Copy object of the same storage.\n\n Args:\n src (str): Path or URL.\n dst (str): Path or URL.\n other_system (pycosio.storage.azure._AzureBaseSystem subclass):\n The source storage system.\n \"\"\"\n with _handle_azure_exception():\n self.client.copy_file(\n copy_source=(other_system or self)._format_src_url(src, self),\n **self.get_client_kwargs(dst))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget base keyword arguments for a specific path.", "response": "def get_client_kwargs(self, path):\n \"\"\"\n Get base keyword arguments for client for a\n specific path.\n\n Args:\n path (str): Absolute path or URL.\n\n Returns:\n dict: client args\n \"\"\"\n # Remove query string from URL\n path = path.split('?', 1)[0]\n\n share_name, relpath = self.split_locator(path)\n kwargs = dict(share_name=share_name)\n\n # Directory\n if relpath and relpath[-1] == '/':\n kwargs['directory_name'] = relpath.rstrip('/')\n\n # File\n elif relpath:\n try:\n kwargs['directory_name'], kwargs['file_name'] = relpath.rsplit(\n '/', 1)\n except ValueError:\n kwargs['directory_name'] = ''\n kwargs['file_name'] = relpath\n\n # Else, Share only\n return kwargs"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns object or bucket HTTP header.", "response": "def _head(self, client_kwargs):\n \"\"\"\n Returns object or bucket HTTP header.\n\n Args:\n client_kwargs (dict): Client arguments.\n\n Returns:\n dict: HTTP header.\n \"\"\"\n with _handle_azure_exception():\n # File\n if 'file_name' in client_kwargs:\n result = self.client.get_file_properties(**client_kwargs)\n\n # Directory\n elif 'directory_name' in client_kwargs:\n result = self.client.get_directory_properties(**client_kwargs)\n\n # Share\n else:\n result = self.client.get_share_properties(**client_kwargs)\n\n return self._model_to_dict(result)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nlist locators. Returns: generator of tuple: locator name str, locator header dict", "response": "def _list_locators(self):\n \"\"\"\n Lists locators.\n\n Returns:\n generator of tuple: locator name str, locator header dict\n \"\"\"\n with _handle_azure_exception():\n for share in self.client.list_shares():\n yield share.name, self._model_to_dict(share)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _list_objects(self, client_kwargs, max_request_entries):\n client_kwargs = self._update_listing_client_kwargs(\n client_kwargs, max_request_entries)\n\n with _handle_azure_exception():\n for obj in self.client.list_directories_and_files(**client_kwargs):\n yield (obj.name, self._model_to_dict(obj),\n isinstance(obj, _Directory))", "response": "A generator that yields objects from the Cloud Storage API."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _make_dir(self, client_kwargs):\n with _handle_azure_exception():\n # Directory\n if 'directory_name' in client_kwargs:\n return self.client.create_directory(\n share_name=client_kwargs['share_name'],\n directory_name=client_kwargs['directory_name'])\n\n # Share\n return self.client.create_share(**client_kwargs)", "response": "Make a directory and a share."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _remove(self, client_kwargs):\n with _handle_azure_exception():\n # File\n if 'file_name' in client_kwargs:\n return self.client.delete_file(\n share_name=client_kwargs['share_name'],\n directory_name=client_kwargs['directory_name'],\n file_name=client_kwargs['file_name'])\n\n # Directory\n elif 'directory_name' in client_kwargs:\n return self.client.delete_directory(\n share_name=client_kwargs['share_name'],\n directory_name=client_kwargs['directory_name'])\n\n # Share\n return self.client.delete_share(\n share_name=client_kwargs['share_name'])", "response": "Remove an object from the Cloud Storage."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _update_range(self, data, **kwargs):\n self._client.update_range(data=data, **kwargs)", "response": "Update the range of the data."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef handle_bodhi(msg):\n\n if 'bodhi.update.comment' in msg.topic:\n username = msg.msg['comment']['author']\n elif 'bodhi.buildroot_override' in msg.topic:\n username = msg.msg['override']['submitter']\n else:\n username = msg.msg.get('update', {}).get('submitter')\n return username", "response": "Given a bodhi message return the FAS username."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef handle_wiki(msg):\n\n if 'wiki.article.edit' in msg.topic:\n username = msg.msg['user']\n elif 'wiki.upload.complete' in msg.topic:\n username = msg.msg['user_text']\n else:\n raise ValueError(\"Unhandled topic.\")\n\n return username", "response": "Given a wiki message return the FAS username."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncheck if file is a local file or a storage file.", "response": "def is_storage(url, storage=None):\n \"\"\"\n Check if file is a local file or a storage file.\n\n File is considered local if:\n - URL is a local path.\n - URL starts by \"file://\"\n - a \"storage\" is provided.\n\n Args:\n url (str): file path or URL\n storage (str): Storage name.\n\n Returns:\n bool: return True if file is local.\n \"\"\"\n if storage:\n return True\n split_url = url.split('://', 1)\n if len(split_url) == 2 and split_url[0].lower() != 'file':\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nchecks if path is storage and format it.", "response": "def format_and_is_storage(path):\n \"\"\"\n Checks if path is storage and format it.\n\n If path is an opened file-like object, returns is storage as True.\n\n Args:\n path (path-like object or file-like object):\n\n Returns:\n tuple: str or file-like object (Updated path),\n bool (True if is storage).\n \"\"\"\n if not hasattr(path, 'read'):\n path = fsdecode(path).replace('\\\\', '/')\n return path, is_storage(path)\n return path, True"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndecorate a cloud object compatible function to provides fall back to standard function if used on local files. Args: std_function (function): standard function to used with local files. Returns: function: new function", "response": "def equivalent_to(std_function):\n \"\"\"\n Decorates a cloud object compatible function\n to provides fall back to standard function if\n used on local files.\n\n Args:\n std_function (function): standard function to\n used with local files.\n\n Returns:\n function: new function\n \"\"\"\n\n def decorate(cos_function):\n \"\"\"Decorator argument handler\"\"\"\n\n @wraps(cos_function)\n def decorated(path, *args, **kwargs):\n \"\"\"Decorated function\"\"\"\n\n # Handles path-like objects\n path = fsdecode(path).replace('\\\\', '/')\n\n # Storage object: Handle with Cloud object storage\n # function\n if is_storage(path):\n with handle_os_exceptions():\n return cos_function(path, *args, **kwargs)\n\n # Local file: Redirect to standard function\n return std_function(path, *args, **kwargs)\n\n return decorated\n\n return decorate"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _handle_oss_error():\n try:\n yield\n\n except _OssError as exception:\n if exception.status in _ERROR_CODES:\n raise _ERROR_CODES[exception.status](\n exception.details.get('Message', ''))\n raise", "response": "Handle OSS exceptions and convert to class IO exceptions"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef copy(self, src, dst, other_system=None):\n copy_source = self.get_client_kwargs(src)\n copy_destination = self.get_client_kwargs(dst)\n with _handle_oss_error():\n bucket = self._get_bucket(copy_destination)\n bucket.copy_object(\n source_bucket_name=copy_source['bucket_name'],\n source_key=copy_source['key'],\n target_key=copy_destination['key'])", "response": "Copy object of the same storage."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_client(self):\n return (_oss.StsAuth if 'security_token' in self._storage_parameters\n else _oss.Auth if self._storage_parameters\n else _oss.AnonymousAuth)(**self._storage_parameters)", "response": "Returns the client object for the current session."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the URL roots for this storage.", "response": "def _get_roots(self):\n \"\"\"\n Return URL roots for this storage.\n\n Returns:\n tuple of str or re.Pattern: URL roots\n \"\"\"\n\n return (\n # OSS Scheme\n # - oss:///\n 'oss://',\n\n # URL (With common aliyuncs.com endpoint):\n # - http://.oss-.aliyuncs.com/\n # - https://.oss-.aliyuncs.com/\n\n # Note: \"oss-.aliyuncs.com\" may be replaced by another\n # endpoint\n\n _re.compile((r'https?://[\\w-]+.%s' % self._endpoint.split(\n '//', 1)[1]).replace('.', r'\\.')))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _get_bucket(self, client_kwargs):\n return _oss.Bucket(self.client, endpoint=self._endpoint,\n bucket_name=client_kwargs['bucket_name'])", "response": "Get the object in the appropriate bucket."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns True if object is a symbolic link.", "response": "def islink(self, path=None, header=None):\n \"\"\"\n Returns True if object is a symbolic link.\n\n Args:\n path (str): File path or URL.\n header (dict): Object header.\n\n Returns:\n bool: True if object is Symlink.\n \"\"\"\n if header is None:\n header = self._head(self.get_client_kwargs(path))\n\n for key in ('x-oss-object-type', 'type'):\n try:\n return header.pop(key) == 'Symlink'\n except KeyError:\n continue\n return False"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn object HTTP header.", "response": "def _head(self, client_kwargs):\n \"\"\"\n Returns object HTTP header.\n\n Args:\n client_kwargs (dict): Client arguments.\n\n Returns:\n dict: HTTP header.\n \"\"\"\n with _handle_oss_error():\n bucket = self._get_bucket(client_kwargs)\n\n # Object\n if 'key' in client_kwargs:\n return bucket.head_object(\n key=client_kwargs['key']).headers\n\n # Bucket\n return bucket.get_bucket_info().headers"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nmaking a directory. args: client_kwargs (dict): Client arguments.", "response": "def _make_dir(self, client_kwargs):\n \"\"\"\n Make a directory.\n\n args:\n client_kwargs (dict): Client arguments.\n \"\"\"\n with _handle_oss_error():\n bucket = self._get_bucket(client_kwargs)\n\n # Object\n if 'key' in client_kwargs:\n return bucket.put_object(\n key=client_kwargs['key'], data=b'')\n\n # Bucket\n return bucket.create_bucket()"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nremoves an object from the cache.", "response": "def _remove(self, client_kwargs):\n \"\"\"\n Remove an object.\n\n args:\n client_kwargs (dict): Client arguments.\n \"\"\"\n with _handle_oss_error():\n bucket = self._get_bucket(client_kwargs)\n\n # Object\n if 'key' in client_kwargs:\n return bucket.delete_object(key=client_kwargs['key'])\n\n # Bucket\n return bucket.delete_bucket()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert OSS model to dict.", "response": "def _model_to_dict(model, ignore):\n \"\"\"\n Convert OSS model to dict.\n\n Args:\n model (oss2.models.RequestResult): Model.\n ignore (tuple of str): Keys to not insert to dict.\n\n Returns:\n dict: Model dict version.\n \"\"\"\n return {attr: value for attr, value in model.__dict__.items()\n if not attr.startswith('_') and attr not in ignore}"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _list_locators(self):\n with _handle_oss_error():\n response = _oss.Service(\n self.client, endpoint=self._endpoint).list_buckets()\n\n for bucket in response.buckets:\n yield bucket.name, self._model_to_dict(bucket, ('name',))", "response": "Lists locators.\n\n Returns:\n generator of tuple: locator name str, locator header dict"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _list_objects(self, client_kwargs, path, max_request_entries):\n kwargs = dict()\n if max_request_entries:\n kwargs['max_keys'] = max_request_entries\n\n bucket = self._get_bucket(client_kwargs)\n\n while True:\n with _handle_oss_error():\n response = bucket.list_objects(prefix=path, **kwargs)\n\n if not response.object_list:\n # In case of empty dir, return empty dir path:\n # if empty result, the dir do not exists.\n raise _ObjectNotFoundError('Not found: %s' % path)\n\n for obj in response.object_list:\n yield obj.key, self._model_to_dict(obj, ('key',))\n\n # Handles results on more than one page\n if response.next_marker:\n client_kwargs['marker'] = response.next_marker\n else:\n # End of results\n break", "response": "A generator function that returns objects from the specified path."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _read_range(self, start, end=0):\n if start >= self._size:\n # EOF. Do not detect using 416 (Out of range) error, 200 returned.\n return bytes()\n\n # Get object bytes range\n with _handle_oss_error():\n response = self._bucket.get_object(key=self._key, headers=dict(\n Range=self._http_range(\n # Returns full file if end > size\n start, end if end <= self._size else self._size)))\n\n # Get object content\n return response.read()", "response": "Read a range of bytes in stream."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _readall(self):\n with _handle_oss_error():\n return self._bucket.get_object(key=self._key).read()", "response": "Read and return all the bytes from the stream until EOF."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nflushing the write buffers of the stream if applicable.", "response": "def _flush(self, buffer):\n \"\"\"\n Flush the write buffers of the stream if applicable.\n\n Args:\n buffer (memoryview): Buffer content.\n \"\"\"\n with _handle_oss_error():\n self._bucket.put_object(key=self._key, data=buffer.tobytes())"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nflushing the write buffers of the stream.", "response": "def _flush(self):\n \"\"\"\n Flush the write buffers of the stream.\n \"\"\"\n # Initialize multipart upload\n if self._upload_id is None:\n with _handle_oss_error():\n self._upload_id = self._bucket.init_multipart_upload(\n self._key).upload_id\n\n # Upload part with workers\n response = self._workers.submit(\n self._bucket.upload_part, key=self._key, upload_id=self._upload_id,\n part_number=self._seek, data=self._get_buffer().tobytes())\n\n # Save part information\n self._write_futures.append(\n dict(response=response, part_number=self._seek))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _close_writable(self):\n # Wait parts upload completion\n parts = [_PartInfo(part_number=future['part_number'],\n etag=future['response'].result().etag)\n for future in self._write_futures]\n\n # Complete multipart upload\n with _handle_oss_error():\n try:\n self._bucket.complete_multipart_upload(\n key=self._key, upload_id=self._upload_id, parts=parts)\n except _OssError:\n # Clean up failed upload\n self._bucket.abort_multipart_upload(\n key=self._key, upload_id=self._upload_id)\n raise", "response": "Close the object in write mode."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfunctioning that opens a file and returns a corresponding file - like object.", "response": "def cos_open(file, mode='r', buffering=-1, encoding=None, errors=None,\n newline=None, storage=None, storage_parameters=None, unsecure=None,\n **kwargs):\n \"\"\"\n Open file and return a corresponding file object.\n\n Equivalent to \"io.open\" or builtin \"open\".\n\n File can also be binary opened file-like object.\n\n Args:\n file (path-like object or file-like object): File path, object URL or\n opened file-like object.\n mode (str): mode in which the file is opened (default to 'rb').\n see \"io.open\" for all possible modes. Note that all modes may\n not be supported by all kind of file and storage.\n buffering (int): Set the buffering policy.\n -1 to use default behavior,\n 0 to switch buffering off,\n 1 to select line buffering (only usable in text mode),\n and an integer > 1 to indicate the size in bytes of a\n fixed-size chunk buffer.\n See \"io.open\" for more information.\n encoding (str): The name of the encoding used to\n decode or encode the file. This should only be used in text mode.\n See \"io.open\" for more information.\n errors (str): Specifies how encoding and decoding errors\n are to be handled.\n This should only be used in text mode.\n See \"io.open\" for more information.\n newline (str): Controls how universal newlines mode works.\n This should only be used in text mode.\n See \"io.open\" for more information.\n storage (str): Storage name.\n storage_parameters (dict): Storage configuration parameters.\n Generally, client configuration and credentials.\n unsecure (bool): If True, disables TLS/SSL to improves\n transfer performance. But makes connection unsecure.\n Default to False.\n kwargs: Other arguments to pass to opened object.\n Note that theses arguments may not be compatible with\n all kind of file and storage.\n\n Returns:\n file-like object: opened file.\n\n Raises:\n OSError: If the file cannot be opened.\n FileExistsError: File open in 'x' mode already exists.\n \"\"\"\n # Handles file-like objects:\n if hasattr(file, 'read'):\n with _text_io_wrapper(file, mode, encoding, errors, newline) as wrapped:\n yield wrapped\n return\n\n # Handles path-like objects\n file = fsdecode(file).replace('\\\\', '/')\n\n # Storage object\n if is_storage(file, storage):\n with get_instance(\n name=file, cls='raw' if buffering == 0 else 'buffered',\n storage=storage, storage_parameters=storage_parameters,\n mode=mode, unsecure=unsecure, **kwargs) as stream:\n with _text_io_wrapper(stream, mode=mode, encoding=encoding,\n errors=errors, newline=newline) as wrapped:\n yield wrapped\n\n # Local file: Redirect to \"io.open\"\n else:\n with io_open(file, mode, buffering, encoding, errors, newline,\n **kwargs) as stream:\n yield stream"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nwrap a binary stream to a Text stream.", "response": "def _text_io_wrapper(stream, mode, encoding, errors, newline):\n \"\"\"Wrap a binary stream to Text stream.\n\n Args:\n stream (file-like object): binary stream.\n mode (str): Open mode.\n encoding (str): Stream encoding.\n errors (str): Decoding error handling.\n newline (str): Universal newlines\n \"\"\"\n # Text mode, if not already a text stream\n # That has the \"encoding\" attribute\n if \"t\" in mode and not hasattr(stream, 'encoding'):\n text_stream = TextIOWrapper(\n stream, encoding=encoding, errors=errors, newline=newline)\n yield text_stream\n text_stream.flush()\n\n # Binary mode (Or already text stream)\n else:\n yield stream"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef random_model(ctx, model_class_name):\n model_identifiers = ctx['model_identifiers'][model_class_name]\n if not model_identifiers:\n return 'None'\n idx = random.randrange(0, len(model_identifiers))\n return '\"%s(%s)\"' % (model_class_name, model_identifiers[idx])", "response": "Returns a random model identifier by class name."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a random model identifier by class name.", "response": "def random_models(ctx, model_class_name, min_count=0, max_count=3):\n \"\"\"\n Get a random model identifier by class name. Example usage::\n\n # db/fixtures/Tag.yml\n {% for i in range(0, 10) %}\n tag{{ i }}:\n name: {{ faker.name() }}\n {% endfor %}\n\n # db/fixtures/Post.yml\n a_blog_post:\n tags: {{ random_models('Tag') }}\n\n Will render to something like the following::\n\n # db/fixtures/Post.yml (rendered)\n a blog_post:\n tags: [\"Tag(tag2, tag5)\"]\n\n :param ctx: The context variables of the current template (passed automatically)\n :param model_class_name: The class name of the models to get.\n :param min_count: The minimum number of models to return.\n :param max_count: The maximum number of models to return.\n \"\"\"\n model_identifiers = ctx['model_identifiers'][model_class_name]\n num_models = random.randint(min_count, min(max_count, len(model_identifiers)))\n if num_models == 0:\n return '[]'\n\n added = set()\n while len(added) < num_models:\n idx = random.randrange(0, len(model_identifiers))\n added.add(model_identifiers[idx])\n return '[\"%s(%s)\"]' % (model_class_name, ','.join(added))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nlist objects. Args: path (str): Path or URL. relative (bool): Path is relative to current root. first_level (bool): It True, returns only first level objects. Else, returns full tree. max_request_entries (int): If specified, maximum entries returned by request. Returns: generator of tuple: object name str, object header dict", "response": "def list_objects(self, path='', relative=False, first_level=False,\n max_request_entries=None):\n \"\"\"\n List objects.\n\n Args:\n path (str): Path or URL.\n relative (bool): Path is relative to current root.\n first_level (bool): It True, returns only first level objects.\n Else, returns full tree.\n max_request_entries (int): If specified, maximum entries returned\n by request.\n\n Returns:\n generator of tuple: object name str, object header dict\n \"\"\"\n entries = 0\n next_values = []\n max_request_entries_arg = None\n\n if not relative:\n path = self.relpath(path)\n\n # From root\n if not path:\n objects = self._list_locators()\n\n # Sub directory\n else:\n objects = self._list_objects(\n self.get_client_kwargs(path), max_request_entries)\n\n # Yield file hierarchy\n for obj in objects:\n # Generate first level objects entries\n try:\n name, header, is_directory = obj\n except ValueError:\n # Locators\n name, header = obj\n is_directory = True\n\n # Start to generate subdirectories content\n if is_directory and not first_level:\n name = next_path = name.rstrip('/') + '/'\n\n if path:\n next_path = '/'.join((path.rstrip('/'), name))\n\n if max_request_entries is not None:\n max_request_entries_arg = max_request_entries - entries\n\n next_values.append((\n name, self._generate_async(self.list_objects(\n next_path, relative=True,\n max_request_entries=max_request_entries_arg))))\n\n entries += 1\n yield name, header\n if entries == max_request_entries:\n return\n\n for next_name, generator in next_values:\n # Generate other levels objects entries\n for name, header in generator:\n\n entries += 1\n yield '/'.join((next_name.rstrip('/'), name)), header\n if entries == max_request_entries:\n return"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nhashes all keys in a timerange to the same value.", "response": "def hash_key(self, key):\n \"\"\" \"Hash\" all keys in a timerange to the same value. \"\"\"\n for i, destination_key in enumerate(self._dict):\n if key < destination_key:\n return destination_key\n\n return key"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_all(self, progress_callback: Optional[callable] = None) -> Dict[str, object]:\n if not self._loaded:\n self._load_data()\n\n # build up a directed acyclic graph to determine the model instantiation order\n dag = nx.DiGraph()\n for model_class_name, dependencies in self.relationships.items():\n dag.add_node(model_class_name)\n for dep in dependencies:\n dag.add_edge(model_class_name, dep)\n\n try:\n creation_order = reversed(list(nx.topological_sort(dag)))\n except nx.NetworkXUnfeasible:\n raise Exception('Circular dependency detected between models: '\n ', '.join(['{a} -> {b}'.format(a=a, b=b)\n for a, b in nx.find_cycle(dag)]))\n\n # create or update the models in the determined order\n rv = {}\n for model_class_name in creation_order:\n for identifier_key, data in self.model_fixtures[model_class_name].items():\n identifier = Identifier(model_class_name, identifier_key)\n data = self.factory.maybe_convert_values(identifier, data)\n self._cache[identifier_key] = data\n\n model_instance, created = self.factory.create_or_update(identifier, data)\n if progress_callback:\n progress_callback(identifier, model_instance, created)\n rv[identifier_key] = model_instance\n\n self.factory.commit()\n return rv", "response": "Creates all the models in the fixture files in the fixtures_dir."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert an individual : class : Identifier to a list of model instances or a list of model instances.", "response": "def convert_identifiers(self, identifiers: Union[Identifier, List[Identifier]]):\n \"\"\"\n Convert an individual :class:`Identifier` to a model instance,\n or a list of Identifiers to a list of model instances.\n \"\"\"\n if not identifiers:\n return identifiers\n\n def _create_or_update(identifier):\n data = self._cache[identifier.key]\n return self.factory.create_or_update(identifier, data)[0]\n\n if isinstance(identifiers, Identifier):\n return _create_or_update(identifiers)\n elif isinstance(identifiers, list) and isinstance(identifiers[0], Identifier):\n return [_create_or_update(identifier) for identifier in identifiers]\n else:\n raise TypeError('`identifiers` must be an Identifier or list of Identifiers.')"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nload all fixtures from given directories and store them in the self. _cache.", "response": "def _load_data(self):\n \"\"\"\n Load all fixtures from :attr:`fixtures_dir`\n \"\"\"\n filenames = []\n model_identifiers = defaultdict(list)\n\n # attempt to load fixture files from given directories (first pass)\n # for each valid model fixture file, read it into the cache and get the\n # list of identifier keys from it\n for fixtures_dir in self.fixture_dirs:\n for filename in os.listdir(fixtures_dir):\n path = os.path.join(fixtures_dir, filename)\n file_ext = filename[filename.find('.')+1:]\n\n # make sure it's a valid fixture file\n if os.path.isfile(path) and file_ext in {'yml', 'yaml'}:\n filenames.append(filename)\n with open(path) as f:\n self._cache[filename] = f.read()\n\n # preload to determine identifier keys\n with self._preloading_env() as env:\n rendered_yaml = env.get_template(filename).render()\n data = yaml.load(rendered_yaml)\n if data:\n class_name = filename[:filename.rfind('.')]\n model_identifiers[class_name] = list(data.keys())\n\n # second pass where we can render the jinja templates with knowledge of all\n # the model identifier keys (allows random_model and random_models to work)\n for filename in filenames:\n self._load_from_yaml(filename, model_identifiers)\n\n self._loaded = True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _load_from_yaml(self, filename: str, model_identifiers: Dict[str, List[str]]):\n class_name = filename[:filename.rfind('.')]\n rendered_yaml = self.env.get_template(filename).render(\n model_identifiers=model_identifiers)\n fixture_data, self.relationships[class_name] = self._post_process_yaml_data(\n yaml.load(rendered_yaml),\n self.factory.get_relationships(class_name))\n\n for identifier_key, data in fixture_data.items():\n self.model_fixtures[class_name][identifier_key] = data", "response": "Load fixtures from the given yaml file."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting and normalize identifier strings to Identifiers and determine the class relationships.", "response": "def _post_process_yaml_data(self,\n fixture_data: Dict[str, Dict[str, Any]],\n relationship_columns: Set[str],\n ) -> Tuple[Dict[str, Dict[str, Any]], List[str]]:\n \"\"\"\n Convert and normalize identifier strings to Identifiers, as well as determine\n class relationships.\n \"\"\"\n rv = {}\n relationships = set()\n if not fixture_data:\n return rv, relationships\n\n for identifier_id, data in fixture_data.items():\n new_data = {}\n for col_name, value in data.items():\n if col_name not in relationship_columns:\n new_data[col_name] = value\n continue\n\n identifiers = normalize_identifiers(value)\n if identifiers:\n relationships.add(identifiers[0].class_name)\n\n if isinstance(value, str) and len(identifiers) <= 1:\n new_data[col_name] = identifiers[0] if identifiers else None\n else:\n new_data[col_name] = identifiers\n\n rv[identifier_id] = new_data\n return rv, list(relationships)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmakes sure the jinja environment is minimally configured.", "response": "def _ensure_env(self, env: Union[jinja2.Environment, None]):\n \"\"\"\n Make sure the jinja environment is minimally configured.\n \"\"\"\n if not env:\n env = jinja2.Environment()\n if not env.loader:\n env.loader = jinja2.FunctionLoader(lambda filename: self._cache[filename])\n if 'faker' not in env.globals:\n faker = Faker()\n faker.seed(1234)\n env.globals['faker'] = faker\n if 'random_model' not in env.globals:\n env.globals['random_model'] = jinja2.contextfunction(random_model)\n if 'random_models' not in env.globals:\n env.globals['random_models'] = jinja2.contextfunction(random_models)\n return env"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _preloading_env(self):\n ctx = self.env.globals\n try:\n ctx['random_model'] = lambda *a, **kw: None\n ctx['random_models'] = lambda *a, **kw: None\n yield self.env\n finally:\n ctx['random_model'] = jinja2.contextfunction(random_model)\n ctx['random_models'] = jinja2.contextfunction(random_models)", "response": "A context manager that preloads the environment variables for the object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_instance(name, cls='system', storage=None, storage_parameters=None,\n unsecure=None, *args, **kwargs):\n \"\"\"\n Get a cloud object storage instance.\n\n Args:\n name (str): File name, path or URL.\n cls (str): Type of class to instantiate.\n 'raw', 'buffered' or 'system'.\n storage (str): Storage name.\n storage_parameters (dict): Storage configuration parameters.\n Generally, client configuration and credentials.\n unsecure (bool): If True, disables TLS/SSL to improves\n transfer performance. But makes connection unsecure.\n Default to False.\n args, kwargs: Instance arguments\n\n Returns:\n pycosio._core.io_base.ObjectIOBase subclass: Instance\n \"\"\"\n system_parameters = _system_parameters(\n unsecure=unsecure, storage_parameters=storage_parameters)\n\n # Gets storage information\n with _MOUNT_LOCK:\n for root in MOUNTED:\n if ((isinstance(root, Pattern) and root.match(name)) or\n (not isinstance(root, Pattern) and\n name.startswith(root))):\n info = MOUNTED[root]\n\n # Get stored storage parameters\n stored_parameters = info.get('system_parameters') or dict()\n if not system_parameters:\n same_parameters = True\n system_parameters = stored_parameters\n elif system_parameters == stored_parameters:\n same_parameters = True\n else:\n same_parameters = False\n # Copy not specified parameters from default\n system_parameters.update({\n key: value for key, value in stored_parameters.items()\n if key not in system_parameters})\n break\n\n # If not found, tries to mount before getting\n else:\n mount_info = mount(\n storage=storage, name=name, **system_parameters)\n info = mount_info[tuple(mount_info)[0]]\n same_parameters = True\n\n # Returns system class\n if cls == 'system':\n if same_parameters:\n return info['system_cached']\n else:\n return info['system'](\n roots=info['roots'], **system_parameters)\n\n # Returns other classes\n if same_parameters:\n if 'storage_parameters' not in system_parameters:\n system_parameters['storage_parameters'] = dict()\n system_parameters['storage_parameters'][\n 'pycosio.system_cached'] = info['system_cached']\n\n kwargs.update(system_parameters)\n return info[cls](name=name, *args, **kwargs)", "response": "Get a cloud object storage instance."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nmounting a new storage at a new location.", "response": "def mount(storage=None, name='', storage_parameters=None,\n unsecure=None, extra_root=None):\n \"\"\"\n Mount a new storage.\n\n Args:\n storage (str): Storage name.\n name (str): File URL. If storage is not specified,\n URL scheme will be used as storage value.\n storage_parameters (dict): Storage configuration parameters.\n Generally, client configuration and credentials.\n unsecure (bool): If True, disables TLS/SSL to improves\n transfer performance. But makes connection unsecure.\n Default to False.\n extra_root (str): Extra root that can be used in\n replacement of root in path. This can be used to\n provides support for shorter URLS.\n Example: with root \"https://www.mycloud.com/user\"\n and extra_root \"mycloud://\" it is possible to access object\n using \"mycloud://container/object\" instead of\n \"https://www.mycloud.com/user/container/object\".\n\n Returns:\n dict: keys are mounted storage, values are dicts of storage information.\n \"\"\"\n # Tries to infer storage from name\n if storage is None:\n if '://' in name:\n storage = name.split('://', 1)[0].lower()\n # Alias HTTPS to HTTP\n storage = 'http' if storage == 'https' else storage\n else:\n raise ValueError(\n 'No storage specified and unable to infer it from file name.')\n\n # Saves get_storage_parameters\n system_parameters = _system_parameters(\n unsecure=unsecure, storage_parameters=storage_parameters)\n storage_info = dict(storage=storage, system_parameters=system_parameters)\n\n # Finds module containing target subclass\n for package in STORAGE_PACKAGE:\n try:\n module = import_module('%s.%s' % (package, storage))\n break\n except ImportError:\n continue\n else:\n raise ImportError('No storage named \"%s\" found' % storage)\n\n # Case module is a mount redirection to mount multiple storage at once\n if hasattr(module, 'MOUNT_REDIRECT'):\n if extra_root:\n raise ValueError(\n (\"Can't define extra_root with %s. \"\n \"%s can't have a common root.\") % (\n storage, ', '.join(extra_root)))\n result = dict()\n for storage in getattr(module, 'MOUNT_REDIRECT'):\n result[storage] = mount(\n storage=storage, storage_parameters=storage_parameters,\n unsecure=unsecure)\n return result\n\n # Finds storage subclass\n classes_items = tuple(_BASE_CLASSES.items())\n for member_name in dir(module):\n member = getattr(module, member_name)\n for cls_name, cls in classes_items:\n\n # Skip if not subclass of the target class\n try:\n if not issubclass(member, cls) or member is cls:\n continue\n except TypeError:\n continue\n\n # The class may have been flag as default or not-default\n default_flag = '_%s__DEFAULT_CLASS' % member.__name__.strip('_')\n try:\n is_default = getattr(member, default_flag)\n except AttributeError:\n is_default = None\n\n # Skip if explicitly flagged as non default\n if is_default is False:\n continue\n\n # Skip if is an abstract class not explicitly flagged as default\n elif is_default is not True and member.__abstractmethods__:\n continue\n\n # Is the default class\n storage_info[cls_name] = member\n break\n\n # Caches a system instance\n storage_info['system_cached'] = storage_info['system'](**system_parameters)\n\n # Gets roots\n roots = storage_info['system_cached'].roots\n\n # Adds extra root\n if extra_root:\n roots = list(roots)\n roots.append(extra_root)\n roots = tuple(roots)\n storage_info['system_cached'].roots = storage_info['roots'] = roots\n\n # Mounts\n with _MOUNT_LOCK:\n for root in roots:\n MOUNTED[root] = storage_info\n\n # Reorder to have correct lookup\n items = OrderedDict(\n (key, MOUNTED[key]) for key in reversed(\n sorted(MOUNTED, key=_compare_root)))\n MOUNTED.clear()\n MOUNTED.update(items)\n\n return {storage: storage_info}"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn system keyword arguments removing Nones.", "response": "def _system_parameters(**kwargs):\n \"\"\"\n Returns system keyword arguments removing Nones.\n\n Args:\n kwargs: system keyword arguments.\n\n Returns:\n dict: system keyword arguments.\n \"\"\"\n return {key: value for key, value in kwargs.items()\n if (value is not None or value == {})}"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef manual_argument_parsing(argv):\n\n # Special case these for a better error message\n if not argv or argv == ['-h'] or argv == ['--help']:\n print_help_and_exit()\n\n try:\n dashdash_index = argv.index('--')\n except ValueError:\n print_std_err('Must separate command by `--`')\n print_help_and_exit()\n\n patches, cmd = argv[:dashdash_index], argv[dashdash_index + 1:]\n\n if '--help' in patches or '-h' in patches:\n print_help_and_exit()\n\n if '--all' in patches:\n all_patches = True\n patches.remove('--all')\n else:\n all_patches = False\n\n unknown_options = [patch for patch in patches if patch.startswith('-')]\n if unknown_options:\n print_std_err('Unknown options: {!r}'.format(unknown_options))\n print_help_and_exit()\n\n if patches and all_patches:\n print_std_err('--all and patches specified: {!r}'.format(patches))\n print_help_and_exit()\n\n return Arguments(all=all_patches, patches=tuple(patches), cmd=tuple(cmd))", "response": "sadness because argparse doesn t quite do what we want."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate a console_script entry point for your application.", "response": "def make_entry_point(patches, original_entry_point):\n \"\"\"Use this to make a console_script entry point for your application\n which applies patches.\n :param patches: iterable of pymonkey patches to apply. Ex: ('my-patch,)\n :param original_entry_point: Such as 'pip'\n \"\"\"\n def entry(argv=None):\n argv = argv if argv is not None else sys.argv[1:]\n return main(\n tuple(patches) + ('--', original_entry_point) + tuple(argv)\n )\n\n return entry"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the client object for this storage instance.", "response": "def client(self):\n \"\"\"\n Storage client\n\n Returns:\n client\n \"\"\"\n if self._client is None:\n self._client = self._get_client()\n return self._client"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef exists(self, path=None, client_kwargs=None, assume_exists=None):\n try:\n self.head(path, client_kwargs)\n except ObjectNotFoundError:\n return False\n except ObjectPermissionError:\n if assume_exists is None:\n raise\n return assume_exists\n return True", "response": "Check if a file exists in the object store."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef getctime(self, path=None, client_kwargs=None, header=None):\n return self._getctime_from_header(\n self.head(path, client_kwargs, header))", "response": "Get the creation time of path."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the time of last access of path.", "response": "def getmtime(self, path=None, client_kwargs=None, header=None):\n \"\"\"\n Return the time of last access of path.\n\n Args:\n path (str): File path or URL.\n client_kwargs (dict): Client arguments.\n header (dict): Object header.\n\n Returns:\n float: The number of seconds since the epoch\n (see the time module).\n \"\"\"\n return self._getmtime_from_header(\n self.head(path, client_kwargs, header))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget time from header", "response": "def _get_time(header, keys, name):\n \"\"\"\n Get time from header\n\n Args:\n header (dict): Object header.\n keys (tuple of str): Header keys.\n name (str): Method name.\n\n Returns:\n float: The number of seconds since the epoch\n \"\"\"\n for key in keys:\n try:\n date_value = header.pop(key)\n except KeyError:\n continue\n try:\n # String to convert\n return to_timestamp(parse(date_value))\n except TypeError:\n # Already number\n return float(date_value)\n raise UnsupportedOperation(name)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef getsize(self, path=None, client_kwargs=None, header=None):\n return self._getsize_from_header(self.head(path, client_kwargs, header))", "response": "Returns the size in bytes of the object at path."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _getsize_from_header(self, header):\n # By default, assumes that information are in a standard HTTP header\n for key in self._SIZE_KEYS:\n try:\n return int(header.pop(key))\n except KeyError:\n continue\n else:\n raise UnsupportedOperation('getsize')", "response": "Return the size in bytes from the header."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchecking if a path is an existing directory.", "response": "def isdir(self, path=None, client_kwargs=None, virtual_dir=True,\n assume_exists=None):\n \"\"\"\n Return True if path is an existing directory.\n\n Args:\n path (str): Path or URL.\n client_kwargs (dict): Client arguments.\n virtual_dir (bool): If True, checks if directory exists virtually\n if an object path if not exists as a specific object.\n assume_exists (bool or None): This value define the value to return\n in the case there is no enough permission to determinate the\n existing status of the file. If set to None, the permission\n exception is reraised (Default behavior). if set to True or\n False, return this value.\n\n Returns:\n bool: True if directory exists.\n \"\"\"\n relative = self.relpath(path)\n if not relative:\n # Root always exists and is a directory\n return True\n\n if path[-1] == '/' or self.is_locator(relative, relative=True):\n exists = self.exists(path=path, client_kwargs=client_kwargs,\n assume_exists=assume_exists)\n if exists:\n return True\n\n # Some directories only exists virtually in object path and don't\n # have headers.\n elif virtual_dir:\n try:\n next(self.list_objects(relative, relative=True,\n max_request_entries=1))\n return True\n except (StopIteration, ObjectNotFoundError,\n UnsupportedOperation):\n return False\n return False"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncheck if a file exists.", "response": "def isfile(self, path=None, client_kwargs=None, assume_exists=None):\n \"\"\"\n Return True if path is an existing regular file.\n\n Args:\n path (str): Path or URL.\n client_kwargs (dict): Client arguments.\n assume_exists (bool or None): This value define the value to return\n in the case there is no enough permission to determinate the\n existing status of the file. If set to None, the permission\n exception is reraised (Default behavior). if set to True or\n False, return this value.\n\n Returns:\n bool: True if file exists.\n \"\"\"\n relative = self.relpath(path)\n if not relative:\n # Root always exists and is a directory\n return False\n\n if path[-1] != '/' and not self.is_locator(path, relative=True):\n return self.exists(path=path, client_kwargs=client_kwargs,\n assume_exists=assume_exists)\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning object HTTP header.", "response": "def head(self, path=None, client_kwargs=None, header=None):\n \"\"\"\n Returns object HTTP header.\n\n Args:\n path (str): Path or URL.\n client_kwargs (dict): Client arguments.\n header (dict): Object header.\n\n Returns:\n dict: HTTP header.\n \"\"\"\n if header is not None:\n return header\n elif client_kwargs is None:\n client_kwargs = self.get_client_kwargs(path)\n return self._head(client_kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget path relative to storage.", "response": "def relpath(self, path):\n \"\"\"\n Get path relative to storage.\n\n args:\n path (str): Absolute path or URL.\n\n Returns:\n str: relative path.\n \"\"\"\n for root in self.roots:\n # Root is regex, convert to matching root string\n if isinstance(root, Pattern):\n match = root.match(path)\n if not match:\n continue\n root = match.group(0)\n\n # Split root and relative path\n try:\n relative = path.split(root, 1)[1]\n # Strip \"/\" only at path start. \"/\" is used to known if\n # path is a directory on some cloud storage.\n return relative.lstrip('/')\n except IndexError:\n continue\n\n return path"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning True if path refer to a locator.", "response": "def is_locator(self, path, relative=False):\n \"\"\"\n Returns True if path refer to a locator.\n\n Depending the storage, locator may be a bucket or container name,\n a hostname, ...\n\n args:\n path (str): path or URL.\n relative (bool): Path is relative to current root.\n\n Returns:\n bool: True if locator.\n \"\"\"\n if not relative:\n path = self.relpath(path)\n # Bucket is the main directory\n return path and '/' not in path.rstrip('/')"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsplits the path into a pair of locator and tail.", "response": "def split_locator(self, path):\n \"\"\"\n Split the path into a pair (locator, path).\n\n args:\n path (str): Absolute path or URL.\n\n Returns:\n tuple of str: locator, path.\n \"\"\"\n relative = self.relpath(path)\n try:\n locator, tail = relative.split('/', 1)\n except ValueError:\n locator = relative\n tail = ''\n return locator, tail"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef make_dir(self, path, relative=False):\n if not relative:\n path = self.relpath(path)\n self._make_dir(self.get_client_kwargs(self.ensure_dir_path(\n path, relative=True)))", "response": "Make a directory.\n\n Args:\n path (str): Path or URL.\n relative (bool): Path is relative to current root."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef remove(self, path, relative=False):\n if not relative:\n path = self.relpath(path)\n self._remove(self.get_client_kwargs(path))", "response": "Remove an object from the cache."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nensure the path is a dir path.", "response": "def ensure_dir_path(self, path, relative=False):\n \"\"\"\n Ensure the path is a dir path.\n\n Should end with '/' except for schemes and locators.\n\n Args:\n path (str): Path or URL.\n relative (bool): Path is relative to current root.\n\n Returns:\n path: dir path\n \"\"\"\n if not relative:\n rel_path = self.relpath(path)\n else:\n rel_path = path\n\n # Locator\n if self.is_locator(rel_path, relative=True):\n path = path.rstrip('/')\n\n # Directory\n elif rel_path:\n path = path.rstrip('/') + '/'\n # else: root\n return path"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef list_objects(self, path='', relative=False, first_level=False,\n max_request_entries=None):\n \"\"\"\n List objects.\n\n Args:\n path (str): Path or URL.\n relative (bool): Path is relative to current root.\n first_level (bool): It True, returns only first level objects.\n Else, returns full tree.\n max_request_entries (int): If specified, maximum entries returned\n by request.\n\n Returns:\n generator of tuple: object name str, object header dict\n \"\"\"\n entries = 0\n max_request_entries_arg = None\n\n if not relative:\n path = self.relpath(path)\n\n # From root\n if not path:\n locators = self._list_locators()\n\n # Yields locators\n if first_level:\n for locator in locators:\n\n entries += 1\n yield locator\n if entries == max_request_entries:\n return\n return\n\n # Yields each locator objects\n for loc_path, loc_header in locators:\n\n # Yields locator itself\n loc_path = loc_path.strip('/')\n\n entries += 1\n yield loc_path, loc_header\n if entries == max_request_entries:\n return\n\n # Yields locator content is read access to it\n if max_request_entries is not None:\n max_request_entries_arg = max_request_entries - entries\n try:\n for obj_path, obj_header in self._list_objects(\n self.get_client_kwargs(loc_path), '',\n max_request_entries_arg):\n\n entries += 1\n yield ('/'.join((loc_path, obj_path.lstrip('/'))),\n obj_header)\n if entries == max_request_entries:\n return\n\n except ObjectPermissionError:\n # No read access to locator\n continue\n return\n\n # From locator or sub directory\n locator, path = self.split_locator(path)\n\n if first_level:\n seen = set()\n\n if max_request_entries is not None:\n max_request_entries_arg = max_request_entries - entries\n\n for obj_path, header in self._list_objects(\n self.get_client_kwargs(locator), path, max_request_entries_arg):\n\n if path:\n try:\n obj_path = obj_path.split(path, 1)[1]\n except IndexError:\n # Not sub path of path\n continue\n obj_path = obj_path.lstrip('/')\n\n # Skips parent directory\n if not obj_path:\n continue\n\n # Yields first level locator objects only\n if first_level:\n # Directory\n try:\n obj_path, _ = obj_path.strip('/').split('/', 1)\n obj_path += '/'\n\n # Avoids to use the header of the object instead of the\n # non existing header of the directory that only exists\n # virtually in object path.\n header = dict()\n\n # File\n except ValueError:\n pass\n\n if obj_path not in seen:\n entries += 1\n yield obj_path, header\n if entries == max_request_entries:\n return\n seen.add(obj_path)\n\n # Yields locator objects\n else:\n entries += 1\n yield obj_path, header\n if entries == max_request_entries:\n return", "response": "A generator function that returns all objects in the specified path."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef stat(self, path=None, client_kwargs=None, header=None):\n # Should contain at least the strict minimum of os.stat_result\n stat = OrderedDict((\n (\"st_mode\", 0), (\"st_ino\", 0), (\"st_dev\", 0), (\"st_nlink\", 0),\n (\"st_uid\", 0), (\"st_gid\", 0), (\"st_size\", 0), (\"st_atime\", 0),\n (\"st_mtime\", 0), (\"st_ctime\", 0)))\n\n # Populate standard os.stat_result values with object header content\n header = self.head(path, client_kwargs, header)\n for key, method in (\n ('st_size', self._getsize_from_header),\n ('st_ctime', self._getctime_from_header),\n ('st_mtime', self._getmtime_from_header),):\n try:\n stat[key] = int(method(header))\n except UnsupportedOperation:\n continue\n\n # File mode\n if self.islink(path=path, header=header):\n # Symlink\n stat['st_mode'] = S_IFLNK\n elif ((not path or path[-1] == '/' or self.is_locator(path)) and not\n stat['st_size']):\n # Directory\n stat['st_mode'] = S_IFDIR\n else:\n # File\n stat['st_mode'] = S_IFREG\n\n # Add storage specific keys\n sub = self._CHAR_FILTER.sub\n for key, value in tuple(header.items()):\n stat['st_' + sub('', key.lower())] = value\n\n # Convert to \"os.stat_result\" like object\n stat_result = namedtuple('stat_result', tuple(stat))\n stat_result.__name__ = 'os.stat_result'\n stat_result.__module__ = 'pycosio'\n return stat_result(**stat)", "response": "Returns os. stat_result object."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ninitializing a connection. Create tables if requested.", "response": "def init(uri=None, alembic_ini=None, engine=None, create=False):\n \"\"\" Initialize a connection. Create tables if requested.\"\"\"\n\n if uri and engine:\n raise ValueError(\"uri and engine cannot both be specified\")\n\n if uri is None and not engine:\n uri = 'sqlite:////tmp/datanommer.db'\n log.warning(\"No db uri given. Using %r\" % uri)\n\n if uri and not engine:\n engine = create_engine(uri)\n\n if 'sqlite' in engine.driver:\n # Enable nested transaction support under SQLite\n # See https://stackoverflow.com/questions/1654857/nested-transactions-with-sqlalchemy-and-sqlite\n @event.listens_for(engine, \"connect\")\n def do_connect(dbapi_connection, connection_record):\n # disable pysqlite's emitting of the BEGIN statement entirely.\n # also stops it from emitting COMMIT before any DDL.\n dbapi_connection.isolation_level = None\n\n @event.listens_for(engine, \"begin\")\n def do_begin(conn):\n # emit our own BEGIN\n conn.execute(\"BEGIN\")\n\n # We need to hang our own attribute on the sqlalchemy session to stop\n # ourselves from initializing twice. That is only a problem is the code\n # calling us isn't consistent.\n if getattr(session, '_datanommer_initialized', None):\n log.warning(\"Session already initialized. Bailing\")\n return\n session._datanommer_initialized = True\n\n session.configure(bind=engine)\n DeclarativeBase.query = session.query_property()\n\n # Loads the alembic configuration and generates the version table, with\n # the most recent revision stamped as head\n if alembic_ini is not None:\n from alembic.config import Config\n from alembic import command\n alembic_cfg = Config(alembic_ini)\n command.stamp(alembic_cfg, \"head\")\n\n if create:\n DeclarativeBase.metadata.create_all(engine)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ntakes a dict - like fedmsg envelope and store the headers and message in the table.", "response": "def add(envelope):\n \"\"\" Take a dict-like fedmsg envelope and store the headers and message\n in the table.\n \"\"\"\n message = envelope['body']\n timestamp = message.get('timestamp', None)\n try:\n if timestamp:\n timestamp = datetime.datetime.utcfromtimestamp(timestamp)\n else:\n timestamp = datetime.datetime.utcnow()\n except Exception:\n pass\n\n headers = envelope.get('headers', None)\n msg_id = message.get('msg_id', None)\n if not msg_id and headers:\n msg_id = headers.get('message-id', None)\n if not msg_id:\n msg_id = six.text_type(timestamp.year) + six.u('-') + six.text_type(uuid.uuid4())\n obj = Message(\n i=message.get('i', 0),\n msg_id=msg_id,\n topic=message['topic'],\n timestamp=timestamp,\n username=message.get('username', None),\n crypto=message.get('crypto', None),\n certificate=message.get('certificate', None),\n signature=message.get('signature', None),\n )\n\n obj.msg = message['msg']\n obj.headers = headers\n\n try:\n session.add(obj)\n session.flush()\n except IntegrityError:\n log.warning('Skipping message from %s with duplicate id: %s',\n message['topic'], msg_id)\n session.rollback()\n return\n\n usernames = fedmsg.meta.msg2usernames(message)\n packages = fedmsg.meta.msg2packages(message)\n\n # Do a little sanity checking on fedmsg.meta results\n if None in usernames:\n # Notify developers so they can fix msg2usernames\n log.error('NoneType found in usernames of %r' % msg_id)\n # And prune out the bad value\n usernames = [name for name in usernames if name is not None]\n\n if None in packages:\n # Notify developers so they can fix msg2packages\n log.error('NoneType found in packages of %r' % msg_id)\n # And prune out the bad value\n packages = [pkg for pkg in packages if pkg is not None]\n\n # If we've never seen one of these users before, then:\n # 1) make sure they exist in the db (create them if necessary)\n # 2) mark an in memory cache so we can remember that they exist without\n # having to hit the db.\n for username in usernames:\n if username not in _users_seen:\n # Create the user in the DB if necessary\n User.get_or_create(username)\n # Then just mark an in memory cache noting that we've seen them.\n _users_seen.add(username)\n\n for package in packages:\n if package not in _packages_seen:\n Package.get_or_create(package)\n _packages_seen.add(package)\n\n session.flush()\n\n # These two blocks would normally be a simple \"obj.users.append(user)\" kind\n # of statement, but here we drop down out of sqlalchemy's ORM and into the\n # sql abstraction in order to gain a little performance boost.\n values = [{'username': username, 'msg': obj.id} for username in usernames]\n if values:\n session.execute(user_assoc_table.insert(), values)\n\n values = [{'package': package, 'msg': obj.id} for package in packages]\n if values:\n session.execute(pack_assoc_table.insert(), values)\n\n # TODO -- can we avoid committing every time?\n session.flush()\n session.commit()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_or_create(cls, name):\n obj = cls.query.filter_by(name=name).one_or_none()\n if obj:\n return obj\n try:\n with session.begin_nested():\n obj = cls(name=name)\n session.add(obj)\n session.flush()\n return obj\n except IntegrityError:\n log.debug('Collision when adding %s(name=\"%s\"), returning existing object',\n cls.__name__, name)\n return cls.query.filter_by(name=name).one()", "response": "Get the instance of the class with the specified name. If it doesn t exist create it."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef isdir(path):\n system = get_instance(path)\n\n # User may use directory path without trailing '/'\n # like on standard file systems\n return system.isdir(system.ensure_dir_path(path))", "response": "Returns True if path is an existing directory."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef relpath(path, start=None):\n relative = get_instance(path).relpath(path)\n if start:\n # Storage relative path\n # Replaces \"\\\" by \"/\" for Windows.\n return os_path_relpath(relative, start=start).replace('\\\\', '/')\n return relative", "response": "Returns a relative file path to path either from the current directory or from a start directory."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef samefile(path1, path2):\n # Handles path-like objects and checks if storage\n path1, path1_is_storage = format_and_is_storage(path1)\n path2, path2_is_storage = format_and_is_storage(path2)\n\n # Local files: Redirects to \"os.path.samefile\"\n if not path1_is_storage and not path2_is_storage:\n return os_path_samefile(path1, path2)\n\n # One path is local, the other storage\n if not path1_is_storage or not path2_is_storage:\n return False\n\n with handle_os_exceptions():\n # Paths don't use same storage\n system = get_instance(path1)\n if system is not get_instance(path2):\n return False\n\n # Relative path are different\n elif system.relpath(path1) != system.relpath(path2):\n return False\n\n # Same files\n return True", "response": "Returns True if both pathname arguments refer to the same file or directory."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsplit the path into a pair of drive tail.", "response": "def splitdrive(path):\n \"\"\"\n Split the path into a pair (drive, tail) where drive is either a\n mount point or the empty string. On systems which do not use drive\n specifications, drive will always be the empty string.\n\n In all cases, drive + tail will be the same as path.\n\n Equivalent to \"os.path.splitdrive\".\n\n Args:\n path (path-like object): Path or URL.\n\n Returns:\n tuple of str: drive, tail.\n \"\"\"\n relative = get_instance(path).relpath(path)\n drive = path.rsplit(relative, 1)[0]\n if drive and not drive[-2:] == '//':\n # Keep \"/\" tail side\n relative = '/' + relative\n drive = drive.rstrip('/')\n return drive, relative"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nhandles pycosio exceptions and raises standard OS exceptions.", "response": "def handle_os_exceptions():\n \"\"\"\n Handles pycosio exceptions and raise standard OS exceptions.\n \"\"\"\n try:\n yield\n\n # Convert pycosio exception to equivalent OSError\n except ObjectException:\n exc_type, exc_value, _ = exc_info()\n raise _OS_EXCEPTIONS.get(exc_type, OSError)(exc_value)\n\n # Re-raise generic exceptions\n except (OSError, same_file_error, UnsupportedOperation):\n raise\n\n # Raise generic OSError for other exceptions\n except Exception:\n exc_type, exc_value, _ = exc_info()\n raise OSError('%s%s' % (\n exc_type, (', %s' % exc_value) if exc_value else ''))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a list containing the names of the entries in the directory given by path.", "response": "def listdir(path='.'):\n \"\"\"\n Return a list containing the names of the entries in the directory given by\n path.\n\n Equivalent to \"os.listdir\".\n\n Args:\n path (path-like object): Path or URL.\n\n Returns:\n list of str: Entries names.\n \"\"\"\n return [name.rstrip('/') for name, _ in\n get_instance(path).list_objects(path, first_level=True)]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a directory and all intermediate ones.", "response": "def makedirs(name, mode=0o777, exist_ok=False):\n \"\"\"\n Super-mkdir; create a leaf directory and all intermediate ones.\n Works like mkdir, except that any intermediate path segment\n (not just the rightmost) will be created if it does not exist.\n\n Equivalent to \"os.makedirs\".\n\n Args:\n name (path-like object): Path or URL.\n mode (int): The mode parameter is passed to os.mkdir();\n see the os.mkdir() description for how it is interpreted.\n Not supported on cloud storage objects.\n exist_ok (bool): Don't raises error if target directory already\n exists.\n\n Raises:\n FileExistsError: if exist_ok is False and if the target directory\n already exists.\n \"\"\"\n system = get_instance(name)\n\n # Checks if directory not already exists\n if not exist_ok and system.isdir(system.ensure_dir_path(name)):\n raise ObjectExistsError(\"File exists: '%s'\" % name)\n\n # Create directory\n system.make_dir(name)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef mkdir(path, mode=0o777, dir_fd=None):\n system = get_instance(path)\n relative = system.relpath(path)\n\n # Checks if parent directory exists\n parent_dir = dirname(relative.rstrip('/'))\n if parent_dir:\n parent = path.rsplit(relative, 1)[0] + parent_dir + '/'\n if not system.isdir(parent):\n raise ObjectNotFoundError(\n \"No such file or directory: '%s'\" % parent)\n\n # Checks if directory not already exists\n if system.isdir(system.ensure_dir_path(path)):\n raise ObjectExistsError(\"File exists: '%s'\" % path)\n\n # Create directory\n system.make_dir(relative, relative=True)", "response": "Creates a directory named path with numeric mode mode."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nremove a file. Equivalent to \"os.remove\" and \"os.unlink\". Args: path (path-like object): Path or URL. dir_fd: directory descriptors; see the os.remove() description for how it is interpreted. Not supported on cloud storage objects.", "response": "def remove(path, dir_fd=None):\n \"\"\"\n Remove a file.\n\n Equivalent to \"os.remove\" and \"os.unlink\".\n\n Args:\n path (path-like object): Path or URL.\n dir_fd: directory descriptors;\n see the os.remove() description for how it is interpreted.\n Not supported on cloud storage objects.\n \"\"\"\n system = get_instance(path)\n\n # Only support files\n if system.is_locator(path) or path[-1] == '/':\n raise is_a_directory_error(\"Is a directory: '%s'\" % path)\n\n # Remove\n system.remove(path)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nremoves a directory. Equivalent to \"os.rmdir\". Args: path (path-like object): Path or URL. dir_fd: directory descriptors; see the os.rmdir() description for how it is interpreted. Not supported on cloud storage objects.", "response": "def rmdir(path, dir_fd=None):\n \"\"\"\n Remove a directory.\n\n Equivalent to \"os.rmdir\".\n\n Args:\n path (path-like object): Path or URL.\n dir_fd: directory descriptors;\n see the os.rmdir() description for how it is interpreted.\n Not supported on cloud storage objects.\n \"\"\"\n system = get_instance(path)\n system.remove(system.ensure_dir_path(path))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn an iterator of os. DirEntry objects corresponding to the entries in the directory given by path.", "response": "def scandir(path='.'):\n \"\"\"\n Return an iterator of os.DirEntry objects corresponding to the entries in\n the directory given by path. The entries are yielded in arbitrary order,\n and the special entries '.' and '..' are not included.\n\n Equivalent to \"os.scandir\".\n\n Args:\n path (path-like object): Path or URL.\n If path is of type bytes (directly or indirectly through the\n PathLike interface), the type of the name and path attributes\n of each os.DirEntry will be bytes; in all other circumstances,\n they will be of type str.\n\n Returns:\n Generator of os.DirEntry: Entries information.\n \"\"\"\n # Handles path-like objects\n scandir_path = fsdecode(path).replace('\\\\', '/')\n\n if not is_storage(scandir_path):\n return os_scandir(scandir_path)\n\n return _scandir_generator(\n is_bytes=isinstance(fspath(path), (bytes, bytearray)),\n scandir_path=scandir_path, system=get_instance(scandir_path))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the base filename of the entry.", "response": "def name(self):\n \"\"\"\n The entry\u2019s base filename, relative to the scandir() path argument.\n\n Returns:\n str: name.\n \"\"\"\n name = self._name.rstrip('/')\n if self._bytes_path:\n name = fsencode(name)\n return name"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the full path name of the entry.", "response": "def path(self):\n \"\"\"\n The entry\u2019s full path name:\n equivalent to os.path.join(scandir_path, entry.name)\n where scandir_path is the scandir() path argument.\n\n The path is only absolute if the scandir() path argument was absolute.\n\n Returns:\n str: name.\n \"\"\"\n path = self._path.rstrip('/')\n if self._bytes_path:\n path = fsencode(path)\n return path"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn True if this entry is a file.", "response": "def is_file(self, follow_symlinks=True):\n \"\"\"\n Return True if this entry is a file or a symbolic link pointing to a\n file; return False if the entry is or points to a directory or other\n non-file entry, or if it doesn\u2019t exist anymore.\n\n The result is cached on the os.DirEntry object.\n\n Args:\n follow_symlinks (bool): Follow symlinks.\n Not supported on cloud storage objects.\n\n Returns:\n bool: True if directory exists.\n \"\"\"\n return self._system.isfile(\n path=self._path, client_kwargs=self._client_kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef stat(self, follow_symlinks=True):\n return self._system.stat(\n path=self._path, client_kwargs=self._client_kwargs,\n header=self._header)", "response": "Return a stat_result object for this entry."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef close(self):\n if self._writable and not self._closed:\n self._closed = True\n with self._seek_lock:\n self._flush_raw_or_buffered()\n if self._seek:\n with handle_os_exceptions():\n self._close_writable()", "response": "Flush the write buffers of the stream if applicable and\n close the object."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef flush(self):\n if self._writable:\n with self._seek_lock:\n self._flush_raw_or_buffered()\n\n # Clear the buffer\n self._write_buffer = bytearray(self._buffer_size)\n self._buffer_seek = 0", "response": "Flush the write buffers of the stream if applicable."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nflush raw or buffered data.", "response": "def _flush_raw_or_buffered(self):\n \"\"\"\n Flush using raw of buffered methods.\n \"\"\"\n # Flush only if bytes written\n # This avoid no required process/thread\n # creation and network call.\n # This step is performed by raw stream.\n if self._buffer_seek and self._seek:\n self._seek += 1\n with handle_os_exceptions():\n self._flush()\n\n # If data lower than buffer size\n # flush data with raw stream to reduce IO calls\n elif self._buffer_seek:\n self._raw._write_buffer = self._get_buffer()\n self._raw._seek = self._buffer_seek\n self._raw.flush()"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef peek(self, size=-1):\n if not self._readable:\n raise UnsupportedOperation('read')\n\n with self._seek_lock:\n self._raw.seek(self._seek)\n return self._raw._peek(size)", "response": "Reads the next size bytes from the stream without advancing the position."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreading up to size bytes from the underlying raw stream and returns up to size bytes.", "response": "def read(self, size=-1):\n \"\"\"\n Read and return up to size bytes,\n with at most one call to the underlying raw stream\u2019s.\n\n Use at most one call to the underlying raw stream\u2019s read method.\n\n Args:\n size (int): Number of bytes to read. -1 to read the\n stream until end.\n\n Returns:\n bytes: Object content\n \"\"\"\n if not self._readable:\n raise UnsupportedOperation('read')\n\n # Checks if EOF\n if self._seek == self._size:\n return b''\n\n # Returns existing buffer with no copy\n if size == self._buffer_size:\n queue_index = self._seek\n\n # Starts initial preloading on first call\n if queue_index == 0:\n self._preload_range()\n\n # Get buffer from future\n with handle_os_exceptions():\n buffer = self._read_queue.pop(queue_index).result()\n\n # Append another buffer preload at end of queue\n buffer_size = self._buffer_size\n index = queue_index + buffer_size * self._max_buffers\n if index < self._size:\n self._read_queue[index] = self._workers.submit(\n self._read_range, index, index + buffer_size)\n\n # Update seek\n self._seek += buffer_size\n else:\n self._seek = self._size\n\n return buffer\n\n # Uses a prealocated buffer\n if size != -1:\n buffer = bytearray(size)\n\n # Uses a mutable buffer\n else:\n buffer = bytearray()\n\n read_size = self.readinto(buffer)\n return memoryview(buffer)[:read_size].tobytes()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef readinto(self, b):\n if not self._readable:\n raise UnsupportedOperation('read')\n\n with self._seek_lock:\n # Gets seek\n seek = self._seek\n\n # Initializes queue\n queue = self._read_queue\n if seek == 0:\n # Starts initial preloading on first call\n self._preload_range()\n\n # Initializes read data buffer\n size = len(b)\n if size:\n # Preallocated buffer:\n # Use memory view to avoid copies\n b_view = memoryview(b)\n size_left = size\n else:\n # Dynamic buffer:\n # Can't avoid copy, read until EOF\n b_view = b\n size_left = -1\n b_end = 0\n\n # Starts reading\n buffer_size = self._buffer_size\n while size_left > 0 or size_left == -1:\n\n # Finds buffer position in queue and buffer seek\n start = seek % buffer_size\n queue_index = seek - start\n\n # Gets preloaded buffer\n try:\n buffer = queue[queue_index]\n except KeyError:\n # EOF\n break\n\n # Get buffer from future\n with handle_os_exceptions():\n try:\n queue[queue_index] = buffer = buffer.result()\n\n # Already evaluated\n except AttributeError:\n pass\n buffer_view = memoryview(buffer)\n data_size = len(buffer)\n\n # Checks if end of file reached\n if not data_size:\n break\n\n # Gets theoretical range to copy\n if size_left != -1:\n end = start + size_left\n else:\n end = data_size - start\n\n # Checks for end of buffer\n if end >= data_size:\n # Adjusts range to copy\n end = data_size\n\n # Removes consumed buffer from queue\n del queue[queue_index]\n\n # Append another buffer preload at end of queue\n index = queue_index + buffer_size * self._max_buffers\n if index < self._size:\n queue[index] = self._workers.submit(\n self._read_range, index, index + buffer_size)\n\n # Gets read size, updates seek and updates size left\n read_size = end - start\n if size_left != -1:\n size_left -= read_size\n seek += read_size\n\n # Defines read buffer range\n b_start = b_end\n b_end = b_start + read_size\n\n # Copy data from preload buffer to read buffer\n b_view[b_start:b_end] = buffer_view[start:end]\n\n # Updates seek and sync raw\n self._seek = seek\n self._raw.seek(seek)\n\n # Returns read size\n return b_end", "response": "Reads bytes into a pre - allocated writable bytes - like object b and returns the number of bytes read."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef seek(self, offset, whence=SEEK_SET):\n if not self._seekable:\n raise UnsupportedOperation('seek')\n\n # Only read mode is seekable\n with self._seek_lock:\n # Set seek using raw method and\n # sync buffered seek with raw seek\n self.raw.seek(offset, whence)\n self._seek = seek = self.raw._seek\n\n # Preload starting from current seek\n self._preload_range()\n\n return seek", "response": "Change the stream position to the given byte offset."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nwriting the given bytes - like object b to the underlying raw stream and returns the number of bytes written.", "response": "def write(self, b):\n \"\"\"\n Write the given bytes-like object, b, to the underlying raw stream,\n and return the number of bytes written.\n\n Args:\n b (bytes-like object): Bytes to write.\n\n Returns:\n int: The number of bytes written.\n \"\"\"\n if not self._writable:\n raise UnsupportedOperation('write')\n\n size = len(b)\n b_view = memoryview(b)\n size_left = size\n buffer_size = self._buffer_size\n max_buffers = self._max_buffers\n\n with self._seek_lock:\n end = self._buffer_seek\n buffer_view = memoryview(self._write_buffer)\n\n while size_left > 0:\n # Get range to copy\n start = end\n end = start + size_left\n\n if end > buffer_size:\n # End of buffer, need flush after copy\n end = buffer_size\n flush = True\n else:\n flush = False\n\n buffer_range = end - start\n\n # Update not remaining data size\n b_start = size - size_left\n size_left -= buffer_range\n\n # Copy data\n buffer_view[start:end] = b_view[b_start: b_start + buffer_range]\n\n # Flush buffer if needed\n if flush:\n # Update buffer seek\n # Needed to write the good amount of data\n self._buffer_seek = end\n\n # Update global seek, this is the number\n # of buffer flushed\n self._seek += 1\n\n # Block flush based on maximum number of\n # buffers in flush progress\n if max_buffers:\n futures = self._write_futures\n flush_wait = self._FLUSH_WAIT\n while sum(1 for future in futures\n if not future.done()) >= max_buffers:\n sleep(flush_wait)\n\n # Flush\n with handle_os_exceptions():\n self._flush()\n\n # Clear buffer\n self._write_buffer = bytearray(buffer_size)\n buffer_view = memoryview(self._write_buffer)\n end = 0\n\n # Update buffer seek\n self._buffer_seek = end\n return size"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create(content, error='H', version=None, mode=None, encoding=None):\n return QRCode(content, error, version, mode, encoding)", "response": "This function creates a new QR code."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef show(self, wait=1.2, scale=10, module_color=(0, 0, 0, 255),\n background=(255, 255, 255, 255), quiet_zone=4):\n \"\"\"Displays this QR code.\n\n This method is mainly intended for debugging purposes.\n\n This method saves the output of the :py:meth:`png` method (with a default\n scaling factor of 10) to a temporary file and opens it with the\n standard PNG viewer application or within the standard webbrowser. The\n temporary file is deleted afterwards.\n\n If this method does not show any result, try to increase the `wait`\n parameter. This parameter specifies the time in seconds to wait till\n the temporary file is deleted. Note, that this method does not return\n until the provided amount of seconds (default: 1.2) has passed.\n\n The other parameters are simply passed on to the `png` method.\n \"\"\"\n import os\n import time\n import tempfile\n import webbrowser\n \n try: # Python 2\n from urlparse import urljoin\n from urllib import pathname2url\n except ImportError: # Python 3\n from urllib.parse import urljoin\n from urllib.request import pathname2url\n\n f = tempfile.NamedTemporaryFile('wb', suffix='.png', delete=False)\n self.png(f, scale=scale, module_color=module_color,\n background=background, quiet_zone=quiet_zone)\n f.close()\n webbrowser.open_new_tab(urljoin('file:', pathname2url(f.name)))\n time.sleep(wait)\n os.unlink(f.name)", "response": "Displays this QR code."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef png_as_base64_str(self, scale=1, module_color=(0, 0, 0, 255),\n background=(255, 255, 255, 255), quiet_zone=4):\n \"\"\"This method uses the png render and returns the PNG image encoded as\n base64 string. This can be useful for creating dynamic PNG images for\n web development, since no file needs to be created.\n \n Example:\n >>> code = pyqrcode.create('Are you suggesting coconuts migrate?')\n >>> image_as_str = code.png_as_base64_str(scale=5)\n >>> html_img = ''.format(image_as_str)\n\n The parameters are passed directly to the :py:meth:`png` method. Refer\n to that method's documentation for the meaning behind the parameters.\n \n .. note::\n This method depends on the pypng module to actually create the\n PNG image.\n\n \"\"\"\n import io\n import base64\n \n with io.BytesIO() as virtual_file:\n self.png(file=virtual_file, scale=scale, module_color=module_color,\n background=background, quiet_zone=quiet_zone)\n image_as_str = base64.b64encode(virtual_file.getvalue()).decode(\"ascii\")\n return image_as_str", "response": "This method uses the png render and returns the PNG image encoded as base64 string."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a string representing an XBM image of the QR code.", "response": "def xbm(self, scale=1, quiet_zone=4):\n \"\"\"Returns a string representing an XBM image of the QR code.\n The XBM format is a black and white image format that looks like a\n C header file. \n \n Because displaying QR codes in Tkinter is the\n primary use case for this renderer, this method does not take a file\n parameter. Instead it retuns the rendered QR code data as a string.\n \n Example of using this renderer with Tkinter:\n >>> import pyqrcode\n >>> import tkinter\n >>> code = pyqrcode.create('Knights who say ni!')\n >>> code_xbm = code.xbm(scale=5)\n >>>\n >>> top = tkinter.Tk()\n >>> code_bmp = tkinter.BitmapImage(data=code_xbm)\n >>> code_bmp.config(foreground=\"black\")\n >>> code_bmp.config(background=\"white\")\n >>> label = tkinter.Label(image=code_bmp)\n >>> label.pack()\n\n \n The *scale* parameter sets how large to draw a single module. By\n default one pixel is used to draw a single module. This may make the\n code too small to be read efficiently. Increasing the scale will make\n the code larger. Only integer scales are usable. This method will\n attempt to coerce the parameter into an integer (e.g. 2.5 will become 2,\n and '3' will become 3). You can use the :py:meth:`get_png_size` method\n to calculate the actual pixel size of this image when displayed.\n\n The *quiet_zone* parameter sets how wide the quiet zone around the code\n should be. According to the standard this should be 4 modules. It is\n left settable because such a wide quiet zone is unnecessary in many\n applications where the QR code is not being printed.\n \"\"\"\n return builder._xbm(self.code, scale, quiet_zone)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _text(code, quiet_zone=4):\n buf = io.StringIO()\n\n border_row = '0' * (len(code[0]) + (quiet_zone*2))\n\n #Every QR code start with a quiet zone at the top\n for b in range(quiet_zone):\n buf.write(border_row)\n buf.write('\\n')\n\n for row in code:\n #Draw the starting quiet zone\n for b in range(quiet_zone):\n buf.write('0')\n\n #Actually draw the QR code\n for bit in row:\n if bit == 1:\n buf.write('1')\n elif bit == 0:\n buf.write('0')\n #This is for debugging unfinished QR codes,\n #unset pixels will be spaces.\n else:\n buf.write(' ')\n \n #Draw the ending quiet zone\n for b in range(quiet_zone):\n buf.write('0')\n buf.write('\\n')\n\n #Every QR code ends with a quiet zone at the bottom\n for b in range(quiet_zone):\n buf.write(border_row)\n buf.write('\\n')\n\n return buf.getvalue()", "response": "This method returns a text based representation of the QR code."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _xbm(code, scale=1, quiet_zone=4):\n try:\n str = unicode # Python 2\n except NameError:\n str = __builtins__['str']\n \n buf = io.StringIO()\n \n # Calculate the width in pixels\n pixel_width = (len(code[0]) + quiet_zone * 2) * scale\n \n # Add the size information and open the pixel data section\n buf.write('#define im_width ')\n buf.write(str(pixel_width))\n buf.write('\\n')\n buf.write('#define im_height ')\n buf.write(str(pixel_width))\n buf.write('\\n')\n buf.write('static char im_bits[] = {\\n')\n \n # Calculate the number of bytes per row\n byte_width = int(math.ceil(pixel_width / 8.0))\n \n # Add the top quiet zone\n buf.write(('0x00,' * byte_width + '\\n') * quiet_zone * scale)\n for row in code:\n # Add the left quiet zone\n row_bits = '0' * quiet_zone * scale\n # Add the actual QR code\n for pixel in row:\n row_bits += str(pixel) * scale\n # Add the right quiet zone\n row_bits += '0' * quiet_zone * scale\n # Format the row\n formated_row = ''\n for b in range(byte_width):\n formated_row += '0x{0:02x},'.format(int(row_bits[:8][::-1], 2))\n row_bits = row_bits[8:]\n formated_row += '\\n'\n # Add the formatted row\n buf.write(formated_row * scale)\n # Add the bottom quiet zone and close the pixel data section\n buf.write(('0x00,' * byte_width + '\\n') * quiet_zone * scale)\n buf.write('};')\n \n return buf.getvalue()", "response": "This function will format the QR code as a X BitMap."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _svg(code, version, file, scale=1, module_color='#000', background=None,\n quiet_zone=4, xmldecl=True, svgns=True, title=None, svgclass='pyqrcode',\n lineclass='pyqrline', omithw=False, debug=False):\n \"\"\"This function writes the QR code out as an SVG document. The\n code is drawn by drawing only the modules corresponding to a 1. They\n are drawn using a line, such that contiguous modules in a row\n are drawn with a single line. The file parameter is used to\n specify where to write the document to. It can either be a writable (binary)\n stream or a file path. The scale parameter is sets how large to draw\n a single module. By default one pixel is used to draw a single\n module. This may make the code to small to be read efficiently.\n Increasing the scale will make the code larger. This method will accept\n fractional scales (e.g. 2.5).\n\n :param module_color: Color of the QR code (default: ``#000`` (black))\n :param background: Optional background color.\n (default: ``None`` (no background))\n :param quiet_zone: Border around the QR code (also known as quiet zone)\n (default: ``4``). Set to zero (``0``) if the code shouldn't\n have a border.\n :param xmldecl: Inidcates if the XML declaration header should be written\n (default: ``True``)\n :param svgns: Indicates if the SVG namespace should be written\n (default: ``True``)\n :param title: Optional title of the generated SVG document.\n :param svgclass: The CSS class of the SVG document\n (if set to ``None``, the SVG element won't have a class).\n :param lineclass: The CSS class of the path element\n (if set to ``None``, the path won't have a class).\n :param omithw: Indicates if width and height attributes should be\n omitted (default: ``False``). If these attributes are omitted,\n a ``viewBox`` attribute will be added to the document.\n :param debug: Inidicates if errors in the QR code should be added to the\n output (default: ``False``).\n \"\"\"\n from functools import partial\n from xml.sax.saxutils import quoteattr\n\n def write_unicode(write_meth, unicode_str):\n \"\"\"\\\n Encodes the provided string into UTF-8 and writes the result using\n the `write_meth`.\n \"\"\"\n write_meth(unicode_str.encode('utf-8'))\n\n def line(x, y, length, relative):\n \"\"\"Returns coordinates to draw a line with the provided length.\n \"\"\"\n return '{0}{1} {2}h{3}'.format(('m' if relative else 'M'), x, y, length)\n\n def errline(col_number, row_number):\n \"\"\"Returns the coordinates to draw an error bit.\n \"\"\"\n # Debug path uses always absolute coordinates\n # .5 == stroke / 2\n return line(col_number + quiet_zone, row_number + quiet_zone + .5, 1, False)\n\n f, autoclose = _get_writable(file, 'wb')\n write = partial(write_unicode, f.write)\n write_bytes = f.write\n # Write the document header\n if xmldecl:\n write_bytes(b'\\n')\n write_bytes(b'')\n if title is not None:\n write('{0}'.format(title))\n\n # Draw a background rectangle if necessary\n if background is not None:\n write(''\n .format(size, background))\n write_bytes(b'')\n if debug and debug_path:\n write_bytes(b''.format(debug_path))\n # Close document\n write_bytes(b'\\n')\n if autoclose:\n f.close()", "response": "This function writes the QR code out as an SVG document."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _hex_to_rgb(color):\n if color[0] == '#':\n color = color[1:]\n if len(color) == 3:\n color = color[0] * 2 + color[1] * 2 + color[2] * 2\n if len(color) != 6:\n raise ValueError('Input #{0} is not in #RRGGBB format'.format(color))\n return [int(n, 16) for n in (color[:2], color[2:4], color[4:])]", "response": "Helper function to convert a color provided in hexadecimal format\n as RGB triple."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef grouper(self, n, iterable, fillvalue=None):\n args = [iter(iterable)] * n\n if hasattr(itertools, 'zip_longest'):\n return itertools.zip_longest(*args, fillvalue=fillvalue)\n return itertools.izip_longest(*args, fillvalue=fillvalue)", "response": "This generator yields a set of tuples where the iterable is evenly sized and fillvalue will be appended to the last tuple."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef encode(self):\n if self.mode == tables.modes['alphanumeric']:\n encoded = self.encode_alphanumeric()\n elif self.mode == tables.modes['numeric']:\n encoded = self.encode_numeric()\n elif self.mode == tables.modes['binary']:\n encoded = self.encode_bytes()\n elif self.mode == tables.modes['kanji']:\n encoded = self.encode_kanji()\n return encoded", "response": "This method encodes the data into a binary string using the appropriate algorithm specified by the mode."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef encode_numeric(self):\n with io.StringIO() as buf:\n #Break the number into groups of three digits\n for triplet in self.grouper(3, self.data):\n number = ''\n for digit in triplet:\n if isinstance(digit, int):\n digit = chr(digit)\n\n #Only build the string if digit is not None\n if digit:\n number = ''.join([number, digit])\n else:\n break\n\n #If the number is one digits, make a 4 bit field\n if len(number) == 1:\n bin = self.binary_string(number, 4)\n\n #If the number is two digits, make a 7 bit field\n elif len(number) == 2:\n bin = self.binary_string(number, 7)\n\n #Three digit numbers use a 10 bit field\n else:\n bin = self.binary_string(number, 10)\n\n buf.write(bin)\n return buf.getvalue()", "response": "This method encodes the QR code s data if its mode is\n numeric. It returns the data encoded as a binary string."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef encode_kanji(self):\n def two_bytes(data):\n \"\"\"Output two byte character code as a single integer.\"\"\"\n def next_byte(b):\n \"\"\"Make sure that character code is an int. Python 2 and\n 3 compatibility.\n \"\"\"\n if not isinstance(b, int):\n return ord(b)\n else:\n return b\n\n #Go through the data by looping to every other character\n for i in range(0, len(data), 2):\n yield (next_byte(data[i]) << 8) | next_byte(data[i+1])\n\n #Force the data into Kanji encoded bytes\n if isinstance(self.data, bytes):\n data = self.data.decode('shiftjis').encode('shiftjis')\n else:\n data = self.data.encode('shiftjis')\n \n #Now perform the algorithm that will make the kanji into 13 bit fields\n with io.StringIO() as buf:\n for asint in two_bytes(data):\n #Shift the two byte value as indicated by the standard\n if 0x8140 <= asint <= 0x9FFC:\n difference = asint - 0x8140\n elif 0xE040 <= asint <= 0xEBBF:\n difference = asint - 0xC140\n\n #Split the new value into most and least significant bytes\n msb = (difference >> 8)\n lsb = (difference & 0x00FF)\n\n #Calculate the actual 13 bit binary value\n buf.write('{0:013b}'.format((msb * 0xC0) + lsb))\n #Return the binary string\n return buf.getvalue()", "response": "This method encodes the QR code s data into a kanji encoded binary string. It returns the data encoded as a binary string."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_data(self):\n #Encode the data into a QR code\n self.buffer.write(self.binary_string(self.mode, 4))\n self.buffer.write(self.get_data_length())\n self.buffer.write(self.encode())\n\n #Converts the buffer into \"code word\" integers.\n #The online debugger outputs them this way, makes\n #for easier comparisons.\n #s = self.buffer.getvalue()\n #for i in range(0, len(s), 8):\n # print(int(s[i:i+8], 2), end=',')\n #print()\n \n #Fix for issue #3: https://github.com/mnooner256/pyqrcode/issues/3#\n #I was performing the terminate_bits() part in the encoding.\n #As per the standard, terminating bits are only supposed to\n #be added after the bit stream is complete. I took that to\n #mean after the encoding, but actually it is after the entire\n #bit stream has been constructed.\n bits = self.terminate_bits(self.buffer.getvalue())\n if bits is not None:\n self.buffer.write(bits)\n\n #delimit_words and add_words can return None\n add_bits = self.delimit_words()\n if add_bits:\n self.buffer.write(add_bits)\n \n fill_bytes = self.add_words()\n if fill_bytes:\n self.buffer.write(fill_bytes)\n \n #Get a numeric representation of the data\n data = [int(''.join(x),2)\n for x in self.grouper(8, self.buffer.getvalue())]\n\n #This is the error information for the code\n error_info = tables.eccwbi[self.version][self.error]\n\n #This will hold our data blocks\n data_blocks = []\n\n #This will hold our error blocks\n error_blocks = []\n\n #Some codes have the data sliced into two different sized blocks\n #for example, first two 14 word sized blocks, then four 15 word\n #sized blocks. This means that slicing size can change over time.\n data_block_sizes = [error_info[2]] * error_info[1]\n if error_info[3] != 0:\n data_block_sizes.extend([error_info[4]] * error_info[3])\n\n #For every block of data, slice the data into the appropriate\n #sized block\n current_byte = 0\n for n_data_blocks in data_block_sizes:\n data_blocks.append(data[current_byte:current_byte+n_data_blocks])\n current_byte += n_data_blocks\n \n #I am not sure about the test after the \"and\". This was added to\n #fix a bug where after delimit_words padded the bit stream, a zero\n #byte ends up being added. After checking around, it seems this extra\n #byte is supposed to be chopped off, but I cannot find that in the\n #standard! I am adding it to solve the bug, I believe it is correct.\n if current_byte < len(data):\n raise ValueError('Too much data for this code version.')\n\n #DEBUG CODE!!!!\n #Print out the data blocks\n #print('Data Blocks:\\n{0}'.format(data_blocks))\n\n #Calculate the error blocks\n for n, block in enumerate(data_blocks):\n error_blocks.append(self.make_error_block(block, n))\n\n #DEBUG CODE!!!!\n #Print out the error blocks\n #print('Error Blocks:\\n{0}'.format(error_blocks))\n\n #Buffer we will write our data blocks into\n data_buffer = io.StringIO()\n\n #Add the data blocks\n #Write the buffer such that: block 1 byte 1, block 2 byte 1, etc.\n largest_block = max(error_info[2], error_info[4])+error_info[0]\n for i in range(largest_block):\n for block in data_blocks:\n if i < len(block):\n data_buffer.write(self.binary_string(block[i], 8))\n\n #Add the error code blocks.\n #Write the buffer such that: block 1 byte 1, block 2 byte 2, etc.\n for i in range(error_info[0]):\n for block in error_blocks:\n data_buffer.write(self.binary_string(block[i], 8))\n\n self.buffer = data_buffer", "response": "This function properly constructs a QR code s data string. It takes the data and returns the data string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nadding words to the data block of the QR code.", "response": "def add_words(self):\n \"\"\"The data block must fill the entire data capacity of the QR code.\n If we fall short, then we must add bytes to the end of the encoded\n data field. The value of these bytes are specified in the standard.\n \"\"\"\n\n data_blocks = len(self.buffer.getvalue()) // 8\n total_blocks = tables.data_capacity[self.version][self.error][0] // 8\n needed_blocks = total_blocks - data_blocks\n\n if needed_blocks == 0:\n return None\n\n #This will return item1, item2, item1, item2, etc.\n block = itertools.cycle(['11101100', '00010001'])\n\n #Create a string of the needed blocks\n return ''.join([next(block) for x in range(needed_blocks)])"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef add_detection_pattern(self, m):\n\n #Draw outer black box\n for i in range(7):\n inv = -(i+1)\n for j in [0,6,-1,-7]:\n m[j][i] = 1\n m[i][j] = 1\n m[inv][j] = 1\n m[j][inv] = 1\n\n #Draw inner white box\n for i in range(1, 6):\n inv = -(i+1)\n for j in [1, 5, -2, -6]:\n m[j][i] = 0\n m[i][j] = 0\n m[inv][j] = 0\n m[j][inv] = 0\n\n #Draw inner black box\n for i in range(2, 5):\n for j in range(2, 5):\n inv = -(i+1)\n m[i][j] = 1\n m[inv][j] = 1\n m[j][inv] = 1\n\n #Draw white border\n for i in range(8):\n inv = -(i+1)\n for j in [7, -8]:\n m[i][j] = 0\n m[j][i] = 0\n m[inv][j] = 0\n m[j][inv] = 0\n\n #To keep the code short, it draws an extra box\n #in the lower right corner, this removes it.\n for i in range(-8, 0):\n for j in range(-8, 0):\n m[i][j] = ' '\n\n #Add the timing pattern\n bit = itertools.cycle([1,0])\n for i in range(8, (len(m)-8)):\n b = next(bit)\n m[i][6] = b\n m[6][i] = b\n\n #Add the extra black pixel\n m[-8][8] = 1", "response": "This method adds the detection patterns to the QR code."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_position_pattern(self, m):\n #Version 1 does not have a position adjustment pattern\n if self.version == 1:\n return\n\n #Get the coordinates for where to place the boxes\n coordinates = tables.position_adjustment[self.version]\n\n #Get the max and min coordinates to handle special cases\n min_coord = coordinates[0]\n max_coord = coordinates[-1]\n\n #Draw a box at each intersection of the coordinates\n for i in coordinates:\n for j in coordinates:\n #Do not draw these boxes because they would\n #interfere with the detection pattern\n if (i == min_coord and j == min_coord) or \\\n (i == min_coord and j == max_coord) or \\\n (i == max_coord and j == min_coord):\n continue\n\n #Center black pixel\n m[i][j] = 1\n\n #Surround the pixel with a white box\n for x in [-1,1]:\n m[i+x][j+x] = 0\n m[i+x][j] = 0\n m[i][j+x] = 0\n m[i-x][j+x] = 0\n m[i+x][j-x] = 0\n\n #Surround the white box with a black box\n for x in [-2,2]:\n for y in [0,-1,1]:\n m[i+x][j+x] = 1\n m[i+x][j+y] = 1\n m[i+y][j+x] = 1\n m[i-x][j+x] = 1\n m[i+x][j-x] = 1", "response": "This method draws the position adjustment patterns onto the QR code code."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nadd the version pattern to the internal table m.", "response": "def add_version_pattern(self, m):\n \"\"\"For QR codes with a version 7 or higher, a special pattern\n specifying the code's version is required.\n\n For further information see:\n http://www.thonky.com/qr-code-tutorial/format-version-information/#example-of-version-7-information-string\n \"\"\"\n if self.version < 7:\n return\n\n #Get the bit fields for this code's version\n #We will iterate across the string, the bit string\n #needs the least significant digit in the zero-th position\n field = iter(tables.version_pattern[self.version][::-1])\n\n #Where to start placing the pattern\n start = len(m)-11\n\n #The version pattern is pretty odd looking\n for i in range(6):\n #The pattern is three modules wide\n for j in range(start, start+3):\n bit = int(next(field))\n\n #Bottom Left\n m[i][j] = bit\n\n #Upper right\n m[j][i] = bit"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef split_pulls(all_issues, project=\"arokem/python-matlab-bridge\"):\n pulls = []\n issues = []\n for i in all_issues:\n if is_pull_request(i):\n pull = get_pull_request(project, i['number'], auth=True)\n pulls.append(pull)\n else:\n issues.append(i)\n return issues, pulls", "response": "split a list of closed issues into non - PR Issues and Pull Requests"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef issues_closed_since(period=timedelta(days=365), project=\"arokem/python-matlab-bridge\", pulls=False):\n\n which = 'pulls' if pulls else 'issues'\n\n if isinstance(period, timedelta):\n since = round_hour(datetime.utcnow() - period)\n else:\n since = period\n url = \"https://api.github.com/repos/%s/%s?state=closed&sort=updated&since=%s&per_page=%i\" % (project, which, since.strftime(ISO8601), PER_PAGE)\n allclosed = get_paged_request(url, headers=make_auth_header())\n \n filtered = [ i for i in allclosed if _parse_datetime(i['closed_at']) > since ]\n if pulls:\n filtered = [ i for i in filtered if _parse_datetime(i['merged_at']) > since ]\n # filter out PRs not against master (backports)\n filtered = [ i for i in filtered if i['base']['ref'] == 'master' ]\n else:\n filtered = [ i for i in filtered if not is_pull_request(i) ]\n \n return filtered", "response": "Get all issues closed since a particular point in time."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef sorted_by_field(issues, field='closed_at', reverse=False):\n return sorted(issues, key = lambda i:i[field], reverse=reverse)", "response": "Return a list of issues sorted by field."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef report(issues, show_urls=False):\n # titles may have unicode in them, so we must encode everything below\n if show_urls:\n for i in issues:\n print(u'#%d: %s' % (i['number'],\n i['title'].replace(u'`', u'``')))\n else:\n for i in issues:\n print(u'* %d: %s' % (i['number'], i['title'].replace(u'`', u'``')))", "response": "Summary report about a list of issues printing number and title."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nwriting a numpy array and its shape to base64 buffers", "response": "def encode_ndarray(obj):\n \"\"\"Write a numpy array and its shape to base64 buffers\"\"\"\n shape = obj.shape\n if len(shape) == 1:\n shape = (1, obj.shape[0])\n if obj.flags.c_contiguous:\n obj = obj.T\n elif not obj.flags.f_contiguous:\n obj = asfortranarray(obj.T)\n else:\n obj = obj.T\n try:\n data = obj.astype(float64).tobytes()\n except AttributeError:\n data = obj.astype(float64).tostring()\n\n data = base64.b64encode(data).decode('utf-8')\n return data, shape"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nextracting a numpy array from a base64 buffer", "response": "def decode_arr(data):\n \"\"\"Extract a numpy array from a base64 buffer\"\"\"\n data = data.encode('utf-8')\n return frombuffer(base64.b64decode(data), float64)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef run_func(self, func_path, *func_args, **kwargs):\n if not self.started:\n raise ValueError('Session not started, use start()')\n\n nargout = kwargs.pop('nargout', 1)\n func_args += tuple(item for pair in zip(kwargs.keys(), kwargs.values())\n for item in pair)\n dname = os.path.dirname(func_path)\n fname = os.path.basename(func_path)\n func_name, ext = os.path.splitext(fname)\n if ext and not ext == '.m':\n raise TypeError('Need to give path to .m file')\n return self._json_response(cmd='eval',\n func_name=func_name,\n func_args=func_args or '',\n dname=dname,\n nargout=nargout)", "response": "Run a function in Matlab and return the result."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nbinds a Matlab function to the instance of the named attribute.", "response": "def _bind_method(self, name, unconditionally=False):\n \"\"\"Generate a Matlab function and bind it to the instance\n\n This is where the magic happens. When an unknown attribute of the\n Matlab class is requested, it is assumed to be a call to a\n Matlab function, and is generated and bound to the instance.\n\n This works because getattr() falls back to __getattr__ only if no\n attributes of the requested name can be found through normal\n routes (__getattribute__, __dict__, class tree).\n\n bind_method first checks whether the requested name is a callable\n Matlab function before generating a binding.\n\n Parameters\n ----------\n name : str\n The name of the Matlab function to call\n e.g. 'sqrt', 'sum', 'svd', etc\n unconditionally : bool, optional\n Bind the method without performing\n checks. Used to bootstrap methods that are required and\n know to exist\n\n Returns\n -------\n MatlabFunction\n A reference to a newly bound MatlabFunction instance if the\n requested name is determined to be a callable function\n\n Raises\n ------\n AttributeError: if the requested name is not a callable\n Matlab function\n\n \"\"\"\n # TODO: This does not work if the function is a mex function inside a folder of the same name\n exists = self.run_func('exist', name)['result'] in [2, 3, 5]\n if not unconditionally and not exists:\n raise AttributeError(\"'Matlab' object has no attribute '%s'\" % name)\n\n # create a new method instance\n method_instance = MatlabFunction(weakref.ref(self), name)\n method_instance.__name__ = name\n\n # bind to the Matlab instance with a weakref (to avoid circular references)\n if sys.version.startswith('3'):\n method = types.MethodType(method_instance, weakref.ref(self))\n else:\n method = types.MethodType(method_instance, weakref.ref(self),\n _Session)\n setattr(self, name, method)\n return getattr(self, name)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nformats a line of Matlab into a markdown line or a code line.", "response": "def format_line(line):\n \"\"\"\n Format a line of Matlab into either a markdown line or a code line.\n\n Parameters\n ----------\n line : str\n The line of code to be formatted. Formatting occurs according to the\n following rules:\n\n - If the line starts with (at least) two %% signs, a new cell will be\n started.\n\n - If the line doesn't start with a '%' sign, it is assumed to be legit\n matlab code. We will continue to add to the same cell until reaching\n the next comment line\n \"\"\"\n if line.startswith('%%'):\n md = True\n new_cell = True\n source = line.split('%%')[1] + '\\n' # line-breaks in md require a line\n # gap!\n\n elif line.startswith('%'):\n md = True\n new_cell = False\n source = line.split('%')[1] + '\\n'\n\n else:\n md = False\n new_cell = False\n source = line\n\n\n return new_cell, md, source"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert the lines of an m file into an IPython notebooknode.", "response": "def lines_to_notebook(lines, name=None):\n \"\"\"\n Convert the lines of an m file into an IPython notebook\n\n Parameters\n ----------\n lines : list\n A list of strings. Each element is a line in the m file\n\n Returns\n -------\n notebook : an IPython NotebookNode class instance, containing the\n information required to create a file\n\n\n \"\"\"\n source = []\n md = np.empty(len(lines), dtype=object)\n new_cell = np.empty(len(lines), dtype=object)\n for idx, l in enumerate(lines):\n new_cell[idx], md[idx], this_source = format_line(l)\n # Transitions between markdown and code and vice-versa merit a new\n # cell, even if no newline, or \"%%\" is found. Make sure not to do this\n # check for the very first line!\n if idx>1 and not new_cell[idx]:\n if md[idx] != md[idx-1]:\n new_cell[idx] = True\n\n source.append(this_source)\n # This defines the breaking points between cells:\n new_cell_idx = np.hstack([np.where(new_cell)[0], -1])\n\n # Listify the sources:\n cell_source = [source[new_cell_idx[i]:new_cell_idx[i+1]]\n for i in range(len(new_cell_idx)-1)]\n cell_md = [md[new_cell_idx[i]] for i in range(len(new_cell_idx)-1)]\n cells = []\n\n # Append the notebook with loading matlab magic extension\n notebook_head = \"import pymatbridge as pymat\\n\" + \"ip = get_ipython()\\n\" \\\n + \"pymat.load_ipython_extension(ip)\"\n cells.append(nbformat.new_code_cell(notebook_head))#, language='python'))\n\n for cell_idx, cell_s in enumerate(cell_source):\n if cell_md[cell_idx]:\n cells.append(nbformat.new_markdown_cell(cell_s))\n else:\n cell_s.insert(0, '%%matlab\\n')\n cells.append(nbformat.new_code_cell(cell_s))#, language='matlab'))\n\n #ws = nbformat.new_worksheet(cells=cells)\n notebook = nbformat.new_notebook(cells=cells)\n return notebook"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef convert_mfile(mfile, outfile=None):\n lines = mfile_to_lines(mfile)\n nb = lines_to_notebook(lines)\n if outfile is None:\n outfile = mfile.split('.m')[0] + '.ipynb'\n with open(outfile, 'w') as fid:\n nbwrite(nb, fid)", "response": "Convert a Matlab m - file into a Matlab notebook in ipynb format"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef load_ipython_extension(ip, **kwargs):\n global _loaded\n if not _loaded:\n ip.register_magics(MatlabMagics(ip, **kwargs))\n _loaded = True", "response": "Load the extension in IPython."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef post_gist(content, description='', filename='file', auth=False):\n post_data = json.dumps({\n \"description\": description,\n \"public\": True,\n \"files\": {\n filename: {\n \"content\": content\n }\n }\n }).encode('utf-8')\n\n headers = make_auth_header() if auth else {}\n response = requests.post(\"https://api.github.com/gists\", data=post_data, headers=headers)\n response.raise_for_status()\n response_data = json.loads(response.text)\n return response_data['html_url']", "response": "Post some text to a Gist and return the URL."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting pull request info by number", "response": "def get_pull_request(project, num, auth=False):\n \"\"\"get pull request info by number\n \"\"\"\n url = \"https://api.github.com/repos/{project}/pulls/{num}\".format(project=project, num=num)\n if auth:\n header = make_auth_header()\n else:\n header = None\n response = requests.get(url, headers=header)\n response.raise_for_status()\n return json.loads(response.text, object_hook=Obj)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_pull_request_files(project, num, auth=False):\n url = \"https://api.github.com/repos/{project}/pulls/{num}/files\".format(project=project, num=num)\n if auth:\n header = make_auth_header()\n else:\n header = None\n return get_paged_request(url, headers=header)", "response": "get list of files in a pull request"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget a full list handling APIv3 s paging", "response": "def get_paged_request(url, headers=None, **params):\n \"\"\"get a full list, handling APIv3's paging\"\"\"\n results = []\n params.setdefault(\"per_page\", 100)\n while True:\n if '?' in url:\n params = None\n print(\"fetching %s\" % url, file=sys.stderr)\n else:\n print(\"fetching %s with %s\" % (url, params), file=sys.stderr)\n response = requests.get(url, headers=headers, params=params)\n response.raise_for_status()\n results.extend(response.json())\n if 'next' in response.links:\n url = response.links['next']['url']\n else:\n break\n return results"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets pull request list", "response": "def get_pulls_list(project, auth=False, **params):\n \"\"\"get pull request list\"\"\"\n params.setdefault(\"state\", \"closed\")\n url = \"https://api.github.com/repos/{project}/pulls\".format(project=project)\n if auth:\n headers = make_auth_header()\n else:\n headers = None\n pages = get_paged_request(url, headers=headers, **params)\n return pages"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef encode_multipart_formdata(fields, boundary=None):\n # copy requests imports in here:\n from io import BytesIO\n from requests.packages.urllib3.filepost import (\n choose_boundary, six, writer, b, get_content_type\n )\n body = BytesIO()\n if boundary is None:\n boundary = choose_boundary()\n\n for fieldname, value in iter_fields(fields):\n body.write(b('--%s\\r\\n' % (boundary)))\n\n if isinstance(value, tuple):\n filename, data = value\n writer(body).write('Content-Disposition: form-data; name=\"%s\"; '\n 'filename=\"%s\"\\r\\n' % (fieldname, filename))\n body.write(b('Content-Type: %s\\r\\n\\r\\n' %\n (get_content_type(filename))))\n else:\n data = value\n writer(body).write('Content-Disposition: form-data; name=\"%s\"\\r\\n'\n % (fieldname))\n body.write(b'Content-Type: text/plain\\r\\n\\r\\n')\n\n if isinstance(data, int):\n data = str(data) # Backwards compatibility\n if isinstance(data, six.text_type):\n writer(body).write(data)\n else:\n body.write(data)\n\n body.write(b'\\r\\n')\n\n body.write(b('--%s--\\r\\n' % (boundary)))\n\n content_type = b('multipart/form-data; boundary=%s' % boundary)\n\n return body.getvalue(), content_type", "response": "Encode a dictionary of fields using the multipart - form - data mime format."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nuploading a file to the GitHub downloads area", "response": "def post_download(project, filename, name=None, description=\"\"):\n \"\"\"Upload a file to the GitHub downloads area\"\"\"\n if name is None:\n name = os.path.basename(filename)\n with open(filename, 'rb') as f:\n filedata = f.read()\n\n url = \"https://api.github.com/repos/{project}/downloads\".format(project=project)\n\n payload = json.dumps(dict(name=name, size=len(filedata),\n description=description))\n response = requests.post(url, data=payload, headers=make_auth_header())\n response.raise_for_status()\n reply = json.loads(response.content)\n s3_url = reply['s3_url']\n\n fields = dict(\n key=reply['path'],\n acl=reply['acl'],\n success_action_status=201,\n Filename=reply['name'],\n AWSAccessKeyId=reply['accesskeyid'],\n Policy=reply['policy'],\n Signature=reply['signature'],\n file=(reply['name'], filedata),\n )\n fields['Content-Type'] = reply['mime_type']\n data, content_type = encode_multipart_formdata(fields)\n s3r = requests.post(s3_url, data=data, headers={'Content-Type': content_type})\n return s3r"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking that the given path is an executable regular file or a symlink to a file.", "response": "def is_executable_file(path):\n \"\"\"Checks that path is an executable regular file (or a symlink to a file).\n\n This is roughly ``os.path isfile(path) and os.access(path, os.X_OK)``, but\n on some platforms :func:`os.access` gives us the wrong answer, so this\n checks permission bits directly.\n\n Note\n ----\n This function is taken from the pexpect module, see module doc-string for\n license.\n \"\"\"\n # follow symlinks,\n fpath = os.path.realpath(path)\n\n # return False for non-files (directories, fifo, etc.)\n if not os.path.isfile(fpath):\n return False\n\n # On Solaris, etc., \"If the process has appropriate privileges, an\n # implementation may indicate success for X_OK even if none of the\n # execute file permission bits are set.\"\n #\n # For this reason, it is necessary to explicitly check st_mode\n\n # get file mode using os.stat, and check if `other',\n # that is anybody, may read and execute.\n mode = os.stat(fpath).st_mode\n if mode & stat.S_IROTH and mode & stat.S_IXOTH:\n return True\n\n # get current user's group ids, and check if `group',\n # when matching ours, may read and execute.\n user_gids = os.getgroups() + [os.getgid()]\n if (os.stat(fpath).st_gid in user_gids and\n mode & stat.S_IRGRP and mode & stat.S_IXGRP):\n return True\n\n # finally, if file owner matches our effective userid,\n # check if `user', may read and execute.\n user_gids = os.getgroups() + [os.getgid()]\n if (os.stat(fpath).st_uid == os.geteuid() and\n mode & stat.S_IRUSR and mode & stat.S_IXUSR):\n return True\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nbuilding the messenger mex for MATLAB", "response": "def build_matlab(static=False):\n \"\"\"build the messenger mex for MATLAB\n\n static : bool\n Determines if the zmq library has been statically linked.\n If so, it will append the command line option -DZMQ_STATIC\n when compiling the mex so it matches libzmq.\n \"\"\"\n cfg = get_config()\n # To deal with spaces, remove quotes now, and add\n # to the full commands themselves.\n if 'matlab_bin' in cfg and cfg['matlab_bin'] != '.':\n matlab_bin = cfg['matlab_bin'].strip('\"')\n else: # attempt to autodetect MATLAB filepath\n matlab_bin = which_matlab()\n if matlab_bin is None:\n raise ValueError(\"specify 'matlab_bin' in cfg file\")\n # Get the extension\n extcmd = esc(os.path.join(matlab_bin, \"mexext\"))\n extension = subprocess.check_output(extcmd, shell=use_shell)\n extension = extension.decode('utf-8').rstrip('\\r\\n')\n\n # Build the mex file\n mex = esc(os.path.join(matlab_bin, \"mex\"))\n paths = \"-L%(zmq_lib)s -I%(zmq_inc)s\" % cfg\n make_cmd = '%s -O %s -lzmq ./src/messenger.c' % (mex, paths)\n if static:\n make_cmd += ' -DZMQ_STATIC'\n do_build(make_cmd, 'messenger.%s' % extension)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nadds an existing function to a class as a method.", "response": "def add_method(function, klass, name=None):\r\n '''Add an existing function to a class as a method.\r\n\r\n Note: Consider using the extend decorator as a more readable alternative\r\n to using this function directly.\r\n\r\n Args:\r\n function: The function to be added to the class klass.\r\n\r\n klass: The class to which the new method will be added.\r\n\r\n name: An optional name for the new method. If omitted or None the\r\n original name of the function is used.\r\n\r\n Returns:\r\n The function argument unmodified.\r\n\r\n Raises:\r\n ValueError: If klass already has an attribute with the same name as the\r\n extension method.\r\n '''\r\n # Should we be using functools.update_wrapper in here?\r\n if name is None:\r\n name = function_name(function)\r\n if hasattr(klass, name):\r\n raise ValueError(\"Cannot replace existing attribute with method \"\r\n \"'{name}'\".format(name=name))\r\n setattr(klass, name, function)\r\n return function"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef extend(klass, name=None):\r\n '''A function decorator for extending an existing class.\r\n\r\n Use as a decorator for functions to add to an existing class.\r\n\r\n Args:\r\n klass: The class to be decorated.\r\n\r\n name: The name the new method is to be given in the klass class.\r\n\r\n Returns:\r\n A decorator function which accepts a single function as its only\r\n argument. The decorated function will be added to class klass.\r\n\r\n Raises:\r\n ValueError: If klass already has an attribute with the same name as the\r\n extension method.\r\n '''\r\n def decorator(f):\r\n return add_method(f, klass, name)\r\n\r\n return decorator", "response": "A function decorator for extending an existing class. Use as a decorator for functions to add to an existing class."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ntransforms each element of a sequence into a new form.", "response": "def select(\n self,\n selector):\n '''Transforms each element of a sequence into a new form.\n\n Each element of the source is transformed through a selector function\n to produce a corresponding element in teh result sequence.\n\n If the selector is identity the method will return self.\n\n Note: This method uses deferred execution.\n\n Args:\n selector: A unary function mapping a value in the source sequence\n to the corresponding value in the generated generated sequence.\n The single positional argument to the selector function is the\n element value. The return value of the selector function\n should be the corresponding element of the result sequence.\n\n Returns:\n A Queryable over generated sequence whose elements are the result\n of invoking the selector function on each element of the source\n sequence.\n\n Raises:\n ValueError: If this Queryable has been closed.\n TypeError: If selector is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call select() on a closed Queryable.\")\n\n try:\n selector = make_selector(selector)\n except ValueError:\n raise TypeError(\"select() parameter selector={selector} cannot be\"\n \"converted into a callable \"\n \"selector\".format(selector=repr(selector)))\n\n if selector is identity:\n return self\n\n return self._create(imap(selector, self))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef select_with_index(\n self,\n selector=IndexedElement,\n transform=identity):\n '''Transforms each element of a sequence into a new form, incorporating\n the index of the element.\n\n Each element is transformed through a selector function which accepts\n the element value and its zero-based index in the source sequence. The\n generated sequence is lazily evaluated.\n\n Note: This method uses deferred execution.\n\n Args:\n selector: A binary function mapping the index of a value in\n the source sequence and the element value itself to the\n corresponding value in the generated sequence. The two\n positional arguments of the selector function are the zero-\n based index of the current element and the value of the current\n element. The return value should be the corresponding value in\n the result sequence. The default selector produces an IndexedElement\n containing the index and the element giving this function\n similar behaviour to the built-in enumerate().\n\n Returns:\n A Queryable whose elements are the result of invoking the selector\n function on each element of the source sequence\n\n Raises:\n ValueError: If this Queryable has been closed.\n TypeError: If selector is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call select_with_index() on a \"\n \"closed Queryable.\")\n\n if not is_callable(selector):\n raise TypeError(\"select_with_index() parameter selector={0} is \"\n \"not callable\".format(repr(selector)))\n\n if not is_callable(transform):\n raise TypeError(\"select_with_index() parameter item_selector={0} is \"\n \"not callable\".format(repr(selector)))\n\n return self._create(itertools.starmap(selector, enumerate(imap(transform, iter(self)))))", "response": "Returns a new sequence where each element of a source sequence is transformed into a new form incorporating\n the index of the element."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\napplies a callable to each element in an input sequence generating a new sequence of 2 - tuples where the first element is the input value and the second is the transformed input value.", "response": "def select_with_correspondence(\n self,\n selector,\n result_selector=KeyedElement):\n '''Apply a callable to each element in an input sequence, generating a new\n sequence of 2-tuples where the first element is the input value and the\n second is the transformed input value.\n\n The generated sequence is lazily evaluated.\n\n Note: This method uses deferred execution.\n\n Args:\n selector: A unary function mapping a value in the source sequence\n to the second argument of the result selector.\n\n result_selector: A binary callable mapping the of a value in\n the source sequence and the transformed value to the\n corresponding value in the generated sequence. The two\n positional arguments of the selector function are the original\n source element and the transformed value. The return value\n should be the corresponding value in the result sequence. The\n default selector produces a KeyedElement containing the index\n and the element giving this function similar behaviour to the\n built-in enumerate().\n\n Returns:\n When using the default selector, a Queryable whose elements are\n KeyedElements where the first element is from the input sequence\n and the second is the result of invoking the transform function on\n the first value.\n\n Raises:\n ValueError: If this Queryable has been closed.\n TypeError: If transform is not callable.\n\n '''\n if self.closed():\n raise ValueError(\"Attempt to call select_with_correspondence() on a \"\n \"closed Queryable.\")\n\n if not is_callable(selector):\n raise TypeError(\"select_with_correspondence() parameter selector={0} is \"\n \"not callable\".format(repr(selector)))\n\n if not is_callable(result_selector):\n raise TypeError(\"select_with_correspondence() parameter result_selector={0} is \"\n \"not callable\".format(repr(result_selector)))\n\n return self._create(result_selector(elem, selector(elem)) for elem in iter(self))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a new Queryable over the elements of the source sequence that are contained in the result sequence.", "response": "def select_many(\n self,\n collection_selector=identity,\n result_selector=identity):\n '''Projects each element of a sequence to an intermediate new sequence,\n flattens the resulting sequences into one sequence and optionally\n transforms the flattened sequence using a selector function.\n\n Note: This method uses deferred execution.\n\n Args:\n collection_selector: A unary function mapping each element of the\n source iterable into an intermediate sequence. The single\n argument of the collection_selector is the value of an element\n from the source sequence. The return value should be an\n iterable derived from that element value. The default\n collection_selector, which is the identity function, assumes\n that each element of the source sequence is itself iterable.\n\n result_selector: An optional unary function mapping the elements in\n the flattened intermediate sequence to corresponding elements\n of the result sequence. The single argument of the\n result_selector is the value of an element from the flattened\n intermediate sequence. The return value should be the\n corresponding value in the result sequence. The default\n result_selector is the identity function.\n\n Returns:\n A Queryable over a generated sequence whose elements are the result\n of applying the one-to-many collection_selector to each element of\n the source sequence, concatenating the results into an intermediate\n sequence, and then mapping each of those elements through the\n result_selector into the result sequence.\n\n Raises:\n ValueError: If this Queryable has been closed.\n TypeError: If either collection_selector or result_selector are not\n callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call select_many() on a closed \"\n \"Queryable.\")\n\n if not is_callable(collection_selector):\n raise TypeError(\"select_many() parameter projector={0} is not \"\n \"callable\".format(repr(collection_selector)))\n\n if not is_callable(result_selector):\n raise TypeError(\"select_many() parameter selector={selector} is \"\n \" not callable\".format(selector=repr(result_selector)))\n\n sequences = self.select(collection_selector)\n chained_sequence = itertools.chain.from_iterable(sequences)\n return self._create(chained_sequence).select(result_selector)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nprojects each element of a sequence to an intermediate new sequence and concatenates the resulting sequence into one sequence and optionally transforms the flattened intermediate sequence into one sequence.", "response": "def select_many_with_index(\n self,\n collection_selector=IndexedElement,\n result_selector=lambda source_element,\n collection_element: collection_element):\n '''Projects each element of a sequence to an intermediate new sequence,\n incorporating the index of the element, flattens the resulting sequence\n into one sequence and optionally transforms the flattened sequence\n using a selector function.\n\n Note: This method uses deferred execution.\n\n Args:\n collection_selector: A binary function mapping each element of the\n source sequence into an intermediate sequence, by incorporating\n its index in the source sequence. The two positional arguments\n to the function are the zero-based index of the source element\n and the value of the element. The result of the function\n should be an iterable derived from the index and element value.\n If no collection_selector is provided, the elements of the\n intermediate sequence will consist of tuples of (index,\n element) from the source sequence.\n\n result_selector:\n An optional binary function mapping the elements in the\n flattened intermediate sequence together with their\n corresponding source elements to elements of the result\n sequence. The two positional arguments of the result_selector\n are, first the source element corresponding to an element from\n the intermediate sequence, and second the actual element from\n the intermediate sequence. The return value should be the\n corresponding value in the result sequence. If no\n result_selector function is provided, the elements of the\n flattened intermediate sequence are returned untransformed.\n\n Returns:\n A Queryable over a generated sequence whose elements are the result\n of applying the one-to-many collection_selector to each element of\n the source sequence which incorporates both the index and value of\n the source element, concatenating the results into an intermediate\n sequence, and then mapping each of those elements through the\n result_selector into the result sequence.\n\n Raises:\n ValueError: If this Queryable has been closed.\n TypeError: If projector [and selector] are not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call select_many_with_index() on a \"\n \"closed Queryable.\")\n\n if not is_callable(collection_selector):\n raise TypeError(\"select_many_with_index() parameter \"\n \"projector={0} is not callable\".format(repr(collection_selector)))\n\n if not is_callable(result_selector):\n raise TypeError(\"select_many_with_index() parameter \"\n \"selector={0} is not callable\".format(repr(result_selector)))\n\n return self._create(\n self._generate_select_many_with_index(collection_selector,\n result_selector))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nproject each element of a sequence to an intermediate new sequence and flattens the resulting sequence into one sequence and uses a selector function to incorporate each corresponding source for each item in the result sequence.", "response": "def select_many_with_correspondence(\n self,\n collection_selector=identity,\n result_selector=KeyedElement):\n '''Projects each element of a sequence to an intermediate new sequence,\n and flattens the resulting sequence, into one sequence and uses a\n selector function to incorporate the corresponding source for each item\n in the result sequence.\n\n Note: This method uses deferred execution.\n\n Args:\n collection_selector: A unary function mapping each element of the\n source iterable into an intermediate sequence. The single\n argument of the collection_selector is the value of an element\n from the source sequence. The return value should be an\n iterable derived from that element value. The default\n collection_selector, which is the identity function, assumes\n that each element of the source sequence is itself iterable.\n\n result_selector:\n An optional binary function mapping the elements in the\n flattened intermediate sequence together with their\n corresponding source elements to elements of the result\n sequence. The two positional arguments of the result_selector\n are, first the source element corresponding to an element from\n the intermediate sequence, and second the actual element from\n the intermediate sequence. The return value should be the\n corresponding value in the result sequence. If no\n result_selector function is provided, the elements of the\n result sequence are KeyedElement namedtuples.\n\n Returns:\n A Queryable over a generated sequence whose elements are the result\n of applying the one-to-many collection_selector to each element of\n the source sequence, concatenating the results into an intermediate\n sequence, and then mapping each of those elements through the\n result_selector which incorporates the corresponding source element\n into the result sequence.\n\n Raises:\n ValueError: If this Queryable has been closed.\n TypeError: If projector or selector are not callable.\n '''\n\n if self.closed():\n raise ValueError(\"Attempt to call \"\n \"select_many_with_correspondence() on a closed Queryable.\")\n\n if not is_callable(collection_selector):\n raise TypeError(\"select_many_with_correspondence() parameter \"\n \"projector={0} is not callable\".format(repr(collection_selector)))\n\n if not is_callable(result_selector):\n raise TypeError(\"select_many_with_correspondence() parameter \"\n \"selector={0} is not callable\".format(repr(result_selector)))\n\n return self._create(\n self._generate_select_many_with_correspondence(collection_selector,\n result_selector))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngroup the elements of the source sequence according to the value of a key extracted by a selector function.", "response": "def group_by(self, key_selector=identity,\n element_selector=identity,\n result_selector=lambda key, grouping: grouping):\n '''Groups the elements according to the value of a key extracted by a\n selector function.\n\n Note: This method has different behaviour to itertools.groupby in the\n Python standard library because it aggregates all items with the\n same key, rather than returning groups of consecutive items of the\n same key.\n\n Note: This method uses deferred execution, but consumption of a single\n result will lead to evaluation of the whole source sequence.\n\n Args:\n key_selector: An optional unary function used to extract a key from\n each element in the source sequence. The default is the\n identity function.\n\n element_selector: A optional unary function to map elements in the\n source sequence to elements in a resulting Grouping. The\n default is the identity function.\n\n result_selector: An optional binary function to create a result\n from each group. The first positional argument is the key\n identifying the group. The second argument is a Grouping object\n containing the members of the group. The default is a function\n which simply returns the Grouping.\n\n Returns:\n A Queryable sequence of elements of the where each element\n represents a group. If the default result_selector is relied upon\n this is a Grouping object.\n\n Raises:\n ValueError: If the Queryable is closed().\n TypeError: If key_selector is not callable.\n TypeError: If element_selector is not callable.\n TypeError: If result_selector is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call group_by() on a closed \"\n \"Queryable.\")\n\n if not is_callable(key_selector):\n raise TypeError(\"group_by() parameter key_selector={0} is not \"\n \"callable\".format(repr(key_selector)))\n\n if not is_callable(element_selector):\n raise TypeError(\"group_by() parameter element_selector={0} is not \"\n \"callable\".format(repr(element_selector)))\n\n if not is_callable(result_selector):\n raise TypeError(\"group_by() parameter result_selector={0} is not \"\n \"callable\".format(repr(result_selector)))\n\n return self._create(self._generate_group_by_result(key_selector,\n element_selector, result_selector))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef where(self, predicate):\n '''Filters elements according to whether they match a predicate.\n\n Note: This method uses deferred execution.\n\n Args:\n predicate: A unary function which is applied to each element in the\n source sequence. Source elements for which the predicate\n returns True will be present in the result.\n\n Returns:\n A Queryable over those elements of the source sequence for which\n the predicate is True.\n\n Raises:\n ValueError: If the Queryable is closed.\n TypeError: If the predicate is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call where() on a closed Queryable.\")\n\n if not is_callable(predicate):\n raise TypeError(\"where() parameter predicate={predicate} is not \"\n \"callable\".format(predicate=repr(predicate)))\n\n return self._create(ifilter(predicate, self))", "response": "Returns a new Queryable with the elements from which the predicate returns True."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef of_type(self, classinfo):\n '''Filters elements according to whether they are of a certain type.\n\n Note: This method uses deferred execution.\n\n Args:\n classinfo: If classinfo is neither a class object nor a type object\n it may be a tuple of class or type objects, or may recursively\n contain other such tuples (other sequence types are not\n accepted).\n\n Returns:\n A Queryable over those elements of the source sequence for which\n the predicate is True.\n\n Raises:\n ValueError: If the Queryable is closed.\n TypeError: If classinfo is not a class, type, or tuple of classes,\n types, and such tuples.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call of_type() on a closed \"\n \"Queryable.\")\n\n if not is_type(classinfo):\n raise TypeError(\"of_type() parameter classinfo={0} is not a class \"\n \"object or a type objector a tuple of class or \"\n \"type objects.\".format(classinfo))\n\n return self.where(lambda x: isinstance(x, classinfo))", "response": "Returns a new Queryable object containing only the elements of the given classinfo."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef order_by(self, key_selector=identity):\n '''Sorts by a key in ascending order.\n\n Introduces a primary sorting order to the sequence. Additional sort\n criteria should be specified by subsequent calls to then_by() and\n then_by_descending(). Calling order_by() or order_by_descending() on\n the results of a call to order_by() will introduce a new primary\n ordering which will override any already established ordering.\n\n This method performs a stable sort. The order of two elements with the\n same key will be preserved.\n\n Note: This method uses deferred execution.\n\n Args:\n key_selector: A unary function which extracts a key from each\n element using which the result will be ordered.\n\n Returns:\n An OrderedQueryable over the sorted elements.\n\n Raises:\n ValueError: If the Queryable is closed.\n TypeError: If the key_selector is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call order_by() on a \"\n \"closed Queryable.\")\n\n if not is_callable(key_selector):\n raise TypeError(\"order_by() parameter key_selector={key_selector} \"\n \"is not callable\".format(key_selector=repr(key_selector)))\n\n return self._create_ordered(iter(self), -1, key_selector)", "response": "Sorts the elements of the elements in ascending order."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef take(self, count=1):\n '''Returns a specified number of elements from the start of a sequence.\n\n If the source sequence contains fewer elements than requested only the\n available elements will be returned and no exception will be raised.\n\n Note: This method uses deferred execution.\n\n Args:\n count: An optional number of elements to take. The default is one.\n\n Returns:\n A Queryable over the first count elements of the source sequence,\n or the all elements of elements in the source, whichever is fewer.\n\n Raises:\n ValueError: If the Queryable is closed()\n '''\n if self.closed():\n raise ValueError(\"Attempt to call take() on a closed Queryable.\")\n\n count = max(0, count)\n\n return self._create(itertools.islice(self, count))", "response": "Returns a specified number of elements from the start of a sequence."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the elements from the beginning of the source sequence for which the predicate returns True or False.", "response": "def take_while(self, predicate):\n '''Returns elements from the start while the predicate is True.\n\n Note: This method uses deferred execution.\n\n Args:\n predicate: A function returning True or False with which elements\n will be tested.\n\n Returns:\n A Queryable over the elements from the beginning of the source\n sequence for which predicate is True.\n\n Raises:\n ValueError: If the Queryable is closed()\n TypeError: If the predicate is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call take_while() on a closed \"\n \"Queryable.\")\n\n if not is_callable(predicate):\n raise TypeError(\"take_while() parameter predicate={0} is \"\n \"not callable\".format(repr(predicate)))\n\n # Cannot use itertools.takewhile here because it is not lazy\n return self._create(self._generate_take_while_result(predicate))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef skip(self, count=1):\n '''Skip the first count contiguous elements of the source sequence.\n\n If the source sequence contains fewer than count elements returns an\n empty sequence and does not raise an exception.\n\n Note: This method uses deferred execution.\n\n Args:\n count: The number of elements to skip from the beginning of the\n sequence. If omitted defaults to one. If count is less than one\n the result sequence will be empty.\n\n Returns:\n A Queryable over the elements of source excluding the first count\n elements.\n\n Raises:\n ValueError: If the Queryable is closed().\n '''\n if self.closed():\n raise ValueError(\"Attempt to call skip() on a closed Queryable.\")\n\n count = max(0, count)\n\n if count == 0:\n return self\n\n # Try an optimised version\n if hasattr(self._iterable, \"__getitem__\"):\n try:\n stop = len(self._iterable)\n return self._create(self._generate_optimized_skip_result(count,\n stop))\n except TypeError:\n pass\n\n # Fall back to the unoptimized version\n return self._create(self._generate_skip_result(count))", "response": "Skip the first count contiguous elements of the source sequence."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef skip_while(self, predicate):\n '''Omit elements from the start for which a predicate is True.\n\n Note: This method uses deferred execution.\n\n Args:\n predicate: A single argument predicate function.\n\n Returns:\n A Queryable over the sequence of elements beginning with the first\n element for which the predicate returns False.\n\n Raises:\n ValueError: If the Queryable is closed().\n TypeError: If predicate is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call take_while() on a \"\n \"closed Queryable.\")\n\n if not is_callable(predicate):\n raise TypeError(\"skip_while() parameter predicate={0} is \"\n \"not callable\".format(repr(predicate)))\n\n return self._create(itertools.dropwhile(predicate, self))", "response": "Return a new Queryable with elements from the start for which a predicate returns True."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconcatenating two sequences. Note: This method uses deferred execution. Args: second_iterable: The sequence to concatenate on to the sequence. Returns: A Queryable over the concatenated sequences. Raises: ValueError: If the Queryable is closed(). TypeError: If second_iterable is not in fact iterable.", "response": "def concat(self, second_iterable):\n '''Concatenates two sequences.\n\n Note: This method uses deferred execution.\n\n Args:\n second_iterable: The sequence to concatenate on to the sequence.\n\n Returns:\n A Queryable over the concatenated sequences.\n\n Raises:\n ValueError: If the Queryable is closed().\n TypeError: If second_iterable is not in fact iterable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call concat() on a closed Queryable.\")\n\n if not is_iterable(second_iterable):\n raise TypeError(\"Cannot compute concat() with second_iterable of \"\n \"non-iterable {0}\".format(str(type(second_iterable))[7: -1]))\n\n return self._create(itertools.chain(self, second_iterable))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef reverse(self):\n '''Returns the sequence reversed.\n\n Note: This method uses deferred execution, but the whole source\n sequence is consumed once execution commences.\n\n Returns:\n The source sequence in reverse order.\n\n Raises:\n ValueError: If the Queryable is closed().\n '''\n if self.closed():\n raise ValueError(\"Attempt to call reverse() on a \"\n \"closed Queryable.\")\n\n # Attempt an optimised version\n try:\n r = reversed(self._iterable)\n return self._create(r)\n except TypeError:\n pass\n\n # Fall through to a sequential version\n return self._create(self._generate_reverse_result())", "response": "Returns the sequence reversed."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the element at the specified index in the source sequence.", "response": "def element_at(self, index):\n '''Return the element at ordinal index.\n\n Note: This method uses immediate execution.\n\n Args:\n index: The index of the element to be returned.\n\n Returns:\n The element at ordinal index in the source sequence.\n\n Raises:\n ValueError: If the Queryable is closed().\n ValueError: If index is out of range.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call element_at() on a \"\n \"closed Queryable.\")\n\n if index < 0:\n raise OutOfRangeError(\"Attempt to use negative index.\")\n\n # Attempt to use __getitem__\n try:\n return self._iterable[index]\n except IndexError:\n raise OutOfRangeError(\"Index out of range.\")\n except TypeError:\n pass\n\n # Fall back to iterating\n for i, item in enumerate(self):\n if i == index:\n return item\n raise OutOfRangeError(\"element_at(index) out of range.\")"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef count(self, predicate=None):\n '''Return the number of elements (which match an optional predicate).\n\n Note: This method uses immediate execution.\n\n Args:\n predicate: An optional unary predicate function used to identify\n elements which will be counted. The single positional argument\n of the function is the element value. The function should\n return True or False.\n\n Returns:\n The number of elements in the sequence if the predicate is None\n (the default), or if the predicate is supplied the number of\n elements for which the predicate evaluates to True.\n\n Raises:\n ValueError: If the Queryable is closed().\n TypeError: If predicate is neither None nor a callable.\n '''\n\n if self.closed():\n raise ValueError(\"Attempt to call element_at() on a \"\n \"closed Queryable.\")\n\n return self._count() if predicate is None else self._count_predicate(predicate)", "response": "Return the number of elements in the sequence that satisfy the predicate."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef any(self, predicate=None):\n '''Determine if the source sequence contains any elements which satisfy\n the predicate.\n\n Only enough of the sequence to satisfy the predicate once is consumed.\n\n Note: This method uses immediate execution.\n\n Args:\n predicate: An optional single argument function used to test each\n element. If omitted, or None, this method returns True if there\n is at least one element in the source.\n\n Returns:\n True if the sequence contains at least one element which satisfies\n the predicate, otherwise False.\n\n Raises:\n ValueError: If the Queryable is closed()\n '''\n if self.closed():\n raise ValueError(\"Attempt to call any() on a closed Queryable.\")\n\n if predicate is None:\n predicate = lambda x: True\n\n if not is_callable(predicate):\n raise TypeError(\"any() parameter predicate={predicate} is not callable\".format(predicate=repr(predicate)))\n\n for item in self.select(predicate):\n if item:\n return True\n return False", "response": "Determines if the source sequence contains any elements which satisfy the predicate."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef all(self, predicate=bool):\n '''Determine if all elements in the source sequence satisfy a condition.\n\n All of the source sequence will be consumed.\n\n Note: This method uses immediate execution.\n\n Args:\n predicate (callable): An optional single argument function used to\n test each elements. If omitted, the bool() function is used\n resulting in the elements being tested directly.\n\n Returns:\n True if all elements in the sequence meet the predicate condition,\n otherwise False.\n\n Raises:\n ValueError: If the Queryable is closed()\n TypeError: If predicate is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call all() on a closed Queryable.\")\n\n if not is_callable(predicate):\n raise TypeError(\"all() parameter predicate={0} is \"\n \"not callable\".format(repr(predicate)))\n\n return all(self.select(predicate))", "response": "Determine if all elements in the source sequence satisfy a condition."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the arithmetic sum of the values in the sequence.", "response": "def sum(self, selector=identity):\n '''Return the arithmetic sum of the values in the sequence..\n\n All of the source sequence will be consumed.\n\n Note: This method uses immediate execution.\n\n Args:\n selector: An optional single argument function which will be used\n to project the elements of the sequence. If omitted, the\n identity function is used.\n\n Returns:\n The total value of the projected sequence, or zero for an empty\n sequence.\n\n Raises:\n ValueError: If the Queryable has been closed.\n '''\n\n if self.closed():\n raise ValueError(\"Attempt to call sum() on a closed Queryable.\")\n\n if not is_callable(selector):\n raise TypeError(\"sum() parameter selector={0} is \"\n \"not callable\".format(repr(selector)))\n\n return sum(self.select(selector))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef average(self, selector=identity):\n '''Return the arithmetic mean of the values in the sequence..\n\n All of the source sequence will be consumed.\n\n Note: This method uses immediate execution.\n\n Args:\n selector: An optional single argument function which will be used\n to project the elements of the sequence. If omitted, the\n identity function is used.\n\n Returns:\n The arithmetic mean value of the projected sequence.\n\n Raises:\n ValueError: If the Queryable has been closed.\n ValueError: I the source sequence is empty.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call average() on a \"\n \"closed Queryable.\")\n\n if not is_callable(selector):\n raise TypeError(\"average() parameter selector={0} is \"\n \"not callable\".format(repr(selector)))\n\n total = 0\n count = 0\n for item in self.select(selector):\n total += item\n count += 1\n if count == 0:\n raise ValueError(\"Cannot compute average() of an empty sequence.\")\n return total / count", "response": "Return the arithmetic mean of the values in the sequence."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef contains(self, value, equality_comparer=operator.eq):\n '''Determines whether the sequence contains a particular value.\n\n Execution is immediate. Depending on the type of the sequence, all or\n none of the sequence may be consumed by this operation.\n\n Note: This method uses immediate execution.\n\n Args:\n value: The value to test for membership of the sequence\n\n Returns:\n True if value is in the sequence, otherwise False.\n\n Raises:\n ValueError: If the Queryable has been closed.\n\n '''\n if self.closed():\n raise ValueError(\"Attempt to call contains() on a \"\n \"closed Queryable.\")\n\n if not is_callable(equality_comparer):\n raise TypeError(\"contains() parameter equality_comparer={0} is \"\n \"not callable\".format(repr(equality_comparer)))\n\n if equality_comparer is operator.eq:\n return value in self._iterable\n\n for item in self:\n if equality_comparer(value, item):\n return True\n return False", "response": "Determines whether the sequence contains a particular value."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef default_if_empty(self, default):\n '''If the source sequence is empty return a single element sequence\n containing the supplied default value, otherwise return the source\n sequence unchanged.\n\n Note: This method uses deferred execution.\n\n Args:\n default: The element to be returned if the source sequence is empty.\n\n Returns:\n The source sequence, or if the source sequence is empty an sequence\n containing a single element with the supplied default value.\n\n Raises:\n ValueError: If the Queryable has been closed.\n '''\n\n if self.closed():\n raise ValueError(\"Attempt to call default_if_empty() on a \"\n \"closed Queryable.\")\n\n return self._create(self._generate_default_if_empty_result(default))", "response": "Returns a new sequence containing the supplied default value if the source sequence is empty."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef distinct(self, selector=identity):\n '''Eliminate duplicate elements from a sequence.\n\n Note: This method uses deferred execution.\n\n Args:\n selector: An optional single argument function the result of which\n is the value compared for uniqueness against elements already\n consumed. If omitted, the element value itself is compared for\n uniqueness.\n\n Returns:\n Unique elements of the source sequence as determined by the\n selector function. Note that it is unprojected elements that are\n returned, even if a selector was provided.\n\n Raises:\n ValueError: If the Queryable is closed.\n TypeError: If the selector is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call distinct() on a \"\n \"closed Queryable.\")\n\n if not is_callable(selector):\n raise TypeError(\"distinct() parameter selector={0} is \"\n \"not callable\".format(repr(selector)))\n\n return self._create(self._generate_distinct_result(selector))", "response": "Eliminate duplicate elements from a sequence."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef difference(self, second_iterable, selector=identity):\n '''Returns those elements which are in the source sequence which are not\n in the second_iterable.\n\n This method is equivalent to the Except() LINQ operator, renamed to a\n valid Python identifier.\n\n Note: This method uses deferred execution, but as soon as execution\n commences the entirety of the second_iterable is consumed;\n therefore, although the source sequence may be infinite the\n second_iterable must be finite.\n\n Args:\n second_iterable: Elements from this sequence are excluded from the\n returned sequence. This sequence will be consumed in its\n entirety, so must be finite.\n\n selector: A optional single argument function with selects from the\n elements of both sequences the values which will be\n compared for equality. If omitted the identity function will\n be used.\n\n Returns:\n A sequence containing all elements in the source sequence except\n those which are also members of the second sequence.\n\n Raises:\n ValueError: If the Queryable has been closed.\n TypeError: If the second_iterable is not in fact iterable.\n TypeError: If the selector is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call difference() on a \"\n \"closed Queryable.\")\n\n if not is_iterable(second_iterable):\n raise TypeError(\"Cannot compute difference() with second_iterable\"\n \"of non-iterable {0}\".format(str(type(second_iterable))[7: -2]))\n\n if not is_callable(selector):\n raise TypeError(\"difference() parameter selector={0} is \"\n \"not callable\".format(repr(selector)))\n\n return self._create(self._generate_difference_result(second_iterable,\n selector))", "response": "Returns the elements which are not in the source sequence except the ones in the second_iterable."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning those elements which are both in the source sequence and in the second_iterable.", "response": "def intersect(self, second_iterable, selector=identity):\n '''Returns those elements which are both in the source sequence and in\n the second_iterable.\n\n Note: This method uses deferred execution.\n\n Args:\n second_iterable: Elements are returned if they are also in the\n sequence.\n\n selector: An optional single argument function which is used to\n project the elements in the source and second_iterables prior\n to comparing them. If omitted the identity function will be\n used.\n\n Returns:\n A sequence containing all elements in the source sequence which\n are also members of the second sequence.\n\n Raises:\n ValueError: If the Queryable has been closed.\n TypeError: If the second_iterable is not in fact iterable.\n TypeError: If the selector is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call intersect() on a \"\n \"closed Queryable.\")\n\n if not is_iterable(second_iterable):\n raise TypeError(\"Cannot compute intersect() with second_iterable \"\n \"of non-iterable {0}\".format(str(type(second_iterable))[7: -1]))\n\n if not is_callable(selector):\n raise TypeError(\"intersect() parameter selector={0} is \"\n \"not callable\".format(repr(selector)))\n\n return self._create(self._generate_intersect_result(second_iterable,\n selector))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef union(self, second_iterable, selector=identity):\n '''Returns those elements which are either in the source sequence or in\n the second_iterable, or in both.\n\n Note: This method uses deferred execution.\n\n Args:\n second_iterable: Elements from this sequence are returns if they\n are not also in the source sequence.\n\n selector: An optional single argument function which is used to\n project the elements in the source and second_iterables prior\n to comparing them. If omitted the identity function will be\n used.\n\n Returns:\n A sequence containing all elements in the source sequence and second\n sequence.\n\n Raises:\n ValueError: If the Queryable has been closed.\n TypeError: If the second_iterable is not in fact iterable.\n TypeError: If the selector is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call union() on a closed Queryable.\")\n\n if not is_iterable(second_iterable):\n raise TypeError(\"Cannot compute union() with second_iterable of \"\n \"non-iterable {0}\".format(str(type(second_iterable))[7: -1]))\n\n return self._create(itertools.chain(self, second_iterable)).distinct(selector)", "response": "Returns the elements which are either in the source sequence or in the second_iterable or in both."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef join(self, inner_iterable, outer_key_selector=identity,\n inner_key_selector=identity,\n result_selector=lambda outer, inner: (outer, inner)):\n '''Perform an inner join with a second sequence using selected keys.\n\n The order of elements from outer is maintained. For each of these the\n order of elements from inner is also preserved.\n\n Note: This method uses deferred execution.\n\n Args:\n inner_iterable: The sequence to join with the outer sequence.\n\n outer_key_selector: An optional unary function to extract keys from\n elements of the outer (source) sequence. The first positional\n argument of the function should accept outer elements and the\n result value should be the key. If omitted, the identity\n function is used.\n\n inner_key_selector: An optional unary function to extract keys\n from elements of the inner_iterable. The first positional\n argument of the function should accept outer elements and the\n result value should be the key. If omitted, the identity\n function is used.\n\n result_selector: An optional binary function to create a result\n element from two matching elements of the outer and inner. If\n omitted the result elements will be a 2-tuple pair of the\n matching outer and inner elements.\n\n Returns:\n A Queryable whose elements are the result of performing an inner-\n join on two sequences.\n\n Raises:\n ValueError: If the Queryable has been closed.\n TypeError: If the inner_iterable is not in fact iterable.\n TypeError: If the outer_key_selector is not callable.\n TypeError: If the inner_key_selector is not callable.\n TypeError: If the result_selector is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call join() on a closed Queryable.\")\n\n if not is_iterable(inner_iterable):\n raise TypeError(\"Cannot compute join() with inner_iterable of \"\n \"non-iterable {0}\".format(str(type(inner_iterable))[7: -1]))\n\n if not is_callable(outer_key_selector):\n raise TypeError(\"join() parameter outer_key_selector={0} is not \"\n \"callable\".format(repr(outer_key_selector)))\n\n if not is_callable(inner_key_selector):\n raise TypeError(\"join() parameter inner_key_selector={0} is not \"\n \"callable\".format(repr(inner_key_selector)))\n\n if not is_callable(result_selector):\n raise TypeError(\"join() parameter result_selector={0} is not \"\n \"callable\".format(repr(result_selector)))\n\n return self._create(self._generate_join_result(inner_iterable, outer_key_selector,\n inner_key_selector, result_selector))", "response": "Perform an inner join with a second sequence using selected keys."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef group_join(self, inner_iterable, outer_key_selector=identity, inner_key_selector=identity,\n result_selector=lambda outer, grouping: grouping):\n '''Match elements of two sequences using keys and group the results.\n\n The group_join() query produces a hierarchical result, with all of the\n inner elements in the result grouped against the matching outer\n element.\n\n The order of elements from outer is maintained. For each of these the\n order of elements from inner is also preserved.\n\n Note: This method uses deferred execution.\n\n Args:\n inner_iterable: The sequence to join with the outer sequence.\n\n outer_key_selector: An optional unary function to extract keys from\n elements of the outer (source) sequence. The first positional\n argument of the function should accept outer elements and the\n result value should be the key. If omitted, the identity\n function is used.\n\n inner_key_selector: An optional unary function to extract keys\n from elements of the inner_iterable. The first positional\n argument of the function should accept outer elements and the\n result value should be the key. If omitted, the identity\n function is used.\n\n result_selector: An optional binary function to create a result\n element from an outer element and the Grouping of matching\n inner elements. The first positional argument is the outer\n elements and the second in the Grouping of inner elements\n which match the outer element according to the key selectors\n used. If omitted, the result elements will be the Groupings\n directly.\n\n Returns:\n A Queryable over a sequence with one element for each group in the\n result as returned by the result_selector. If the default result\n selector is used, the result is a sequence of Grouping objects.\n\n Raises:\n ValueError: If the Queryable has been closed.\n TypeError: If the inner_iterable is not in fact iterable.\n TypeError: If the outer_key_selector is not callable.\n TypeError: If the inner_key_selector is not callable.\n TypeError: If the result_selector is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call group_join() on a closed Queryable.\")\n\n if not is_iterable(inner_iterable):\n raise TypeError(\"Cannot compute group_join() with inner_iterable of non-iterable {type}\".format(\n type=str(type(inner_iterable))[7: -1]))\n\n if not is_callable(outer_key_selector):\n raise TypeError(\"group_join() parameter outer_key_selector={outer_key_selector} is not callable\".format(\n outer_key_selector=repr(outer_key_selector)))\n\n if not is_callable(inner_key_selector):\n raise TypeError(\"group_join() parameter inner_key_selector={inner_key_selector} is not callable\".format(\n inner_key_selector=repr(inner_key_selector)))\n\n if not is_callable(result_selector):\n raise TypeError(\"group_join() parameter result_selector={result_selector} is not callable\".format(\n result_selector=repr(result_selector)))\n\n return self._create(self._generate_group_join_result(inner_iterable, outer_key_selector,\n inner_key_selector, result_selector))", "response": "Match elements of two sequences using keys and group the results."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef first(self, predicate=None):\n '''The first element in a sequence (optionally satisfying a predicate).\n\n If the predicate is omitted or is None this query returns the first\n element in the sequence; otherwise, it returns the first element in\n the sequence for which the predicate evaluates to True. Exceptions are\n raised if there is no such element.\n\n Note: This method uses immediate execution.\n\n Args:\n predicate: An optional unary predicate function, the only argument\n to which is the element. The return value should be True for\n matching elements, otherwise False. If the predicate is\n omitted or None the first element of the source sequence will\n be returned.\n\n Returns:\n The first element of the sequence if predicate is None, otherwise\n the first element for which the predicate returns True.\n\n Raises:\n ValueError: If the Queryable is closed.\n ValueError: If the source sequence is empty.\n ValueError: If there are no elements matching the predicate.\n TypeError: If the predicate is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call first() on a closed Queryable.\")\n\n return self._first() if predicate is None else self._first_predicate(predicate)", "response": "Returns the first element in a sequence satisfying a predicate."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef first_or_default(self, default, predicate=None):\n '''The first element (optionally satisfying a predicate) or a default.\n\n If the predicate is omitted or is None this query returns the first\n element in the sequence; otherwise, it returns the first element in\n the sequence for which the predicate evaluates to True. If there is no\n such element the value of the default argument is returned.\n\n Note: This method uses immediate execution.\n\n Args:\n default: The value which will be returned if either the sequence is\n empty or there are no elements matching the predicate.\n\n predicate: An optional unary predicate function, the only argument\n to which is the element. The return value should be True for\n matching elements, otherwise False. If the predicate is\n omitted or None the first element of the source sequence will\n be returned.\n\n Returns:\n The first element of the sequence if predicate is None, otherwise\n the first element for which the predicate returns True. If there is\n no such element, the default argument is returned.\n\n Raises:\n ValueError: If the Queryable is closed.\n TypeError: If the predicate is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call first_or_default() on a \"\n \"closed Queryable.\")\n\n return self._first_or_default(default) if predicate is None else self._first_or_default_predicate(default, predicate)", "response": "Returns the first element of the sequence or a default."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the only element in the source sequence for which the predicate evaluates to True.", "response": "def single(self, predicate=None):\n '''The only element (which satisfies a condition).\n\n If the predicate is omitted or is None this query returns the only\n element in the sequence; otherwise, it returns the only element in\n the sequence for which the predicate evaluates to True. Exceptions are\n raised if there is either no such element or more than one such\n element.\n\n Note: This method uses immediate execution.\n\n Args:\n predicate: An optional unary predicate function, the only argument\n to which is the element. The return value should be True for\n matching elements, otherwise False. If the predicate is\n omitted or None the only element of the source sequence will\n be returned.\n\n Returns:\n The only element of the sequence if predicate is None, otherwise\n the only element for which the predicate returns True.\n\n Raises:\n ValueError: If the Queryable is closed.\n ValueError: If, when predicate is None the source sequence contains\n more than one element.\n ValueError: If there are no elements matching the predicate or more\n then one element matching the predicate.\n TypeError: If the predicate is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call single() on a closed Queryable.\")\n\n return self._single() if predicate is None else self._single_predicate(predicate)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the only element in the sequence or a default value if the predicate evaluates to True.", "response": "def single_or_default(self, default, predicate=None):\n '''The only element (which satisfies a condition) or a default.\n\n If the predicate is omitted or is None this query returns the only\n element in the sequence; otherwise, it returns the only element in\n the sequence for which the predicate evaluates to True. A default value\n is returned if there is no such element. An exception is raised if\n there is more than one such element.\n\n Note: This method uses immediate execution.\n\n Args:\n default: The value which will be returned if either the sequence is\n empty or there are no elements matching the predicate.\n\n predicate: An optional unary predicate function, the only argument\n to which is the element. The return value should be True for\n matching elements, otherwise False. If the predicate is\n omitted or None the only element of the source sequence will\n be returned.\n\n Returns:\n The only element of the sequence if predicate is None, otherwise\n the only element for which the predicate returns True. If there are\n no such elements the default value will returned.\n\n Raises:\n ValueError: If the Queryable is closed.\n ValueError: If, when predicate is None the source sequence contains\n more than one element.\n ValueError: If there is more then one element matching the\n predicate.\n TypeError: If the predicate is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call single_or_default() on a closed Queryable.\")\n\n return self._single_or_default(default) if predicate is None else self._single_or_default_predicate(default, predicate)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the last element in a sequence for which the predicate evaluates to True.", "response": "def last(self, predicate=None):\n '''The last element in a sequence (optionally satisfying a predicate).\n\n If the predicate is omitted or is None this query returns the last\n element in the sequence; otherwise, it returns the last element in\n the sequence for which the predicate evaluates to True. Exceptions are\n raised if there is no such element.\n\n Note: This method uses immediate execution.\n\n Args:\n predicate: An optional unary predicate function, the only argument\n to which is the element. The return value should be True for\n matching elements, otherwise False. If the predicate is\n omitted or None the last element of the source sequence will\n be returned.\n\n Returns:\n The last element of the sequence if predicate is None, otherwise\n the last element for which the predicate returns True.\n\n Raises:\n ValueError: If the Queryable is closed.\n ValueError: If the source sequence is empty.\n ValueError: If there are no elements matching the predicate.\n TypeError: If the predicate is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call last() on a closed Queryable.\")\n\n return self._last() if predicate is None else self._last_predicate(predicate)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\napplying a function over a sequence to produce a single result.", "response": "def aggregate(self, reducer, seed=default, result_selector=identity):\n '''Apply a function over a sequence to produce a single result.\n\n Apply a binary function cumulatively to the elements of the source\n sequence so as to reduce the iterable to a single value.\n\n Note: This method uses immediate execution.\n\n Args:\n reducer: A binary function the first positional argument of which\n is an accumulated value and the second is the update value from\n the source sequence. The return value should be the new\n accumulated value after the update value has been incorporated.\n\n seed: An optional value used to initialise the accumulator before\n iteration over the source sequence. If seed is omitted the\n and the source sequence contains only one item, then that item\n is returned.\n\n result_selector: An optional unary function applied to the final\n accumulator value to produce the result. If omitted, defaults\n to the identity function.\n\n Raises:\n ValueError: If called on an empty sequence with no seed value.\n TypeError: If reducer is not callable.\n TypeError: If result_selector is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call aggregate() on a \"\n \"closed Queryable.\")\n\n if not is_callable(reducer):\n raise TypeError(\"aggregate() parameter reducer={0} is \"\n \"not callable\".format(repr(reducer)))\n\n if not is_callable(result_selector):\n raise TypeError(\"aggregate() parameter result_selector={0} is \"\n \"not callable\".format(repr(result_selector)))\n\n if seed is default:\n try:\n return result_selector(fold(reducer, self))\n except TypeError as e:\n if 'empty sequence' in str(e):\n raise ValueError(\"Cannot aggregate() empty sequence with \"\n \"no seed value\")\n return result_selector(fold(reducer, self, seed))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef to_list(self):\n '''Convert the source sequence to a list.\n\n Note: This method uses immediate execution.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call to_list() on a closed Queryable.\")\n\n # Maybe use with closable(self) construct to achieve this.\n if isinstance(self._iterable, list):\n return self._iterable\n lst = list(self)\n # Ideally we would close here. Why can't we - what is the problem?\n #self.close()\n return lst", "response": "Convert the source sequence to a list."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert the source sequence to a tuple.", "response": "def to_tuple(self):\n '''Convert the source sequence to a tuple.\n\n Note: This method uses immediate execution.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call to_tuple() on a closed Queryable.\")\n\n if isinstance(self._iterable, tuple):\n return self._iterable\n tup = tuple(self)\n # Ideally we would close here\n #self.close()\n return tup"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef to_set(self):\n '''Convert the source sequence to a set.\n\n Note: This method uses immediate execution.\n\n Raises:\n ValueError: If duplicate keys are in the projected source sequence.\n ValueError: If the Queryable is closed().\n '''\n if self.closed():\n raise ValueError(\"Attempt to call to_set() on a closed Queryable.\")\n\n if isinstance(self._iterable, set):\n return self._iterable\n s = set()\n for item in self:\n if item in s:\n raise ValueError(\"Duplicate item value {0} in sequence \"\n \"during to_set()\".format(repr(item)))\n s.add(item)\n # Ideally we would close here\n #self.close()\n return s", "response": "Convert the source sequence to a set."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a Lookup object using the provided selector to generate a key for each item.", "response": "def to_lookup(self, key_selector=identity, value_selector=identity):\n '''Returns a Lookup object, using the provided selector to generate a\n key for each item.\n\n Note: This method uses immediate execution.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call to_lookup() on a closed Queryable.\")\n\n if not is_callable(key_selector):\n raise TypeError(\"to_lookup() parameter key_selector={key_selector} is not callable\".format(\n key_selector=repr(key_selector)))\n\n if not is_callable(value_selector):\n raise TypeError(\"to_lookup() parameter value_selector={value_selector} is not callable\".format(\n value_selector=repr(value_selector)))\n\n key_value_pairs = self.select(lambda item: (key_selector(item), value_selector(item)))\n lookup = Lookup(key_value_pairs)\n # Ideally we would close here\n #self.close()\n return lookup"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef to_dictionary(self, key_selector=identity, value_selector=identity):\n if self.closed():\n raise ValueError(\"Attempt to call to_dictionary() on a closed Queryable.\")\n\n if not is_callable(key_selector):\n raise TypeError(\"to_dictionary() parameter key_selector={key_selector} is not callable\".format(\n key_selector=repr(key_selector)))\n\n if not is_callable(value_selector):\n raise TypeError(\"to_dictionary() parameter value_selector={value_selector} is not callable\".format(\n value_selector=repr(value_selector)))\n\n dictionary = {}\n for key, value in self.select(lambda x: (key_selector(x), value_selector(x))):\n dictionary[key] = value\n return dictionary", "response": "Builds a dictionary from the source sequence."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a string from the source sequence.", "response": "def to_str(self, separator=''):\n '''Build a string from the source sequence.\n\n The elements of the query result will each coerced to a string and then\n the resulting strings concatenated to return a single string. This\n allows the natural processing of character sequences as strings. An\n optional separator which will be inserted between each item may be\n specified.\n\n Note: this method uses immediate execution.\n\n Args:\n separator: An optional separator which will be coerced to a string\n and inserted between each source item in the resulting string.\n\n Returns:\n A single string which is the result of stringifying each element\n and concatenating the results into a single string.\n\n Raises:\n TypeError: If any element cannot be coerced to a string.\n TypeError: If the separator cannot be coerced to a string.\n ValueError: If the Queryable is closed.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call to_str() on a closed Queryable.\")\n\n return str(separator).join(self.select(str))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndetermines whether two sequences are equal by elementwise comparison.", "response": "def sequence_equal(self, second_iterable, equality_comparer=operator.eq):\n '''\n Determine whether two sequences are equal by elementwise comparison.\n\n Sequence equality is defined as the two sequences being equal length\n and corresponding elements being equal as determined by the equality\n comparer.\n\n Note: This method uses immediate execution.\n\n Args:\n second_iterable: The sequence which will be compared with the\n source sequence.\n\n equality_comparer: An optional binary predicate function which is\n used to compare corresponding elements. Should return True if\n the elements are equal, otherwise False. The default equality\n comparer is operator.eq which calls __eq__ on elements of the\n source sequence with the corresponding element of the second\n sequence as a parameter.\n\n Returns:\n True if the sequences are equal, otherwise False.\n\n Raises:\n ValueError: If the Queryable is closed.\n TypeError: If second_iterable is not in fact iterable.\n TypeError: If equality_comparer is not callable.\n\n '''\n if self.closed():\n raise ValueError(\"Attempt to call to_tuple() on a closed Queryable.\")\n\n if not is_iterable(second_iterable):\n raise TypeError(\"Cannot compute sequence_equal() with second_iterable of non-iterable {type}\".format(\n type=str(type(second_iterable))[7: -1]))\n\n if not is_callable(equality_comparer):\n raise TypeError(\"aggregate() parameter equality_comparer={equality_comparer} is not callable\".format(\n equality_comparer=repr(equality_comparer)))\n\n # Try to check the lengths directly as an optimization\n try:\n if len(self._iterable) != len(second_iterable):\n return False\n except TypeError:\n pass\n\n sentinel = object()\n for first, second in izip_longest(self, second_iterable, fillvalue=sentinel):\n if first is sentinel or second is sentinel:\n return False\n if not equality_comparer(first, second):\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef log(self, logger=None, label=None, eager=False):\n '''\n Log query result consumption details to a logger.\n\n Args:\n logger: Any object which supports a debug() method which accepts a\n str, such as a Python standard library logger object from the\n logging module. If logger is not provided or is None, this\n method has no logging side effects.\n\n label: An optional label which will be inserted into each line of\n logging output produced by this particular use of log\n\n eager: An optional boolean which controls how the query result will\n be consumed. If True, the sequence will be consumed and logged\n in its entirety. If False (the default) the sequence will be\n evaluated and logged lazily as it consumed.\n\n Warning: Use of eager=True requires use of sufficient memory to\n hold the entire sequence which is obviously not possible with\n infinite sequences. Use with care!\n\n Returns:\n A queryable over the unaltered source sequence.\n\n Raises:\n AttributeError: If logger does not support a debug() method.\n ValueError: If the Queryable has been closed.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call log() on a closed Queryable.\")\n\n if logger is None:\n return self\n\n if label is None:\n label = repr(self)\n\n if eager:\n return self._create(self._eager_log_result(logger, label))\n\n return self._create(self._generate_lazy_log_result(logger, label))", "response": "Log the query result consumption details to a logger."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef scan(self, func=operator.add):\n '''\n An inclusive prefix sum which returns the cumulative application of the\n supplied function up to an including the current element.\n\n Args:\n func: An optional binary function which is commutative - that is,\n the order of the arguments is unimportant. Defaults to a\n summing operator.\n\n Returns:\n A Queryable such that the nth element is the sum of the first n\n elements of the source sequence.\n\n Raises:\n ValueError: If the Queryable has been closed.\n TypeError: If func is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call scan() on a \"\n \"closed Queryable.\")\n\n if not is_callable(func):\n raise TypeError(\"scan() parameter func={0} is \"\n \"not callable\".format(repr(func)))\n\n return self._create(self._generate_scan_result(func))", "response": "Returns a new sequence where the n - th element of the source sequence is the sum of the cumulative application of the n - th element of the source sequence."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef pre_scan(self, func=operator.add, seed=0):\n '''\n An exclusive prefix sum which returns the cumulative application of the\n supplied function up to but excluding the current element.\n\n Args:\n func: An optional binary function which is commutative - that is,\n the order of the arguments is unimportant. Defaults to a\n summing operator.\n\n seed: The first element of the prefix sum and therefore also the\n first element of the returned sequence.\n\n Returns:\n A Queryable such that the nth element is the sum of the first n-1\n elements of the source sequence.\n\n Raises:\n ValueError: If the Queryable has been closed.\n TypeError: If func is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call pre_scan() on a \"\n \"closed Queryable.\")\n\n if not is_callable(func):\n raise TypeError(\"pre_scan() parameter func={0} is \"\n \"not callable\".format(repr(func)))\n\n return self._create(self._generate_pre_scan_result(func, seed))", "response": "Returns a new sequence where the first n - elements of the source sequence are the cumulative application of the function up to but excluding the current element."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nintroducing subsequent ordering to the sequence with an optional key.", "response": "def then_by(self, key_selector=identity):\n '''Introduce subsequent ordering to the sequence with an optional key.\n\n The returned sequence will be sorted in ascending order by the\n selected key.\n\n Note: This method uses deferred execution.\n\n Args:\n key_selector: A unary function the only positional argument to\n which is the element value from which the key will be\n selected. The return value should be the key from that\n element.\n\n Returns:\n An OrderedQueryable over the sorted items.\n\n Raises:\n ValueError: If the OrderedQueryable is closed().\n TypeError: If key_selector is not callable.\n '''\n if self.closed():\n raise ValueError(\"Attempt to call then_by() on a \"\n \"closed OrderedQueryable.\")\n\n if not is_callable(key_selector):\n raise TypeError(\"then_by() parameter key_selector={key_selector} \"\n \"is not callable\".format(key_selector=repr(key_selector)))\n\n self._funcs.append((-1, key_selector))\n return self"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nbuilding a dictionary from the source sequence.", "response": "def to_dictionary(\n self,\n key_selector=lambda item: item.key,\n value_selector=list):\n \"\"\"Build a dictionary from the source sequence.\n\n Args:\n key_selector: A unary callable to extract a key from each item.\n By default the key of the Grouping.\n\n value_selector: A unary callable to extract a value from each item.\n By default the value is the list of items from the Grouping.\n\n Note: This method uses immediate execution.\n\n Raises:\n ValueError: If the Queryable is closed.\n TypeError: If key_selector is not callable.\n TypeError: If value_selector is not callable.\n \"\"\"\n return super(Lookup, self).to_dictionary(key_selector, value_selector)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nbuild a dictionary from the source sequence.", "response": "def to_dictionary(\n self,\n key_selector=None,\n value_selector=None):\n \"\"\"Build a dictionary from the source sequence.\n\n Args:\n key_selector: A unary callable to extract a key from each item or None.\n If None, the default key selector produces a single dictionary key, which\n if the key of this Grouping.\n\n value_selector: A unary callable to extract a value from each item.\n If None, the default value selector produces a list, which contains all\n elements from this Grouping.\n\n Note: This method uses immediate execution.\n\n Raises:\n ValueError: If the Queryable is closed.\n TypeError: If key_selector is not callable.\n TypeError: If value_selector is not callable.\n \"\"\"\n if key_selector is None:\n key_selector = lambda _: self.key\n\n if value_selector is None:\n value_selector = lambda _: self.to_list()\n\n return super(Grouping, self).to_dictionary(key_selector, value_selector)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a Queryable over the specified range of integers.", "response": "def integers(start, count):\r\n '''Generates in sequence the integral numbers within a range.\r\n\r\n Note: This method uses deferred execution.\r\n\r\n Args:\r\n start: The first integer in the sequence.\r\n count: The number of sequential integers to generate.\r\n\r\n Returns:\r\n A Queryable over the specified range of integers.\r\n\r\n Raises:\r\n ValueError: If count is negative.\r\n '''\r\n if count < 0:\r\n raise ValueError(\"integers() count cannot be negative\")\r\n return query(irange(start, start + count))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef repeat(element, count):\r\n '''Generate a sequence with one repeated value.\r\n\r\n Note: This method uses deferred execution.\r\n\r\n Args:\r\n element: The value to be repeated.\r\n count: The number of times to repeat the value.\r\n\r\n Raises:\r\n ValueError: If the count is negative.\r\n '''\r\n if count < 0:\r\n raise ValueError(\"repeat() count cannot be negative\")\r\n return query(itertools.repeat(element, count))", "response": "Generate a sequence with one repeated value."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef geometric_partitions(iterable, floor=1, ceiling=32768):\r\n '''\r\n Partition an iterable into chunks. Returns an iterator over partitions.\r\n '''\r\n partition_size = floor\r\n run_length = multiprocessing.cpu_count()\r\n run_count = 0\r\n\r\n try:\r\n while True:\r\n #print(\"partition_size =\", partition_size)\r\n # Split the iterable and replace the original iterator to avoid\r\n # advancing it\r\n partition, iterable = itertools.tee(iterable)\r\n\r\n # Yield the first partition, limited to the partition size\r\n yield Queryable(partition).take(partition_size)\r\n\r\n # Advance to the start of the next partition, this will raise\r\n # StopIteration if the iterator is exhausted\r\n for i in range(partition_size):\r\n next(iterable)\r\n\r\n # If we've reached the end of a run of this size, double the\r\n # partition size\r\n run_count += 1\r\n if run_count >= run_length:\r\n partition_size *= 2\r\n run_count = 0\r\n\r\n # Unless we have hit the ceiling\r\n if partition_size > ceiling:\r\n partition_size = ceiling\r\n\r\n except StopIteration:\r\n pass", "response": "Returns an iterator over partitions."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ntransforms each element of a sequence into a new form. Each element is transformed through a selector function to produce a new value for each value in the source sequence. Each element is transformed through a selector function to produce a new value for each value in the source sequence. The generated sequence is lazily evaluated.", "response": "def select(self, selector):\r\n '''Transforms each element of a sequence into a new form.\r\n\r\n Each element is transformed through a selector function to produce a\r\n value for each value in the source sequence. The generated sequence is\r\n lazily evaluated.\r\n\r\n Args:\r\n selector: A unary function mapping a value in the source sequence\r\n to the corresponding value in the generated generated sequence.\r\n The argument of the selector function (which can have any name)\r\n is,\r\n\r\n Args:\r\n element: The value of the element\r\n\r\n Returns:\r\n The selected value derived from the element value\r\n\r\n Returns:\r\n A generated sequence whose elements are the result of invoking the\r\n selector function on each element of the source sequence.\r\n '''\r\n return self._create(self._pool.imap_unordered(selector, iter(self),\r\n self._chunksize))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ntransforms each element of a sequence into a new form incorporating the index of the element.", "response": "def select_with_index(self, selector):\r\n '''Transforms each element of a sequence into a new form, incorporating\r\n the index of the element.\r\n\r\n Each element is transformed through a selector function which accepts\r\n the element value and its zero-based index in the source sequence. The\r\n generated sequence is lazily evaluated.\r\n\r\n Args:\r\n selector: A two argument function mapping the index of a value in\r\n the source sequence and the element value itself to the\r\n corresponding value in the generated sequence. The two\r\n arguments of the selector function (which can have any names)\r\n and its return value are,\r\n\r\n Args:\r\n index: The zero-based index of the element\r\n element: The value of the element\r\n\r\n Returns:\r\n The selected value derived from the index and element\r\n\r\n Returns:\r\n A generated sequence whose elements are the result of invoking the\r\n selector function on each element of the source sequence\r\n '''\r\n\r\n return self._create(self._pool.imap_unordered(star,\r\n zip(itertools.repeat(selector),\r\n enumerate(iter(self))), self._chunksize))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef select_many(self, projector, selector=identity):\r\n '''Projects each element of a sequence to an intermediate new sequence,\r\n flattens the resulting sequence into one sequence and optionally\r\n transforms the flattened sequence using a selector function.\r\n\r\n Args:\r\n projector: A unary function mapping each element of the source\r\n sequence into an intermediate sequence. If no projection\r\n function is provided, the intermediate sequence will consist of\r\n the single corresponding element from the source sequence. The\r\n projector function argument (which can have any name) and\r\n return values are,\r\n\r\n Args:\r\n element: The value of the element\r\n\r\n Returns:\r\n An iterable derived from the element value\r\n\r\n selector: An optional unary function mapping the elements in the\r\n flattened intermediate sequence to corresponding elements of\r\n the result sequence. If no selector function is provided, the\r\n identity function is used. The selector function argument and\r\n return values are,\r\n\r\n Args:\r\n element: The value of the intermediate element from the\r\n concatenated sequences arising from the projector\r\n function.\r\n\r\n Returns:\r\n The selected value derived from the element value\r\n Returns:\r\n A generated sequence whose elements are the result of projecting\r\n each element of the source sequence using projector function and\r\n then mapping each element through an optional selector function.\r\n '''\r\n sequences = (self._create(item).select(projector) for item in iter(self))\r\n # TODO: [asq 2.0] Without the list() to force evaluation\r\n # multiprocessing deadlocks...\r\n chained_sequence = list(itertools.chain.from_iterable(sequences))\r\n return self._create(self._pool.imap_unordered(selector,\r\n chained_sequence, self._chunksize))", "response": "Projects each element of a sequence to an intermediate new sequence and optionally transforms the flattened sequence into one sequence and optionally returns the selected value."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ntransform each element of a sequence into a new form. Each element is transformed through a selector function to produce a new value for each value in the source sequence. Each element is transformed through a selector function to produce a new value for each value in the source sequence. The generated sequence is lazily evaluated.", "response": "def select(self, selector):\r\n '''Transforms each element of a sequence into a new form.\r\n\r\n Each element is transformed through a selector function to produce a\r\n value for each value in the source sequence. The generated sequence is\r\n lazily evaluated.\r\n\r\n Args:\r\n selector: A unary function mapping a value in the source sequence\r\n to the corresponding value in the generated generated sequence.\r\n The argument of the selector function (which can have any name)\r\n is,\r\n\r\n Args:\r\n element: The value of the element\r\n\r\n Returns:\r\n The selected value derived from the element value\r\n\r\n Returns:\r\n A generated sequence whose elements are the result of invoking the\r\n selector function on each element of the source sequence.\r\n '''\r\n return self.create(self._pool.imap(selector, iter(self), self._chunksize))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate a selector callable from the supplied value.", "response": "def make_selector(value):\r\n '''Create a selector callable from the supplied value.\r\n\r\n Args:\r\n value: If is a callable, then returned unchanged. If a string is used\r\n then create an attribute selector. If in an integer is used then\r\n create a key selector.\r\n\r\n Returns:\r\n A callable selector based on the supplied value.\r\n\r\n Raises:\r\n ValueError: If a selector cannot be created from the value.\r\n '''\r\n if is_callable(value):\r\n return value\r\n if is_string(value):\r\n return a_(value)\r\n raise ValueError(\"Unable to create callable selector from '{0}'\".format(value))"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nadds with carry returns a two element tuple of the form result carry + other", "response": "def adc(self, other : 'BitVector', carry : Bit) -> tp.Tuple['BitVector', Bit]:\n \"\"\"\n add with carry\n\n returns a two element tuple of the form (result, carry)\n\n \"\"\"\n T = type(self)\n other = _coerce(T, other)\n carry = _coerce(T.unsized_t[1], carry)\n\n a = self.zext(1)\n b = other.zext(1)\n c = carry.zext(T.size)\n\n res = a + b + c\n return res[0:-1], res[-1]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef convert(self, value, *args, **kwargs): # pylint: disable=arguments-differ\n value = os.path.expanduser(value)\n return super(ExpandPath, self).convert(value, *args, **kwargs)", "response": "Take a path with $HOME variables and resolve it to full path."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef make_table(headers=None, rows=None):\n if callable(headers):\n headers = headers()\n if callable(rows):\n rows = rows()\n assert isinstance(headers, list)\n assert isinstance(rows, list)\n assert all(len(row) == len(headers) for row in rows)\n\n plain_headers = [strip_ansi(six.text_type(v)) for v in headers]\n plain_rows = [row for row in [strip_ansi(six.text_type(v)) for v in rows]]\n\n plain_headers = []\n column_widths = []\n\n for k, v in enumerate(headers):\n v = six.text_type(v)\n plain = strip_ansi(v)\n plain_headers.append(plain)\n column_widths.append(len(plain))\n\n if len(v) == len(plain):\n # Value was unstyled, make it bold\n v = click.style(v, bold=True)\n\n headers[k] = v\n\n plain_rows = []\n for row in rows:\n plain_row = []\n for k, v in enumerate(row):\n v = six.text_type(v)\n plain = strip_ansi(v)\n plain_row.append(plain)\n column_widths[k] = max(column_widths[k], len(plain))\n\n plain_rows.append(plain_row)\n\n return Table(\n headers=headers,\n plain_headers=plain_headers,\n rows=rows,\n plain_rows=plain_rows,\n column_widths=column_widths,\n )", "response": "Make a table from headers and rows."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef help(opts, bot, _):\n command = opts['']\n if command is None:\n return bot.help_text()\n\n if command not in bot.commands:\n return \"%r is not a known command\" % command\n\n return bot.commands[command].__doc__", "response": "Help text for a specific command"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef command(cls, name=None):\n # adapted from https://github.com/docopt/docopt/blob/master/examples/interactive_example.py\n\n def decorator(func):\n\n @functools.wraps(func)\n def _cmd_wrapper(rest, *args, **kwargs):\n try:\n usage = _cmd_wrapper.__doc__.partition('\\n')[0]\n opts = docopt(usage, rest)\n except (SystemExit, DocoptExit) as e:\n # opts did not match\n return str(e)\n\n return func(opts, *args, **kwargs)\n\n cls.commands[name or func.__name__] = _cmd_wrapper\n\n return _cmd_wrapper\n return decorator", "response": "A decorator to convert a function to a command."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a slack - formatted list of commands with their usage.", "response": "def help_text(cls):\n \"\"\"Return a slack-formatted list of commands with their usage.\"\"\"\n docs = [cmd_func.__doc__ for cmd_func in cls.commands.values()]\n\n # Don't want to include 'usage: ' or explanation.\n usage_lines = [doc.partition('\\n')[0] for doc in docs]\n terse_lines = [line[len('Usage: '):] for line in usage_lines]\n terse_lines.sort()\n return '\\n'.join(['Available commands:\\n'] + terse_lines)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef run_forever(self):\n res = self.slack.rtm.start()\n self.log.info(\"current channels: %s\",\n ','.join(c['name'] for c in res.body['channels']\n if c['is_member']))\n self.id = res.body['self']['id']\n self.name = res.body['self']['name']\n self.my_mention = \"<@%s>\" % self.id\n\n self.ws = websocket.WebSocketApp(\n res.body['url'],\n on_message=self._on_message,\n on_error=self._on_error,\n on_close=self._on_close,\n on_open=self._on_open)\n self.prepare_connection(self.config)\n self.ws.run_forever()", "response": "Run the bot blocking forever."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _bot_identifier(self, message):\n\n text = message['text']\n\n formatters = [\n lambda identifier: \"%s \" % identifier,\n lambda identifier: \"%s:\" % identifier,\n ]\n my_identifiers = [formatter(identifier) for identifier in [self.name, self.my_mention] for formatter in formatters]\n\n for identifier in my_identifiers:\n if text.startswith(identifier):\n self.log.debug(\"sent to me:\\n%s\", pprint.pformat(message))\n return identifier\n\n return None", "response": "Return the identifier used to address this bot in this message."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nhandle the response from a slacker command.", "response": "def _handle_command_response(self, res, event):\n \"\"\"Either send a message (choosing between rtm and postMessage) or ignore the response.\n\n :param event: a slacker event dict\n :param res: a string, a dict, or None.\n See the command docstring for what these represent.\n \"\"\"\n\n response_handler = None\n\n if isinstance(res, basestring):\n response_handler = functools.partial(self._send_rtm_message, event['channel'])\n elif isinstance(res, dict):\n response_handler = self._send_api_message\n\n if response_handler is not None:\n response_handler(res)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsplits the slack response string or dict into multiple events", "response": "def _handle_long_response(self, res):\n \"\"\"Splits messages that are too long into multiple events\n :param res: a slack response string or dict\n \"\"\"\n\n is_rtm_message = isinstance(res, basestring)\n is_api_message = isinstance(res, dict)\n\n if is_rtm_message:\n text = res\n elif is_api_message:\n text = res['text']\n\n message_length = len(text)\n\n if message_length <= SLACK_MESSAGE_LIMIT:\n return [res]\n\n remaining_str = text\n responses = []\n\n while remaining_str:\n less_than_limit = len(remaining_str) < SLACK_MESSAGE_LIMIT\n\n if less_than_limit:\n last_line_break = None\n else:\n last_line_break = remaining_str[:SLACK_MESSAGE_LIMIT].rfind('\\n')\n\n if is_rtm_message:\n responses.append(remaining_str[:last_line_break])\n elif is_api_message:\n template = res.copy()\n template['text'] = remaining_str[:last_line_break]\n responses.append(template)\n\n if less_than_limit:\n remaining_str = None\n else:\n remaining_str = remaining_str[last_line_break:]\n\n self.log.debug(\"_handle_long_response: splitting long response %s, returns: \\n %s\",\n pprint.pformat(res), pprint.pformat(responses))\n return responses"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsends a Slack message over RTM.", "response": "def _send_rtm_message(self, channel_id, text):\n \"\"\"Send a Slack message to a channel over RTM.\n\n :param channel_id: a slack channel id.\n :param text: a slack message. Serverside formatting is done\n in a similar way to normal user message; see\n `Slack's docs `__.\n \"\"\"\n\n message = {\n 'id': self._current_message_id,\n 'type': 'message',\n 'channel': channel_id,\n 'text': text,\n }\n self.ws.send(json.dumps(message))\n self._current_message_id += 1"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _send_api_message(self, message):\n\n self.slack.chat.post_message(**message)\n self.log.debug(\"sent api message %r\", message)", "response": "Send a Slack message via the chat. postMessage api."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\niterate across the complete set of child validators.", "response": "def _validators(self):\n\t\t\"\"\"Iterate across the complete set of child validators.\"\"\"\n\t\t\n\t\tfor validator in self.__validators__.values():\n\t\t\tyield validator\n\t\t\n\t\tif self.validators:\n\t\t\tfor validator in self.validators:\n\t\t\t\tyield validator"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a suitable user agent for identifying the CLI process.", "response": "def make_user_agent(prefix=None):\n \"\"\"Get a suitable user agent for identifying the CLI process.\"\"\"\n prefix = (prefix or platform.platform(terse=1)).strip().lower()\n return \"cloudsmith-cli/%(prefix)s cli:%(version)s api:%(api_version)s\" % {\n \"version\": get_cli_version(),\n \"api_version\": get_api_version(),\n \"prefix\": prefix,\n }"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef pretty_print_table(headers, rows):\n table = make_table(headers=headers, rows=rows)\n pretty_print_table_instance(table)", "response": "Pretty print a table from headers and rows."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef print_rate_limit_info(opts, rate_info, atexit=False):\n if not rate_info:\n return\n\n show_info = (\n opts.always_show_rate_limit\n or atexit\n or rate_info.interval > opts.rate_limit_warning\n )\n\n if not show_info:\n return\n\n click.echo(err=True)\n click.secho(\n \"Throttling (rate limited) for: %(throttle)s seconds ... \"\n % {\"throttle\": click.style(six.text_type(rate_info.interval), reverse=True)},\n err=True,\n reset=False,\n )", "response": "Tell the user when we re being rate limited."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef confirm_operation(prompt, prefix=None, assume_yes=False, err=False):\n if assume_yes:\n return True\n\n prefix = prefix or click.style(\n \"Are you %s certain you want to\" % (click.style(\"absolutely\", bold=True))\n )\n\n prompt = \"%(prefix)s %(prompt)s?\" % {\"prefix\": prefix, \"prompt\": prompt}\n\n if click.confirm(prompt, err=err):\n return True\n\n click.echo(err=err)\n click.secho(\"OK, phew! Close call. :-)\", fg=\"green\", err=err)\n return False", "response": "Prompt the user for confirmation for dangerous actions."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef validate_login(ctx, param, value):\n # pylint: disable=unused-argument\n value = value.strip()\n if not value:\n raise click.BadParameter(\"The value cannot be blank.\", param=param)\n return value", "response": "Ensure that login is not blank."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef create_config_files(ctx, opts, api_key):\n # pylint: disable=unused-argument\n config_reader = opts.get_config_reader()\n creds_reader = opts.get_creds_reader()\n has_config = config_reader.has_default_file()\n has_creds = creds_reader.has_default_file()\n\n if has_config and has_creds:\n create = False\n else:\n click.echo()\n create = click.confirm(\n \"No default config file(s) found, do you want to create them?\"\n )\n\n click.echo()\n if not create:\n click.secho(\n \"For reference here are your default config file locations:\", fg=\"yellow\"\n )\n else:\n click.secho(\n \"Great! Let me just create your default configs for you now ...\", fg=\"green\"\n )\n\n configs = (\n ConfigValues(reader=config_reader, present=has_config, mode=None, data={}),\n ConfigValues(\n reader=creds_reader,\n present=has_creds,\n mode=stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IWGRP,\n data={\"api_key\": api_key},\n ),\n )\n\n has_errors = False\n for config in configs:\n click.echo(\n \"%(name)s config file: %(filepath)s ... \"\n % {\n \"name\": click.style(config.reader.config_name.capitalize(), bold=True),\n \"filepath\": click.style(\n config.reader.get_default_filepath(), fg=\"magenta\"\n ),\n },\n nl=False,\n )\n\n if not config.present and create:\n try:\n ok = config.reader.create_default_file(\n data=config.data, mode=config.mode\n )\n except (OSError, IOError) as exc:\n ok = False\n error_message = exc.strerror\n has_errors = True\n\n if ok:\n click.secho(\"CREATED\", fg=\"green\")\n else:\n click.secho(\"ERROR\", fg=\"red\")\n click.secho(\n \"The following error occurred while trying to \"\n \"create the file: %(message)s\"\n % {\"message\": click.style(error_message, fg=\"red\")}\n )\n continue\n\n click.secho(\"EXISTS\" if config.present else \"NOT CREATED\", fg=\"yellow\")\n\n return create, has_errors", "response": "Create default config files."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef token(ctx, opts, login, password):\n click.echo(\n \"Retrieving API token for %(login)s ... \"\n % {\"login\": click.style(login, bold=True)},\n nl=False,\n )\n\n context_msg = \"Failed to retrieve the API token!\"\n with handle_api_exceptions(ctx, opts=opts, context_msg=context_msg):\n with maybe_spinner(opts):\n api_key = get_user_token(login=login, password=password)\n\n click.secho(\"OK\", fg=\"green\")\n\n click.echo(\n \"Your API key/token is: %(token)s\"\n % {\"token\": click.style(api_key, fg=\"magenta\")}\n )\n\n create, has_errors = create_config_files(ctx, opts, api_key=api_key)\n\n if has_errors:\n click.echo()\n click.secho(\"Oops, please fix the errors and try again!\", fg=\"red\")\n return\n\n if opts.api_key != api_key:\n click.echo()\n if opts.api_key:\n click.secho(\n \"Note: The above API key doesn't match what you have in \"\n \"your default credentials config file.\",\n fg=\"yellow\",\n )\n elif not create:\n click.secho(\n \"Note: Don't forget to put your API key in a config file, \"\n \"export it on the environment, or set it via -k.\",\n fg=\"yellow\",\n )\n click.secho(\n \"If you need more help please see the documentation: \"\n \"%(website)s\" % {\"website\": click.style(get_help_website(), bold=True)}\n )\n click.echo()\n\n click.secho(\"You're ready to rock, let's start automating!\", fg=\"green\")", "response": "Retrieve your API authentication token for the given login and password."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting the synchronisation status for a package.", "response": "def status(ctx, opts, owner_repo_package):\n \"\"\"\n Get the synchronisation status for a package.\n\n - OWNER/REPO/PACKAGE: Specify the OWNER namespace (i.e. user or org), the\n REPO name where the package is stored, and the PACKAGE name (slug) of the\n package itself. All separated by a slash.\n\n Example: 'your-org/awesome-repo/better-pkg'.\n \"\"\"\n owner, repo, slug = owner_repo_package\n\n click.echo(\n \"Getting status of %(package)s in %(owner)s/%(repo)s ... \"\n % {\n \"owner\": click.style(owner, bold=True),\n \"repo\": click.style(repo, bold=True),\n \"package\": click.style(slug, bold=True),\n },\n nl=False,\n )\n\n context_msg = \"Failed to get status of package!\"\n with handle_api_exceptions(ctx, opts=opts, context_msg=context_msg):\n with maybe_spinner(opts):\n res = get_package_status(owner, repo, slug)\n ok, failed, _, status_str, stage_str, reason = res\n\n click.secho(\"OK\", fg=\"green\")\n\n if not stage_str:\n package_status = status_str\n else:\n package_status = \"%(status)s / %(stage)s\" % {\n \"status\": status_str,\n \"stage\": stage_str,\n }\n\n if ok:\n status_colour = \"green\"\n elif failed:\n status_colour = \"red\"\n else:\n status_colour = \"magenta\"\n\n click.secho(\n \"The package status is: %(status)s\"\n % {\"status\": click.style(package_status, fg=status_colour)}\n )\n\n if reason:\n click.secho(\n \"Reason given: %(reason)s\" % {\"reason\": click.style(reason, fg=\"yellow\")},\n fg=status_colour,\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nmap positional to keyword arguments identify invalid assignments and return the result.", "response": "def _process_arguments(self, args, kw):\n\t\t\"\"\"Map positional to keyword arguments, identify invalid assignments, and return the result.\n\t\t\n\t\tThis is likely generic enough to be useful as a standalone utility function, and goes to a fair amount of\n\t\teffort to ensure raised exceptions are as Python-like as possible.\n\t\t\"\"\"\n\t\t\n\t\t# Ensure we were not passed too many arguments.\n\t\tif len(args) > len(self.__attributes__):\n\t\t\traise TypeError('{0} takes no more than {1} argument{2} ({3} given)'.format(\n\t\t\t\t\tself.__class__.__name__,\n\t\t\t\t\tlen(self.__attributes__),\n\t\t\t\t\t'' if len(self.__attributes__) == 1 else 's',\n\t\t\t\t\tlen(args)\n\t\t\t\t))\n\t\t\n\t\t# Retrieve the names associated with the positional parameters.\n\t\tnames = [name for name in self.__attributes__.keys() if name[0] != '_' or name == '__name__'][:len(args)]\n\t\t\n\t\t# Sets provide a convienent way to identify intersections.\n\t\tduplicates = set(kw.keys()) & set(names)\n\t\t\n\t\t# Given duplicate values, explode gloriously.\n\t\tif duplicates:\n\t\t\traise TypeError('{0} got multiple values for keyword argument{1}: {2}'.format(\n\t\t\t\t\tself.__class__.__name__,\n\t\t\t\t\t'' if len(duplicates) == 1 else 's',\n\t\t\t\t\t', '.join(duplicates)\n\t\t\t\t))\n\t\t\n\t\tdef field_values(args, kw):\n\t\t\t\"\"\"A little closure to yield out fields and their assigned values in field order.\"\"\"\n\t\t\t\n\t\t\tfor i, arg in enumerate(self.__attributes__.keys()):\n\t\t\t\tif len(args):\n\t\t\t\t\tyield arg, args.popleft()\n\t\t\t\t\n\t\t\t\tif arg in kw:\n\t\t\t\t\tyield arg, kw.pop(arg)\n\t\t\n\t\tresult = odict(field_values(deque(args), dict(kw)))\n\t\t\n\t\t# Again use sets, this time to identify unknown keys.\n\t\tunknown = set(kw.keys()) - set(result.keys())\n\t\t\n\t\t# Given unknown keys, explode gloriously.\n\t\tif unknown:\n\t\t\traise TypeError('{0} got unexpected keyword argument{1}: {2}'.format(\n\t\t\t\t\tself.__class__.__name__,\n\t\t\t\t\t'' if len(unknown) == 1 else 's',\n\t\t\t\t\t', '.join(unknown)\n\t\t\t\t))\n\t\t\n\t\treturn result"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nimporting and wrap deprecated methods in a module dict.", "response": "def import_and_wrap_deprecated(module_name, module_dict, warn_import=True):\n \"\"\"\n Usage:\n import_and_wrap_deprecated('conda.common.connection', locals())\n # looks for conda.common.connection.__all__\n \"\"\"\n if warn_import:\n deprecated_import(module_name)\n\n from importlib import import_module\n module = import_module(module_name)\n for attr in module.__all__:\n module_dict[attr] = deprecated(getattr(module, attr))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef deprecate_module_with_proxy(module_name, module_dict, deprecated_attributes=None):\n def _ModuleProxy(module, depr):\n \"\"\"Return a wrapped object that warns about deprecated accesses\"\"\"\n # http://stackoverflow.com/a/922693/2127762\n class Wrapper(object):\n def __getattr__(self, attr):\n if depr is None or attr in depr:\n warnings.warn(\"Property %s is deprecated\" % attr)\n\n return getattr(module, attr)\n\n def __setattr__(self, attr, value):\n if depr is None or attr in depr:\n warnings.warn(\"Property %s is deprecated\" % attr)\n return setattr(module, attr, value)\n return Wrapper()\n\n deprecated_import(module_name)\n\n deprs = set()\n for key in deprecated_attributes or module_dict:\n if key.startswith('_'):\n continue\n if callable(module_dict[key]) and not isbuiltin(module_dict[key]):\n module_dict[key] = deprecated(module_dict[key])\n else:\n deprs.add(key)\n sys.modules[module_name] = _ModuleProxy(sys.modules[module_name], deprs or None)", "response": "Deprecate a module by adding a proxy to the deprecated_attributes list."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef native(self, value, context=None):\n\t\t\n\t\tvalue = super().native(value, context)\n\t\t\n\t\tif self.none and (value is None):\n\t\t\treturn None\n\t\t\n\t\ttry:\n\t\t\tvalue = value.lower()\n\t\texcept AttributeError:\n\t\t\treturn bool(value)\n\t\t\n\t\tif value in self.truthy:\n\t\t\treturn True\n\t\t\n\t\tif value in self.falsy:\n\t\t\treturn False\n\t\t\n\t\traise Concern(\"Unable to convert {0!r} to a boolean value.\", value)", "response": "Convert a foreign value to a native boolean."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef foreign(self, value, context=None):\n\t\t\n\t\tif self.none and value is None:\n\t\t\treturn ''\n\t\t\n\t\ttry:\n\t\t\tvalue = self.native(value, context)\n\t\texcept Concern:\n\t\t\t# The value might not be in the lists; bool() evaluate it instead.\n\t\t\tvalue = bool(value.strip() if self.strip and hasattr(value, 'strip') else value)\n\t\t\n\t\tif value in self.truthy or value:\n\t\t\treturn self.truthy[self.use]\n\t\t\n\t\treturn self.falsy[self.use]", "response": "Convert a native value to a textual boolean."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_root_path():\n root_path = __file__\n return os.path.dirname(os.path.realpath(root_path))", "response": "Get the root path for the application."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreads the specific file into a string in its entirety.", "response": "def read(path, root_path=None):\n \"\"\"Read the specific file into a string in its entirety.\"\"\"\n try:\n root_path = root_path or get_root_path()\n real_path = os.path.realpath(os.path.join(root_path, path))\n with open(real_path) as fp:\n return fp.read().strip()\n except IOError:\n return \"\""} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_long_description():\n root_path = get_root_path()\n readme_path = os.path.join(root_path, \"README.md\")\n\n try:\n import pypandoc\n\n return pypandoc.convert(readme_path, \"rst\").strip()\n except ImportError:\n return \"Cloudsmith CLI\"", "response": "Grok the readme turn it into whine ( rst )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef tan_rand(q, seed=9):\n # probably need a check in case we get a parallel vector\n rs = np.random.RandomState(seed)\n rvec = rs.rand(q.shape[0])\n\n qd = np.cross(rvec, q)\n qd = qd / np.linalg.norm(qd)\n\n while np.dot(q, qd) > 1e-6:\n rvec = rs.rand(q.shape[0])\n qd = np.cross(rvec, q)\n qd = qd / np.linalg.norm(qd)\n\n return qd", "response": "Find a random vector in the tangent space of the n sphere"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef perturb_vec(q, cone_half_angle=2):\n rand_vec = tan_rand(q)\n cross_vector = attitude.unit_vector(np.cross(q, rand_vec))\n\n s = np.random.uniform(0, 1, 1)\n r = np.random.uniform(0, 1, 1)\n\n h = np.cos(np.deg2rad(cone_half_angle))\n\n phi = 2 * np.pi * s\n z = h + ( 1- h) * r\n sinT = np.sqrt(1 - z**2)\n x = np.cos(phi) * sinT\n y = np.sin(phi) * sinT\n\n perturbed = rand_vec * x + cross_vector * y + q * z\n \n return perturbed", "response": "r Perturb a vector randomly and return the perturbed version of the random vector."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef common_package_action_options(f):\n\n @click.option(\n \"-s\",\n \"--skip-errors\",\n default=False,\n is_flag=True,\n help=\"Skip/ignore errors when copying packages.\",\n )\n @click.option(\n \"-W\",\n \"--no-wait-for-sync\",\n default=False,\n is_flag=True,\n help=\"Don't wait for package synchronisation to complete before \" \"exiting.\",\n )\n @click.option(\n \"-I\",\n \"--wait-interval\",\n default=5.0,\n type=float,\n show_default=True,\n help=\"The time in seconds to wait between checking synchronisation.\",\n )\n @click.option(\n \"--sync-attempts\",\n default=3,\n type=int,\n help=\"Number of times to attempt package synchronisation. If the \"\n \"package fails the first time, the client will attempt to \"\n \"automatically resynchronise it.\",\n )\n @click.pass_context\n @functools.wraps(f)\n def wrapper(ctx, *args, **kwargs):\n # pylint: disable=missing-docstring\n return ctx.invoke(f, *args, **kwargs)\n\n return wrapper", "response": "Add common options for package actions."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef common_cli_config_options(f):\n\n @click.option(\n \"-C\",\n \"--config-file\",\n envvar=\"CLOUDSMITH_CONFIG_FILE\",\n type=click.Path(dir_okay=True, exists=True, writable=False, resolve_path=True),\n help=\"The path to your config.ini file.\",\n )\n @click.option(\n \"--credentials-file\",\n envvar=\"CLOUDSMITH_CREDENTIALS_FILE\",\n type=click.Path(dir_okay=True, exists=True, writable=False, resolve_path=True),\n help=\"The path to your credentials.ini file.\",\n )\n @click.option(\n \"-P\",\n \"--profile\",\n default=None,\n envvar=\"CLOUDSMITH_PROFILE\",\n help=\"The name of the profile to use for configuration.\",\n )\n @click.pass_context\n @functools.wraps(f)\n def wrapper(ctx, *args, **kwargs):\n # pylint: disable=missing-docstring\n opts = config.get_or_create_options(ctx)\n profile = kwargs.pop(\"profile\")\n config_file = kwargs.pop(\"config_file\")\n creds_file = kwargs.pop(\"credentials_file\")\n opts.load_config_file(path=config_file, profile=profile)\n opts.load_creds_file(path=creds_file, profile=profile)\n kwargs[\"opts\"] = opts\n return ctx.invoke(f, *args, **kwargs)\n\n return wrapper", "response": "Add common CLI config options to commands."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nadds common CLI output options to commands.", "response": "def common_cli_output_options(f):\n \"\"\"Add common CLI output options to commands.\"\"\"\n\n @click.option(\n \"-d\",\n \"--debug\",\n default=False,\n is_flag=True,\n help=\"Produce debug output during processing.\",\n )\n @click.option(\n \"-F\",\n \"--output-format\",\n default=\"pretty\",\n type=click.Choice([\"pretty\", \"json\", \"pretty_json\"]),\n help=\"Determines how output is formatted. This is only supported by a \"\n \"subset of the commands at the moment (e.g. list).\",\n )\n @click.option(\n \"-v\",\n \"--verbose\",\n is_flag=True,\n default=False,\n help=\"Produce more output during processing.\",\n )\n @click.pass_context\n @functools.wraps(f)\n def wrapper(ctx, *args, **kwargs):\n # pylint: disable=missing-docstring\n opts = config.get_or_create_options(ctx)\n opts.debug = kwargs.pop(\"debug\")\n opts.output = kwargs.pop(\"output_format\")\n opts.verbose = kwargs.pop(\"verbose\")\n kwargs[\"opts\"] = opts\n return ctx.invoke(f, *args, **kwargs)\n\n return wrapper"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef common_cli_list_options(f):\n\n @click.option(\n \"-p\",\n \"--page\",\n default=1,\n type=int,\n help=\"The page to view for lists, where 1 is the first page\",\n callback=validators.validate_page,\n )\n @click.option(\n \"-l\",\n \"--page-size\",\n default=30,\n type=int,\n help=\"The amount of items to view per page for lists.\",\n callback=validators.validate_page_size,\n )\n @click.pass_context\n @functools.wraps(f)\n def wrapper(ctx, *args, **kwargs):\n # pylint: disable=missing-docstring\n opts = config.get_or_create_options(ctx)\n kwargs[\"opts\"] = opts\n return ctx.invoke(f, *args, **kwargs)\n\n return wrapper", "response": "Add common list options to commands."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef common_api_auth_options(f):\n\n @click.option(\n \"-k\",\n \"--api-key\",\n hide_input=True,\n envvar=\"CLOUDSMITH_API_KEY\",\n help=\"The API key for authenticating with the API.\",\n )\n @click.pass_context\n @functools.wraps(f)\n def wrapper(ctx, *args, **kwargs):\n # pylint: disable=missing-docstring\n opts = config.get_or_create_options(ctx)\n opts.api_key = kwargs.pop(\"api_key\")\n kwargs[\"opts\"] = opts\n return ctx.invoke(f, *args, **kwargs)\n\n return wrapper", "response": "Add common API authentication options to commands."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef initialise_api(f):\n\n @click.option(\n \"--api-host\", envvar=\"CLOUDSMITH_API_HOST\", help=\"The API host to connect to.\"\n )\n @click.option(\n \"--api-proxy\",\n envvar=\"CLOUDSMITH_API_PROXY\",\n help=\"The API proxy to connect through.\",\n )\n @click.option(\n \"--api-user-agent\",\n envvar=\"CLOUDSMITH_API_USER_AGENT\",\n help=\"The user agent to use for requests.\",\n )\n @click.option(\n \"--api-headers\",\n envvar=\"CLOUDSMITH_API_HEADERS\",\n help=\"A CSV list of extra headers (key=value) to send to the API.\",\n )\n @click.option(\n \"-R\",\n \"--without-rate-limit\",\n default=False,\n is_flag=True,\n help=\"Don't obey the suggested rate limit interval. The CLI will \"\n \"automatically sleep between commands to ensure that you do \"\n \"not hit the server-side rate limit.\",\n )\n @click.option(\n \"--rate-limit-warning\",\n default=30,\n help=\"When rate limiting, display information that it is happening \"\n \"if wait interval is higher than this setting. By default no \"\n \"information will be printed. Set to zero to always see it.\",\n )\n @click.option(\n \"--error-retry-max\",\n default=6,\n help=\"The maximum amount of times to retry on errors received from \"\n \"the API, as determined by the --error-retry-codes parameter.\",\n )\n @click.option(\n \"--error-retry-backoff\",\n default=0.23,\n type=float,\n help=\"The backoff factor determines how long to wait in seconds \"\n \"between error retries. The backoff factor is multiplied by the \"\n \"amount of retries so far. So if 0.1, then the wait is 0.1s then \"\n \"0.2s, then 0.4s, and so forth.\",\n )\n @click.option(\n \"--error-retry-codes\",\n default=\"500,502,503,504\",\n help=\"The status codes that when received from the API will cause \"\n \"a retry (if --error-retry-max is > 0). By default this will be for \"\n \"500, 502, 503 and 504 error codes.\",\n )\n @click.pass_context\n @functools.wraps(f)\n def wrapper(ctx, *args, **kwargs):\n # pylint: disable=missing-docstring\n opts = config.get_or_create_options(ctx)\n opts.api_host = kwargs.pop(\"api_host\")\n opts.api_proxy = kwargs.pop(\"api_proxy\")\n opts.api_user_agent = kwargs.pop(\"api_user_agent\")\n opts.api_headers = kwargs.pop(\"api_headers\")\n opts.rate_limit = not kwargs.pop(\"without_rate_limit\")\n opts.rate_limit_warning = kwargs.pop(\"rate_limit_warning\")\n opts.error_retry_max = kwargs.pop(\"error_retry_max\")\n opts.error_retry_backoff = kwargs.pop(\"error_retry_backoff\")\n opts.error_retry_codes = kwargs.pop(\"error_retry_codes\")\n\n def call_print_rate_limit_info_with_opts(rate_info, atexit=False):\n utils.print_rate_limit_info(opts, rate_info, atexit=atexit)\n\n opts.api_config = _initialise_api(\n debug=opts.debug,\n host=opts.api_host,\n key=opts.api_key,\n proxy=opts.api_proxy,\n user_agent=opts.api_user_agent,\n headers=opts.api_headers,\n rate_limit=opts.rate_limit,\n rate_limit_callback=call_print_rate_limit_info_with_opts,\n error_retry_max=opts.error_retry_max,\n error_retry_backoff=opts.error_retry_backoff,\n error_retry_codes=opts.error_retry_codes,\n )\n\n kwargs[\"opts\"] = opts\n return ctx.invoke(f, *args, **kwargs)\n\n return wrapper", "response": "Initialise the Cloudsmith API for use."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef list_entitlements(owner, repo, page, page_size, show_tokens):\n client = get_entitlements_api()\n\n with catch_raise_api_exception():\n data, _, headers = client.entitlements_list_with_http_info(\n owner=owner,\n repo=repo,\n page=page,\n page_size=page_size,\n show_tokens=show_tokens,\n )\n\n ratelimits.maybe_rate_limit(client, headers)\n page_info = PageInfo.from_headers(headers)\n entitlements = [ent.to_dict() for ent in data] # pylint: disable=no-member\n return entitlements, page_info", "response": "Get a list of entitlements on a repository."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create_entitlement(owner, repo, name, token, show_tokens):\n client = get_entitlements_api()\n\n data = {}\n if name is not None:\n data[\"name\"] = name\n\n if token is not None:\n data[\"token\"] = token\n\n with catch_raise_api_exception():\n data, _, headers = client.entitlements_create_with_http_info(\n owner=owner, repo=repo, data=data, show_tokens=show_tokens\n )\n\n ratelimits.maybe_rate_limit(client, headers)\n return data.to_dict()", "response": "Create an entitlement in a repository."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef delete_entitlement(owner, repo, identifier):\n client = get_entitlements_api()\n\n with catch_raise_api_exception():\n _, _, headers = client.entitlements_delete_with_http_info(\n owner=owner, repo=repo, identifier=identifier\n )\n\n ratelimits.maybe_rate_limit(client, headers)", "response": "Delete an entitlement from a repository."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef update_entitlement(owner, repo, identifier, name, token, show_tokens):\n client = get_entitlements_api()\n\n data = {}\n if name is not None:\n data[\"name\"] = name\n\n if token is not None:\n data[\"token\"] = token\n\n with catch_raise_api_exception():\n data, _, headers = client.entitlements_partial_update_with_http_info(\n owner=owner,\n repo=repo,\n identifier=identifier,\n data=data,\n show_tokens=show_tokens,\n )\n\n ratelimits.maybe_rate_limit(client, headers)\n return data.to_dict()", "response": "Update an entitlement in a repository."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nrefreshes an entitlement in a repository.", "response": "def refresh_entitlement(owner, repo, identifier, show_tokens):\n \"\"\"Refresh an entitlement in a repository.\"\"\"\n client = get_entitlements_api()\n\n with catch_raise_api_exception():\n data, _, headers = client.entitlements_refresh_with_http_info(\n owner=owner, repo=repo, identifier=identifier, show_tokens=show_tokens\n )\n\n ratelimits.maybe_rate_limit(client, headers)\n return data.to_dict()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_api_client(cls):\n config = cloudsmith_api.Configuration()\n client = cls()\n client.config = config\n client.api_client.rest_client = RestClient()\n\n user_agent = getattr(config, \"user_agent\", None)\n if user_agent:\n client.api_client.user_agent = user_agent\n\n headers = getattr(config, \"headers\", None)\n if headers:\n for k, v in six.iteritems(headers):\n client.api_client.set_default_header(k, v)\n\n return client", "response": "Get an API client with configuration."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconfigure a new API key.", "response": "def set_api_key(config, key):\n \"\"\"Configure a new API key.\"\"\"\n if not key and \"X-Api-Key\" in config.api_key:\n del config.api_key[\"X-Api-Key\"]\n else:\n config.api_key[\"X-Api-Key\"] = key"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nvalidating parameters for requesting a file upload.", "response": "def validate_request_file_upload(owner, repo, filepath, md5_checksum=None):\n \"\"\"Validate parameters for requesting a file upload.\"\"\"\n client = get_files_api()\n md5_checksum = md5_checksum or calculate_file_md5(filepath)\n\n with catch_raise_api_exception():\n _, _, headers = client.files_validate_with_http_info(\n owner=owner,\n repo=repo,\n data={\"filename\": os.path.basename(filepath), \"md5_checksum\": md5_checksum},\n )\n\n ratelimits.maybe_rate_limit(client, headers)\n return md5_checksum"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef request_file_upload(owner, repo, filepath, md5_checksum=None):\n client = get_files_api()\n md5_checksum = md5_checksum or calculate_file_md5(filepath)\n\n with catch_raise_api_exception():\n data, _, headers = client.files_create_with_http_info(\n owner=owner,\n repo=repo,\n data={\"filename\": os.path.basename(filepath), \"md5_checksum\": md5_checksum},\n )\n\n # pylint: disable=no-member\n # Pylint detects the returned value as a tuple\n ratelimits.maybe_rate_limit(client, headers)\n return data.identifier, data.upload_url, data.upload_fields", "response": "Request a new file upload."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef upload_file(upload_url, upload_fields, filepath, callback=None):\n upload_fields = list(six.iteritems(upload_fields))\n upload_fields.append(\n (\"file\", (os.path.basename(filepath), click.open_file(filepath, \"rb\")))\n )\n encoder = MultipartEncoder(upload_fields)\n monitor = MultipartEncoderMonitor(encoder, callback=callback)\n\n config = cloudsmith_api.Configuration()\n if config.proxy:\n proxies = {\"http\": config.proxy, \"https\": config.proxy}\n else:\n proxies = None\n\n headers = {\"content-type\": monitor.content_type}\n\n client = get_files_api()\n headers[\"user-agent\"] = client.api_client.user_agent\n\n session = create_requests_session()\n resp = session.post(upload_url, data=monitor, headers=headers, proxies=proxies)\n\n try:\n resp.raise_for_status()\n except requests.RequestException as exc:\n raise ApiException(\n resp.status_code, headers=exc.response.headers, body=exc.response.content\n )", "response": "Upload a pre - signed file to Cloudsmith."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef handle_api_exceptions(\n ctx, opts, context_msg=None, nl=False, exit_on_error=True, reraise_on_error=False\n):\n \"\"\"Context manager that handles API exceptions.\"\"\"\n # flake8: ignore=C901\n # Use stderr for messages if the output is something else (e.g. # JSON)\n use_stderr = opts.output != \"pretty\"\n\n try:\n yield\n except ApiException as exc:\n if nl:\n click.echo(err=use_stderr)\n click.secho(\"ERROR: \", fg=\"red\", nl=False, err=use_stderr)\n else:\n click.secho(\"ERROR\", fg=\"red\", err=use_stderr)\n\n context_msg = context_msg or \"Failed to perform operation!\"\n click.secho(\n \"%(context)s (status: %(code)s - %(code_text)s)\"\n % {\n \"context\": context_msg,\n \"code\": exc.status,\n \"code_text\": exc.status_description,\n },\n fg=\"red\",\n err=use_stderr,\n )\n\n detail, fields = get_details(exc)\n if detail or fields:\n click.echo(err=use_stderr)\n\n if detail:\n click.secho(\n \"Detail: %(detail)s\"\n % {\"detail\": click.style(detail, fg=\"red\", bold=False)},\n bold=True,\n err=use_stderr,\n )\n\n if fields:\n for k, v in six.iteritems(fields):\n field = \"%s Field\" % k.capitalize()\n click.secho(\n \"%(field)s: %(message)s\"\n % {\n \"field\": click.style(field, bold=True),\n \"message\": click.style(v, fg=\"red\"),\n },\n err=use_stderr,\n )\n\n hint = get_error_hint(ctx, opts, exc)\n if hint:\n click.echo(\n \"Hint: %(hint)s\" % {\"hint\": click.style(hint, fg=\"yellow\")},\n err=use_stderr,\n )\n\n if opts.verbose and not opts.debug:\n if exc.headers:\n click.echo(err=use_stderr)\n click.echo(\"Headers in Reply:\", err=use_stderr)\n for k, v in six.iteritems(exc.headers):\n click.echo(\n \"%(key)s = %(value)s\" % {\"key\": k, \"value\": v}, err=use_stderr\n )\n\n if reraise_on_error:\n six.reraise(*sys.exc_info())\n\n if exit_on_error:\n ctx.exit(exc.status or 1)", "response": "Context manager that handles API exceptions."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting the details from the exception.", "response": "def get_details(exc):\n \"\"\"Get the details from the exception.\"\"\"\n detail = None\n fields = collections.OrderedDict()\n\n if exc.detail:\n detail = exc.detail\n\n if exc.fields:\n for k, v in six.iteritems(exc.fields):\n try:\n field_detail = v[\"detail\"]\n except (TypeError, KeyError):\n field_detail = v\n\n if isinstance(field_detail, (list, tuple)):\n field_detail = \" \".join(field_detail)\n\n if k == \"non_field_errors\":\n if detail:\n detail += \" \" + field_detail\n else:\n detail = field_detail\n continue\n\n fields[k] = field_detail\n\n return detail, fields"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_error_hint(ctx, opts, exc):\n module = sys.modules[__name__]\n get_specific_error_hint = getattr(module, \"get_%s_error_hint\" % exc.status, None)\n if get_specific_error_hint:\n return get_specific_error_hint(ctx, opts, exc)\n return None", "response": "Get a hint to show to the user if any."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef make_env_key(app_name, key):\n key = key.replace('-', '_').replace(' ', '_')\n return str(\"_\".join((x.upper() for x in (app_name, key))))", "response": "Creates an environment key - equivalent for the given key"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting the environment variable by prepending the app_name to key.", "response": "def set_env(self, key, value):\n \"\"\"Sets environment variables by prepending the app_name to `key`. Also registers the\n environment variable with the instance object preventing an otherwise-required call to\n `reload()`.\n \"\"\"\n os.environ[make_env_key(self.appname, key)] = str(value) # must coerce to string\n self._registered_env_keys.add(key)\n self._clear_memoization()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nremoving an environment variable using the prepended app_name convention with key.", "response": "def unset_env(self, key):\n \"\"\"Removes an environment variable using the prepended app_name convention with `key`.\"\"\"\n os.environ.pop(make_env_key(self.appname, key), None)\n self._registered_env_keys.discard(key)\n self._clear_memoization()"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _reload(self, force=False):\n self._config_map = dict()\n self._registered_env_keys = set()\n self.__reload_sources(force)\n self.__load_environment_keys()\n self.verify()\n self._clear_memoization()", "response": "Reloads the configuration from the file and environment variables. Useful if using\n os. environ instead of this class s set_env method."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\npromote a package from one repository to another repository.", "response": "def move(\n ctx,\n opts,\n owner_repo_package,\n destination,\n skip_errors,\n wait_interval,\n no_wait_for_sync,\n sync_attempts,\n yes,\n):\n \"\"\"\n Move (promote) a package to another repo.\n\n This requires appropriate permissions for both the source\n repository/package and the destination repository.\n\n - OWNER/REPO/PACKAGE: Specify the OWNER namespace (i.e. user or org), the\n REPO name where the package is stored, and the PACKAGE name (slug) of the\n package itself. All separated by a slash.\n\n Example: 'your-org/awesome-repo/better-pkg'.\n\n - DEST: Specify the DEST (destination) repository to move the package to.\n This *must* be in the same namespace as the source repository.\n\n Example: 'other-repo'\n\n Full CLI example:\n\n $ cloudsmith mv your-org/awesome-repo/better-pkg other-repo\n \"\"\"\n owner, source, slug = owner_repo_package\n\n move_args = {\n \"slug\": click.style(slug, bold=True),\n \"source\": click.style(source, bold=True),\n \"dest\": click.style(destination, bold=True),\n }\n\n prompt = \"move the %(slug)s from %(source)s to %(dest)s\" % move_args\n if not utils.confirm_operation(prompt, assume_yes=yes):\n return\n\n click.echo(\n \"Moving %(slug)s package from %(source)s to %(dest)s ... \" % move_args, nl=False\n )\n\n context_msg = \"Failed to move package!\"\n with handle_api_exceptions(\n ctx, opts=opts, context_msg=context_msg, reraise_on_error=skip_errors\n ):\n with maybe_spinner(opts):\n _, new_slug = move_package(\n owner=owner, repo=source, identifier=slug, destination=destination\n )\n\n click.secho(\"OK\", fg=\"green\")\n\n if no_wait_for_sync:\n return\n\n wait_for_package_sync(\n ctx=ctx,\n opts=opts,\n owner=owner,\n repo=destination,\n slug=new_slug,\n wait_interval=wait_interval,\n skip_errors=skip_errors,\n attempts=sync_attempts,\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a requests session that retries some errors.", "response": "def create_requests_session(\n retries=None,\n backoff_factor=None,\n status_forcelist=None,\n pools_size=4,\n maxsize=4,\n ssl_verify=None,\n ssl_cert=None,\n proxy=None,\n session=None,\n):\n \"\"\"Create a requests session that retries some errors.\"\"\"\n # pylint: disable=too-many-branches\n config = Configuration()\n\n if retries is None:\n if config.error_retry_max is None:\n retries = 5\n else:\n retries = config.error_retry_max\n\n if backoff_factor is None:\n if config.error_retry_backoff is None:\n backoff_factor = 0.23\n else:\n backoff_factor = config.error_retry_backoff\n\n if status_forcelist is None:\n if config.error_retry_codes is None:\n status_forcelist = [500, 502, 503, 504]\n else:\n status_forcelist = config.error_retry_codes\n\n if ssl_verify is None:\n ssl_verify = config.verify_ssl\n\n if ssl_cert is None:\n if config.cert_file and config.key_file:\n ssl_cert = (config.cert_file, config.key_file)\n elif config.cert_file:\n ssl_cert = config.cert_file\n\n if proxy is None:\n proxy = Configuration().proxy\n\n session = session or requests.Session()\n session.verify = ssl_verify\n session.cert = ssl_cert\n\n if proxy:\n session.proxies = {\"http\": proxy, \"https\": proxy}\n\n retry = Retry(\n backoff_factor=backoff_factor,\n connect=retries,\n method_whitelist=False,\n read=retries,\n status_forcelist=tuple(status_forcelist),\n total=retries,\n )\n\n adapter = HTTPAdapter(\n max_retries=retry,\n pool_connections=pools_size,\n pool_maxsize=maxsize,\n pool_block=True,\n )\n\n session.mount(\"http://\", adapter)\n session.mount(\"https://\", adapter)\n\n return session"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef data(self):\n if self._data is None:\n self._data = self.response.content.decode(\"utf-8\")\n return self._data", "response": "Get the content for the response."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting a given response header.", "response": "def getheader(self, name, default=None):\n \"\"\"\n Return a given response header.\n \"\"\"\n return self.response.headers.get(name, default)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef request(\n self,\n method,\n url,\n query_params=None,\n headers=None,\n body=None,\n post_params=None,\n _preload_content=True,\n _request_timeout=None,\n ):\n \"\"\"\n :param method: http request method\n :param url: http request url\n :param query_params: query parameters in the url\n :param headers: http request headers\n :param body: request json body, for `application/json`\n :param post_params: request post parameters,\n `application/x-www-form-urlencoded`\n and `multipart/form-data`\n :param _preload_content: if False, the response object will be returned without\n reading/decoding response data. Default is True.\n :param _request_timeout: timeout setting for this request. If one number provided, it will be total request\n timeout. It can also be a pair (tuple) of (connection, read) timeouts.\n \"\"\"\n # Based on the RESTClientObject class generated by Swagger\n method = method.upper()\n assert method in [\"GET\", \"HEAD\", \"DELETE\", \"POST\", \"PUT\", \"PATCH\", \"OPTIONS\"]\n\n post_params = post_params or {}\n headers = headers or {}\n\n if \"Content-Type\" not in headers:\n headers[\"Content-Type\"] = \"application/json\"\n\n request_kwargs = {}\n\n if query_params:\n url += \"?\" + urlencode(query_params)\n\n if method in [\"POST\", \"PUT\", \"PATCH\", \"OPTIONS\", \"DELETE\"]:\n if re.search(\"json\", headers[\"Content-Type\"], re.IGNORECASE):\n request_body = None\n if body:\n request_body = json.dumps(body)\n request_kwargs[\"data\"] = request_body\n elif headers[\"Content-Type\"] == \"application/x-www-form-urlencoded\":\n request_kwargs[\"data\"] = post_params\n elif headers[\"Content-Type\"] == \"multipart/form-data\":\n del headers[\"Content-Type\"]\n request_kwargs[\"data\"] = post_params\n elif isinstance(body, str):\n request_kwargs[\"data\"] = body\n else:\n # Cannot generate the request from given parameters\n msg = \"\"\"Cannot prepare a request message for provided arguments.\n Please check that your arguments match declared content type.\"\"\"\n raise ApiException(status=0, reason=msg)\n\n try:\n resp = self.session.request(\n method,\n url,\n timeout=_request_timeout,\n stream=not _preload_content,\n headers=headers,\n **request_kwargs\n )\n except requests.exceptions.RequestException as exc:\n msg = \"{0}\\n{1}\".format(type(exc).__name__, str(exc))\n raise ApiException(status=0, reason=msg)\n\n resp.encoding = resp.apparent_encoding or \"utf-8\"\n rest_resp = RestResponse(resp)\n\n if _preload_content:\n logger.debug(\"response body: %s\", rest_resp.data)\n\n if not 200 <= rest_resp.status <= 299:\n raise ApiException(http_resp=rest_resp)\n\n return rest_resp", "response": "This method is used to make a request to the API."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_root_path():\n return os.path.realpath(os.path.join(os.path.dirname(__file__), os.pardir))", "response": "Get the root directory for the application."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef read_file(*path):\n real_path = os.path.realpath(os.path.join(*path))\n with click.open_file(real_path, \"r\") as fp:\n return fp.read()", "response": "Read the specific file into a string in its entirety."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalculating an MD5 hash for a file.", "response": "def calculate_file_md5(filepath, blocksize=2 ** 20):\n \"\"\"Calculate an MD5 hash for a file.\"\"\"\n checksum = hashlib.md5()\n\n with click.open_file(filepath, \"rb\") as f:\n\n def update_chunk():\n \"\"\"Add chunk to checksum.\"\"\"\n buf = f.read(blocksize)\n if buf:\n checksum.update(buf)\n return bool(buf)\n\n while update_chunk():\n pass\n\n return checksum.hexdigest()"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconstructs page and page size kwargs (if present ).", "response": "def get_page_kwargs(**kwargs):\n \"\"\"Construct page and page size kwargs (if present).\"\"\"\n page_kwargs = {}\n\n page = kwargs.get(\"page\")\n if page is not None and page > 0:\n page_kwargs[\"page\"] = page\n\n page_size = kwargs.get(\"page_size\")\n if page_size is not None and page_size > 0:\n page_kwargs[\"page_size\"] = page_size\n\n return page_kwargs"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_query_kwargs(**kwargs):\n query_kwargs = {}\n\n query = kwargs.pop(\"query\")\n if query:\n query_kwargs[\"query\"] = query\n\n return query_kwargs", "response": "Construct page and page size kwargs."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a PEP440 - compliant version derived from the git tag.", "response": "def _get_version_from_git_tag(path):\n \"\"\"Return a PEP440-compliant version derived from the git status.\n If that fails for any reason, return the changeset hash.\n \"\"\"\n m = GIT_DESCRIBE_REGEX.match(_git_describe_tags(path) or '')\n if m is None:\n return None\n version, post_commit, hash = m.groups()\n return version if post_commit == '0' else \"{0}.post{1}+{2}\".format(version, post_commit, hash)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_version(dunder_file):\n path = abspath(expanduser(dirname(dunder_file)))\n try:\n return _get_version_from_version_file(path) or _get_version_from_git_tag(path)\n except CalledProcessError as e:\n log.warn(repr(e))\n return None\n except Exception as e:\n log.exception(e)\n return None", "response": "Returns a version string for the current package derived\n from git or from a. version file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncount the time spent in some context", "response": "def ctime(message=None):\n \"Counts the time spent in some context\"\n global t_elapsed\n t_elapsed = 0.0\n print('\\n')\n t = time.time()\n yield\n if message:\n print(message + \": \", end='')\n t_elapsed = time.time() - t\n print(round(t_elapsed, 4), \"sec\")"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef rot1(angle, form='c'):\n cos_a = np.cos(angle)\n sin_a = np.sin(angle)\n rot_mat = np.identity(3)\n\n if form=='c':\n rot_mat[1, 1] = cos_a\n rot_mat[1, 2] = -sin_a\n rot_mat[2, 1] = sin_a\n rot_mat[2, 2] = cos_a\n elif form=='r':\n rot_mat[1, 1] = cos_a\n rot_mat[1, 2] = sin_a\n rot_mat[2, 1] = -sin_a\n rot_mat[2, 2] = cos_a\n else:\n print(\"Unknown input. 'r' or 'c' for row/column notation.\")\n return 1\n\n return rot_mat", "response": "Euler rotation about first axis"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting DCM to body Euler 3 - 1 - 3 angles", "response": "def dcm2body313(dcm):\n \"\"\"Convert DCM to body Euler 3-1-3 angles\n\n \"\"\"\n theta = np.zeros(3)\n\n theta[0] = np.arctan2(dcm[2, 0], dcm[2, 1])\n theta[1] = np.arccos(dcm[2, 2])\n theta[2] = np.arctan2(dcm[0, 2], -dcm[1, 2])\n\n return theta"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef hat_map(vec):\n vec = np.squeeze(vec)\n skew = np.array([\n [0, -vec[2], vec[1]],\n [vec[2], 0, -vec[0]],\n [-vec[1], vec[0], 0]])\n\n return skew", "response": "Return that hat map of a vector"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef vee_map(skew):\n\n vec = 1/2 * np.array([skew[2,1] - skew[1,2],\n skew[0,2] - skew[2,0],\n skew[1,0] - skew[0,1]])\n\n return vec", "response": "Return the vee map of a vector\n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dcmtoquat(dcm):\n quat = np.zeros(4)\n quat[-1] = 1/2*np.sqrt(np.trace(dcm)+1)\n quat[0:3] = 1/4/quat[-1]*vee_map(dcm-dcm.T)\n\n return quat", "response": "Convert DCM to quaternion"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts quaternion to DCM", "response": "def quattodcm(quat):\n \"\"\"Convert quaternion to DCM\n\n This function will convert a quaternion to the equivalent rotation matrix,\n or direction cosine matrix.\n\n Parameters:\n ----------\n quat - (4,) numpy array\n Array defining a quaterion where the quaternion is defined in terms of\n a vector and a scalar part. The vector is related to the eigen axis\n and equivalent in both reference frames [x y z w]\n\n Returns:\n --------\n dcm - (3,3) numpy array\n Numpy rotation matrix which defines a rotation from the b to a frame\n \"\"\"\n\n dcm = (quat[-1]**2-np.inner(quat[0:3], quat[0:3]))*np.eye(3,3) + 2*np.outer(quat[0:3],quat[0:3]) + 2*quat[-1]*hat_map(quat[0:3])\n\n return dcm"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert a rotation matrix to angular velocity w Omega", "response": "def dcmdottoang_vel(R,Rdot):\n \"\"\"Convert a rotation matrix to angular velocity\n \n w - angular velocity in inertial frame\n Omega - angular velocity in body frame\n \"\"\"\n w = vee_map(Rdot.dot(R.T))\n Omega = vee_map(R.T.dot(Rdot))\n\n return (w, Omega)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncomputing kinematics for axis angle representation", "response": "def ang_veltoaxisangledot(angle, axis, Omega):\n \"\"\"Compute kinematics for axis angle representation\n\n \"\"\"\n angle_dot = axis.dot(Omega)\n axis_dot = 1/2*(hat_map(axis) - 1/np.tan(angle/2) * hat_map(axis).dot(hat_map(axis))).dot(Omega)\n return angle_dot, axis_dot"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert axis angle represetnation to angular velocity in body frame", "response": "def axisangledottoang_vel(angle,axis, angle_dot,axis_dot):\n \"\"\"Convert axis angle represetnation to angular velocity in body frame\n\n \"\"\"\n \n Omega = angle_dot*axis + np.sin(angle)*axis_dot - (1-np.cos(angle))*hat_map(axis).dot(axis_dot)\n return Omega"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef normalize(num_in, lower=0, upper=360, b=False):\n if lower >= upper:\n ValueError(\"lower must be lesser than upper\")\n if not b:\n if not ((lower + upper == 0) or (lower == 0)):\n raise ValueError('When b=False lower=0 or range must be symmetric about 0.')\n else:\n if not (lower + upper == 0):\n raise ValueError('When b=True range must be symmetric about 0.')\n\n # abs(num + upper) and abs(num - lower) are needed, instead of\n # abs(num), since the lower and upper limits need not be 0. We need\n # to add half size of the range, so that the final result is lower +\n # or upper - , respectively.\n if not hasattr(num_in, \"__iter__\"):\n num_in = np.asarray([num_in], dtype=np.float)\n\n res = []\n for num in num_in:\n if not b:\n if num > upper or num == lower:\n num = lower + abs(num + upper) % (abs(lower) + abs(upper))\n if num < lower or num == upper:\n num = upper - abs(num - lower) % (abs(lower) + abs(upper))\n\n res.append(lower if num == upper else num)\n else:\n total_length = abs(lower) + abs(upper)\n if num < -total_length:\n num += ceil(num / (-2 * total_length)) * 2 * total_length\n if num > total_length:\n num -= floor(num / (2 * total_length)) * 2 * total_length\n if num > upper:\n num = total_length - num\n if num < lower:\n num = -total_length - num\n\n res.append(num)\n\n return np.asarray(res, dtype=np.float)", "response": "Normalizes a number to range [ lower upper ) or [ lower upper )."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef list_repos(owner=None, **kwargs):\n client = get_repos_api()\n\n api_kwargs = {}\n api_kwargs.update(utils.get_page_kwargs(**kwargs))\n\n # pylint: disable=fixme\n # FIXME: Compatibility code until we work out how to conflate\n # the overlapping repos_list methods into one.\n repos_list = client.repos_list_with_http_info\n\n if owner is not None:\n api_kwargs[\"owner\"] = owner\n if hasattr(client, \"repos_list0_with_http_info\"):\n # pylint: disable=no-member\n repos_list = client.repos_list0_with_http_info\n else:\n if hasattr(client, \"repos_all_list_with_http_info\"):\n # pylint: disable=no-member\n repos_list = client.repos_all_list_with_http_info\n\n with catch_raise_api_exception():\n res, _, headers = repos_list(**api_kwargs)\n\n ratelimits.maybe_rate_limit(client, headers)\n page_info = PageInfo.from_headers(headers)\n return [x.to_dict() for x in res], page_info", "response": "List repositories in a namespace."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncopy a package from one repository to another.", "response": "def copy(\n ctx,\n opts,\n owner_repo_package,\n destination,\n skip_errors,\n wait_interval,\n no_wait_for_sync,\n sync_attempts,\n):\n \"\"\"\n Copy a package to another repository.\n\n This requires appropriate permissions for both the source\n repository/package and the destination repository.\n\n - OWNER/REPO/PACKAGE: Specify the OWNER namespace (i.e. user or org), the\n REPO name where the package is stored, and the PACKAGE name (slug) of the\n package itself. All separated by a slash.\n\n Example: 'your-org/awesome-repo/better-pkg'.\n\n - DEST: Specify the DEST (destination) repository to copy the package to.\n This *must* be in the same namespace as the source repository.\n\n Example: 'other-repo'\n\n Full CLI example:\n\n $ cloudsmith cp your-org/awesome-repo/better-pkg other-repo\n \"\"\"\n owner, source, slug = owner_repo_package\n\n click.echo(\n \"Copying %(slug)s package from %(source)s to %(dest)s ... \"\n % {\n \"slug\": click.style(slug, bold=True),\n \"source\": click.style(source, bold=True),\n \"dest\": click.style(destination, bold=True),\n },\n nl=False,\n )\n\n context_msg = \"Failed to copy package!\"\n with handle_api_exceptions(\n ctx, opts=opts, context_msg=context_msg, reraise_on_error=skip_errors\n ):\n with maybe_spinner(opts):\n _, new_slug = copy_package(\n owner=owner, repo=source, identifier=slug, destination=destination\n )\n\n click.secho(\"OK\", fg=\"green\")\n\n if no_wait_for_sync:\n return\n\n wait_for_package_sync(\n ctx=ctx,\n opts=opts,\n owner=owner,\n repo=destination,\n slug=new_slug,\n wait_interval=wait_interval,\n skip_errors=skip_errors,\n attempts=sync_attempts,\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rm_file_or_dir(path, ignore_errors=True):\n if os.path.exists(path):\n if os.path.isdir(path):\n if os.path.islink(path):\n os.unlink(path)\n else:\n shutil.rmtree(path, ignore_errors=ignore_errors)\n else:\n if os.path.islink(path):\n os.unlink(path)\n else:\n os.remove(path)", "response": "Helper function to remove a certain filepath from the system."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchecking whether the column has a factorization that exists and is not older than the source", "response": "def cache_valid(self, col):\n \"\"\"\n Checks whether the column has a factorization that exists and is not older than the source\n\n :param col:\n :return:\n \"\"\"\n cache_valid = False\n\n if self.rootdir:\n col_org_file_check = self[col].rootdir + '/__attrs__'\n col_values_file_check = self[col].rootdir + '.values/__attrs__'\n cache_valid = os.path.exists(col_org_file_check) and os.path.exists(col_values_file_check)\n\n return cache_valid"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef group_cache_valid(self, col_list):\n cache_valid = False\n\n if self.rootdir:\n col_values_file_check = os.path.join(self.rootdir, self.create_group_base_name(col_list)) + \\\n '.values/__attrs__'\n\n exists_group_index = os.path.exists(col_values_file_check)\n missing_col_check = [1 for col in col_list if not os.path.exists(self[col].rootdir + '/__attrs__')]\n cache_valid = (exists_group_index and not missing_col_check)\n\n return cache_valid", "response": "Checks whether the group has a factorization that exists and is not older than the source ArcGIS group."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nexisting todos here are: these should be hidden helper carrays As in: not normal columns that you would normally see as a user The factor (label index) carray is as long as the original carray (and the rest of the table therefore) But the (unique) values carray is not as long (as long as the number of unique values) :param col_list: :param refresh: :return:", "response": "def cache_factor(self, col_list, refresh=False):\n \"\"\"\n Existing todos here are: these should be hidden helper carrays\n As in: not normal columns that you would normally see as a user\n\n The factor (label index) carray is as long as the original carray\n (and the rest of the table therefore)\n But the (unique) values carray is not as long (as long as the number\n of unique values)\n\n :param col_list:\n :param refresh:\n :return:\n \"\"\"\n\n if not self.rootdir:\n raise TypeError('Only out-of-core ctables can have '\n 'factorization caching at the moment')\n\n if not isinstance(col_list, list):\n col_list = [col_list]\n\n if refresh:\n kill_list = [x for x in os.listdir(self.rootdir) if '.factor' in x or '.values' in x]\n for kill_dir in kill_list:\n rm_file_or_dir(os.path.join(self.rootdir, kill_dir))\n\n for col in col_list:\n\n # create cache if needed\n if refresh or not self.cache_valid(col):\n # todo: also add locking mechanism here\n\n # create directories\n col_rootdir = self[col].rootdir\n col_factor_rootdir = col_rootdir + '.factor'\n col_factor_rootdir_tmp = tempfile.mkdtemp(prefix='bcolz-')\n col_values_rootdir = col_rootdir + '.values'\n col_values_rootdir_tmp = tempfile.mkdtemp(prefix='bcolz-')\n\n # create factor\n carray_factor = \\\n bcolz.carray([], dtype='int64', expectedlen=self.size,\n rootdir=col_factor_rootdir_tmp, mode='w')\n _, values = \\\n ctable_ext.factorize(self[col], labels=carray_factor)\n carray_factor.flush()\n\n rm_file_or_dir(col_factor_rootdir, ignore_errors=True)\n shutil.move(col_factor_rootdir_tmp, col_factor_rootdir)\n\n # create values\n carray_values = \\\n bcolz.carray(np.fromiter(values.values(), dtype=self[col].dtype),\n rootdir=col_values_rootdir_tmp, mode='w')\n carray_values.flush()\n rm_file_or_dir(col_values_rootdir, ignore_errors=True)\n shutil.move(col_values_rootdir_tmp, col_values_rootdir)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef unique(self, col_or_col_list):\n\n if isinstance(col_or_col_list, list):\n col_is_list = True\n col_list = col_or_col_list\n else:\n col_is_list = False\n col_list = [col_or_col_list]\n\n output = []\n\n for col in col_list:\n\n if self.auto_cache or self.cache_valid(col):\n # create factorization cache\n if not self.cache_valid(col):\n self.cache_factor([col])\n\n # retrieve values from existing disk-based factorization\n col_values_rootdir = self[col].rootdir + '.values'\n carray_values = bcolz.carray(rootdir=col_values_rootdir, mode='r')\n values = list(carray_values)\n else:\n # factorize on-the-fly\n _, values = ctable_ext.factorize(self[col])\n values = values.values()\n\n output.append(values)\n\n if not col_is_list:\n output = output[0]\n\n return output", "response": "Return a list of unique values of a column or a list of lists of column list\n "} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef aggregate_groups(self, ct_agg, nr_groups, skip_key,\n carray_factor, groupby_cols, agg_ops,\n dtype_dict, bool_arr=None):\n '''Perform aggregation and place the result in the given ctable.\n\n Args:\n ct_agg (ctable): the table to hold the aggregation\n nr_groups (int): the number of groups (number of rows in output table)\n skip_key (int): index of the output row to remove from results (used for filtering)\n carray_factor: the carray for each row in the table a reference to the the unique group index\n groupby_cols: the list of 'dimension' columns that are used to perform the groupby over\n output_agg_ops (list): list of tuples of the form: (input_col, agg_op)\n input_col (string): name of the column to act on\n agg_op (int): aggregation operation to perform\n bool_arr: a boolean array containing the filter\n\n '''\n\n # this creates the groupby columns\n for col in groupby_cols:\n\n result_array = ctable_ext.groupby_value(self[col], carray_factor,\n nr_groups, skip_key)\n\n if bool_arr is not None:\n result_array = np.delete(result_array, skip_key)\n\n ct_agg.addcol(result_array, name=col)\n del result_array\n\n # this creates the aggregation columns\n for input_col_name, output_col_name, agg_op in agg_ops:\n\n input_col = self[input_col_name]\n output_col_dtype = dtype_dict[output_col_name]\n\n input_buffer = np.empty(input_col.chunklen, dtype=input_col.dtype)\n output_buffer = np.zeros(nr_groups, dtype=output_col_dtype)\n\n if agg_op == 'sum':\n ctable_ext.aggregate_sum(input_col, carray_factor, nr_groups,\n skip_key, input_buffer, output_buffer)\n elif agg_op == 'mean':\n ctable_ext.aggregate_mean(input_col, carray_factor, nr_groups,\n skip_key, input_buffer, output_buffer)\n elif agg_op == 'std':\n ctable_ext.aggregate_std(input_col, carray_factor, nr_groups,\n skip_key, input_buffer, output_buffer)\n elif agg_op == 'count':\n ctable_ext.aggregate_count(input_col, carray_factor, nr_groups,\n skip_key, input_buffer, output_buffer)\n elif agg_op == 'count_distinct':\n ctable_ext.aggregate_count_distinct(input_col, carray_factor, nr_groups,\n skip_key, input_buffer, output_buffer)\n elif agg_op == 'sorted_count_distinct':\n ctable_ext.aggregate_sorted_count_distinct(input_col, carray_factor, nr_groups,\n skip_key, input_buffer, output_buffer)\n else:\n raise KeyError('Unknown aggregation operation ' + str(agg_op))\n\n if bool_arr is not None:\n output_buffer = np.delete(output_buffer, skip_key)\n\n ct_agg.addcol(output_buffer, name=output_col_name)\n del output_buffer\n\n ct_agg.delcol('tmp_col_bquery__')", "response": "Perform aggregation and place the result in the given ctable."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef factorize_groupby_cols(self, groupby_cols):\n # first check if the factorized arrays already exist\n # unless we need to refresh the cache\n factor_list = []\n values_list = []\n\n # factorize the groupby columns\n for col in groupby_cols:\n\n if self.auto_cache or self.cache_valid(col):\n # create factorization cache if needed\n if not self.cache_valid(col):\n self.cache_factor([col])\n\n col_rootdir = self[col].rootdir\n col_factor_rootdir = col_rootdir + '.factor'\n col_values_rootdir = col_rootdir + '.values'\n col_carray_factor = \\\n bcolz.carray(rootdir=col_factor_rootdir, mode='r')\n col_carray_values = \\\n bcolz.carray(rootdir=col_values_rootdir, mode='r')\n else:\n col_carray_factor, values = ctable_ext.factorize(self[col])\n col_carray_values = \\\n bcolz.carray(np.fromiter(values.values(), dtype=self[col].dtype))\n\n factor_list.append(col_carray_factor)\n values_list.append(col_carray_values)\n\n return factor_list, values_list", "response": "factorizes all columns that are used in the groupby\n "} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create_group_column_factor(self, factor_list, groupby_cols, cache=False):\n if not self.rootdir:\n # in-memory scenario\n input_rootdir = None\n col_rootdir = None\n col_factor_rootdir = None\n col_values_rootdir = None\n col_factor_rootdir_tmp = None\n col_values_rootdir_tmp = None\n else:\n # temporary\n input_rootdir = tempfile.mkdtemp(prefix='bcolz-')\n col_factor_rootdir_tmp = tempfile.mkdtemp(prefix='bcolz-')\n col_values_rootdir_tmp = tempfile.mkdtemp(prefix='bcolz-')\n\n # create combination of groupby columns\n group_array = bcolz.zeros(0, dtype=np.int64, expectedlen=len(self), rootdir=input_rootdir, mode='w')\n factor_table = bcolz.ctable(factor_list, names=groupby_cols)\n ctable_iter = factor_table.iter(outcols=groupby_cols, out_flavor=tuple)\n ctable_ext.create_group_index(ctable_iter, len(groupby_cols), group_array)\n\n # now factorize the results\n carray_factor = \\\n bcolz.carray([], dtype='int64', expectedlen=self.size, rootdir=col_factor_rootdir_tmp, mode='w')\n carray_factor, values = ctable_ext.factorize(group_array, labels=carray_factor)\n carray_factor.flush()\n\n carray_values = \\\n bcolz.carray(np.fromiter(values.values(), dtype=np.int64), rootdir=col_values_rootdir_tmp, mode='w')\n carray_values.flush()\n\n del group_array\n if cache:\n # clean up the temporary file\n rm_file_or_dir(input_rootdir, ignore_errors=True)\n\n if cache:\n # official end destination\n col_rootdir = os.path.join(self.rootdir, self.create_group_base_name(groupby_cols))\n col_factor_rootdir = col_rootdir + '.factor'\n col_values_rootdir = col_rootdir + '.values'\n lock_file = col_rootdir + '.lock'\n\n # only works for linux\n if not os.path.exists(lock_file):\n uid = str(uuid.uuid4())\n try:\n with open(lock_file, 'a+') as fn:\n fn.write(uid + '\\n')\n with open(lock_file, 'r') as fn:\n temp = fn.read().splitlines()\n if temp[0] == uid:\n lock = True\n else:\n lock = False\n del temp\n except:\n lock = False\n else:\n lock = False\n\n if lock:\n rm_file_or_dir(col_factor_rootdir, ignore_errors=False)\n shutil.move(col_factor_rootdir_tmp, col_factor_rootdir)\n carray_factor = bcolz.carray(rootdir=col_factor_rootdir, mode='r')\n\n rm_file_or_dir(col_values_rootdir, ignore_errors=False)\n shutil.move(col_values_rootdir_tmp, col_values_rootdir)\n carray_values = bcolz.carray(rootdir=col_values_rootdir, mode='r')\n else:\n # another process has a lock, we will work with our current files and clean up later\n self._dir_clean_list.append(col_factor_rootdir)\n self._dir_clean_list.append(col_values_rootdir)\n\n return carray_factor, carray_values", "response": "Create a unique factorized column out of several individual columns."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate unique groups for groupby loop", "response": "def make_group_index(self, groupby_cols, bool_arr):\n '''Create unique groups for groupby loop\n\n Args:\n factor_list:\n values_list:\n groupby_cols:\n bool_arr:\n\n Returns:\n carray: (carray_factor)\n int: (nr_groups) the number of resulting groups\n int: (skip_key)\n '''\n factor_list, values_list = self.factorize_groupby_cols(groupby_cols)\n\n # create unique groups for groupby loop\n if len(factor_list) == 0:\n # no columns to groupby over, so directly aggregate the measure\n # columns to 1 total\n tmp_rootdir = self.create_tmp_rootdir()\n carray_factor = bcolz.zeros(len(self), dtype='int64', rootdir=tmp_rootdir, mode='w')\n carray_values = ['Total']\n elif len(factor_list) == 1:\n # single column groupby, the groupby output column\n # here is 1:1 to the values\n carray_factor = factor_list[0]\n carray_values = values_list[0]\n else:\n # multi column groupby\n # first combine the factorized columns to single values\n if self.group_cache_valid(col_list=groupby_cols):\n # there is a group cache that we can use\n col_rootdir = os.path.join(self.rootdir, self.create_group_base_name(groupby_cols))\n col_factor_rootdir = col_rootdir + '.factor'\n carray_factor = bcolz.carray(rootdir=col_factor_rootdir)\n col_values_rootdir = col_rootdir + '.values'\n carray_values = bcolz.carray(rootdir=col_values_rootdir)\n else:\n # create a brand new groupby col combination\n carray_factor, carray_values = \\\n self.create_group_column_factor(factor_list, groupby_cols, cache=self.auto_cache)\n\n nr_groups = len(carray_values)\n skip_key = None\n\n if bool_arr is not None:\n # make all non relevant combinations -1\n tmp_rootdir = self.create_tmp_rootdir()\n carray_factor = bcolz.eval(\n '(factor + 1) * bool - 1',\n user_dict={'factor': carray_factor, 'bool': bool_arr}, rootdir=tmp_rootdir, mode='w')\n # now check how many unique values there are left\n tmp_rootdir = self.create_tmp_rootdir()\n labels = bcolz.carray([], dtype='int64', expectedlen=len(carray_factor), rootdir=tmp_rootdir, mode='w')\n carray_factor, values = ctable_ext.factorize(carray_factor, labels)\n # values might contain one value too much (-1) (no direct lookup\n # possible because values is a reversed dict)\n filter_check = \\\n [key for key, value in values.items() if value == -1]\n if filter_check:\n skip_key = filter_check[0]\n # the new nr of groups depends on the outcome after filtering\n nr_groups = len(values)\n\n # using nr_groups as a total length might be one one off due to the skip_key\n # (skipping a row in aggregation)\n # but that is okay normally\n\n if skip_key is None:\n # if we shouldn't skip a row, set it at the first row after the total number of groups\n skip_key = nr_groups\n\n return carray_factor, nr_groups, skip_key"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a temporary rootdir that we can destroy later again", "response": "def create_tmp_rootdir(self):\n \"\"\"\n create a rootdir that we can destroy later again\n\n Returns\n -------\n\n \"\"\"\n if self.rootdir:\n tmp_rootdir = tempfile.mkdtemp(prefix='bcolz-')\n self._dir_clean_list.append(tmp_rootdir)\n else:\n tmp_rootdir = None\n return tmp_rootdir"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nclean up all used temporary rootdirs", "response": "def clean_tmp_rootdir(self):\n \"\"\"\n clean up all used temporary rootdirs\n\n Returns\n -------\n\n \"\"\"\n for tmp_rootdir in list(self._dir_clean_list):\n rm_file_or_dir(tmp_rootdir)\n self._dir_clean_list.remove(tmp_rootdir)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a container for the output table a dictionary describing it s columns and a list of tuples describing aggregation operations to perform.", "response": "def create_agg_ctable(self, groupby_cols, agg_list, expectedlen, rootdir):\n '''Create a container for the output table, a dictionary describing it's\n columns and a list of tuples describing aggregation\n operations to perform.\n\n Args:\n groupby_cols (list): a list of columns to groupby over\n agg_list (list): the aggregation operations (see groupby for more info)\n expectedlen (int): expected length of output table\n rootdir (string): the directory to write the table to\n\n Returns:\n ctable: A table in the correct format for containing the output of\n the specified aggregation operations.\n dict: (dtype_dict) dictionary describing columns to create\n list: (agg_ops) list of tuples of the form:\n (input_col_name, output_col_name, agg_op)\n input_col_name (string): name of the column to act on\n output_col_name (string): name of the column to output to\n agg_op (int): aggregation operation to perform\n '''\n dtype_dict = {}\n\n # include all the groupby columns\n for col in groupby_cols:\n dtype_dict[col] = self[col].dtype\n\n agg_ops_list = ['sum', 'count', 'count_distinct', 'sorted_count_distinct', 'mean', 'std']\n agg_ops = []\n\n for agg_info in agg_list:\n\n if not isinstance(agg_info, list):\n # example: ['m1', 'm2', ...]\n # default operation (sum) and default output column name (same is input)\n output_col_name = agg_info\n input_col_name = agg_info\n agg_op = 'sum'\n else:\n input_col_name = agg_info[0]\n agg_op = agg_info[1]\n\n if len(agg_info) == 2:\n # example: [['m1', 'sum'], ['m2', 'mean], ...]\n # default output column name\n output_col_name = input_col_name\n else:\n # example: [['m1', 'sum', 'mnew1'], ['m1, 'mean','mnew2'], ...]\n # fully specified\n output_col_name = agg_info[2]\n\n if agg_op not in agg_ops_list:\n raise NotImplementedError(\n 'Unknown Aggregation Type: ' + str(agg_op))\n\n # choose output column dtype based on aggregation operation and\n # input column dtype\n # TODO: check if the aggregation columns is numeric\n # NB: we could build a concatenation for strings like pandas, but I would really prefer to see that as a\n # separate operation\n if agg_op in ('count', 'count_distinct', 'sorted_count_distinct'):\n output_col_dtype = np.dtype(np.int64)\n elif agg_op in ('mean', 'std'):\n output_col_dtype = np.dtype(np.float64)\n else:\n output_col_dtype = self[input_col_name].dtype\n\n dtype_dict[output_col_name] = output_col_dtype\n\n # save output\n agg_ops.append((input_col_name, output_col_name, agg_op))\n\n # create aggregation table\n ct_agg = bcolz.ctable(\n np.zeros(expectedlen, [('tmp_col_bquery__', np.bool)]),\n expectedlen=expectedlen,\n rootdir=rootdir)\n\n return ct_agg, dtype_dict, agg_ops"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a boolean array where the terms in term_list are true.", "response": "def where_terms(self, term_list, cache=False):\n \"\"\"\n Create a boolean array where `term_list` is true.\n A terms list has a [(col, operator, value), ..] construction.\n Eg. [('sales', '>', 2), ('state', 'in', ['IL', 'AR'])]\n\n :param term_list:\n :param outcols:\n :param limit:\n :param skip:\n :return: :raise ValueError:\n \"\"\"\n\n if type(term_list) not in [list, set, tuple]:\n raise ValueError(\"Only term lists are supported\")\n\n col_list = []\n op_list = []\n value_list = []\n\n for term in term_list:\n # get terms\n filter_col = term[0]\n filter_operator = term[1].lower().strip(' ')\n filter_value = term[2]\n\n # check values\n if filter_col not in self.cols:\n raise KeyError(unicode(filter_col) + ' not in table')\n\n if filter_operator in ['==', 'eq']:\n op_id = 1\n elif filter_operator in ['!=', 'neq']:\n op_id = 2\n elif filter_operator in ['in']:\n op_id = 3\n elif filter_operator in ['nin', 'not in']:\n op_id = 4\n elif filter_operator in ['>']:\n op_id = 5\n elif filter_operator in ['>=']:\n op_id = 6\n elif filter_operator in ['<']:\n op_id = 7\n elif filter_operator in ['<=']:\n op_id = 8\n else:\n raise KeyError(unicode(filter_operator) + ' is not an accepted operator for filtering')\n\n if op_id in [3, 4]:\n if type(filter_value) not in [list, set, tuple]:\n raise ValueError(\"In selections need lists, sets or tuples\")\n\n if len(filter_value) < 1:\n raise ValueError(\"A value list needs to have values\")\n\n # optimize lists of 1 value\n if len(filter_value) == 1:\n if op_id == 3:\n op_id = 1\n else:\n op_id = 2\n\n filter_value = filter_value[0]\n else:\n filter_value = set(filter_value)\n\n # prepare input for filter creation\n col_list.append(filter_col)\n op_list.append(op_id)\n value_list.append(filter_value)\n\n # rootdir\n if cache:\n # nb: this directory is not destroyed until the end of the groupby\n rootdir = self.create_tmp_rootdir()\n else:\n rootdir = None\n\n # create boolean array and fill it\n boolarr = bcolz.carray(np.ones(0, dtype=np.bool), expectedlen=self.len, rootdir=rootdir, mode='w')\n ctable_iter = self[col_list].iter(out_flavor='tuple')\n ctable_ext.apply_where_terms(ctable_iter, op_list, value_list, boolarr)\n\n return boolarr"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nchecking for where terms are applicable", "response": "def where_terms_factorization_check(self, term_list):\n \"\"\"\n check for where terms if they are applicable\n Create a boolean array where `term_list` is true.\n A terms list has a [(col, operator, value), ..] construction.\n Eg. [('sales', '>', 2), ('state', 'in', ['IL', 'AR'])]\n\n :param term_list:\n :param outcols:\n :param limit:\n :param skip:\n :return: :raise ValueError:\n \"\"\"\n\n if type(term_list) not in [list, set, tuple]:\n raise ValueError(\"Only term lists are supported\")\n\n valid = True\n\n for term in term_list:\n # get terms\n filter_col = term[0]\n filter_operator = term[1].lower().strip(' ')\n filter_value = term[2]\n\n # check values\n if filter_col not in self.cols:\n raise KeyError(unicode(filter_col) + ' not in table')\n\n col_values_rootdir = os.path.join(self.rootdir, filter_col + '.values')\n\n if not os.path.exists(col_values_rootdir):\n # no factorization available\n break\n\n col_carray = bcolz.carray(rootdir=col_values_rootdir, mode='r')\n col_values = set(col_carray)\n\n if filter_operator in ['in', 'not in', 'nin']:\n if type(filter_value) not in [list, set, tuple]:\n raise ValueError(\"In selections need lists, sets or tuples\")\n if len(filter_value) < 1:\n raise ValueError(\"A value list needs to have values\")\n\n # optimize lists of 1 value\n if len(filter_value) == 1:\n filter_value = filter_value[0]\n if filter_operator == 'in':\n filter_operator = '=='\n else:\n filter_operator = '!='\n else:\n filter_value = set(filter_value)\n\n if filter_operator in ['==', 'eq']:\n valid = filter_value in col_values\n elif filter_operator in ['!=', 'neq']:\n valid = any(val for val in col_values if val != filter_value)\n elif filter_operator in ['in']:\n valid = any(val for val in filter_value if val in col_values)\n elif filter_operator in ['nin', 'not in']:\n valid = any(val for val in col_values if val not in filter_value)\n elif filter_operator in ['>']:\n valid = any(val for val in col_values if val > filter_value)\n elif filter_operator in ['>=']:\n valid = any(val for val in col_values if val >= filter_value)\n elif filter_operator in ['<']:\n valid = any(val for val in col_values if val < filter_value)\n elif filter_operator in ['<=']:\n valid = any(val for val in col_values if val >= filter_value)\n else:\n raise KeyError(str(filter_operator) + ' is not an accepted operator for filtering')\n\n # if one of the filters is blocking, we can stop\n if not valid:\n break\n\n return valid"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef is_in_ordered_subgroups(self, basket_col=None, bool_arr=None,\n _max_len_subgroup=1000):\n \"\"\"\n Expands the filter using a specified column\n\n Parameters\n ----------\n basket_col\n bool_arr\n _max_len_subgroup\n\n Returns\n -------\n\n \"\"\"\n assert basket_col is not None\n\n if bool_arr is None:\n return None\n\n if self.auto_cache and bool_arr.rootdir is not None:\n rootdir = self.create_tmp_rootdir()\n else:\n rootdir = None\n\n return \\\n ctable_ext.is_in_ordered_subgroups(\n self[basket_col], bool_arr=bool_arr, rootdir=rootdir,\n _max_len_subgroup=_max_len_subgroup)", "response": "Returns True if the entry is in the ordered subgroups of the entry."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef with_metaclass(Type, skip_attrs=set(('__dict__', '__weakref__'))):\n\n def _clone_with_metaclass(Class):\n attrs = dict((key, value) for key, value in items(vars(Class))\n if key not in skip_attrs)\n return Type(Class.__name__, Class.__bases__, attrs)\n\n return _clone_with_metaclass", "response": "Class decorator to set metaclass for base classes."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nyielding length - sized tuples from the given collection.", "response": "def ensure_tuple(length, tuples):\n\t\"\"\"Yield `length`-sized tuples from the given collection.\n\t\n\tWill truncate longer tuples to the desired length, and pad using the leading element if shorter.\n\t\"\"\"\n\t\n\tfor elem in tuples:\n\t\t# Handle non-tuples and non-lists as a single repeated element.\n\t\tif not isinstance(elem, (tuple, list)):\n\t\t\tyield (elem, ) * length\n\t\t\tcontinue\n\t\t\n\t\tl = len(elem)\n\t\t\n\t\t# If we have the correct length already, yield it.\n\t\tif l == length:\n\t\t\tyield elem\n\t\t\n\t\t# If we're too long, truncate.\n\t\telif l > length:\n\t\t\tyield tuple(elem[:length])\n\t\t\n\t\t# If we're too short, pad the *leading* element out.\n\t\telif l < length:\n\t\t\tyield (elem[0], ) * (length - l) + tuple(elem)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nprinting the environment versions.", "response": "def print_version():\n \"\"\"Print the environment versions.\"\"\"\n click.echo(\"Versions:\")\n click.secho(\n \"CLI Package Version: %(version)s\"\n % {\"version\": click.style(get_cli_version(), bold=True)}\n )\n click.secho(\n \"API Package Version: %(version)s\"\n % {\"version\": click.style(get_api_version(), bold=True)}\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nhandle entrypoint to CLI.", "response": "def main(ctx, opts, version):\n \"\"\"Handle entrypoint to CLI.\"\"\"\n # pylint: disable=unused-argument\n if version:\n print_version()\n elif ctx.invoked_subcommand is None:\n click.echo(ctx.get_help())"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef validate_upload_file(ctx, opts, owner, repo, filepath, skip_errors):\n filename = click.format_filename(filepath)\n basename = os.path.basename(filename)\n\n click.echo(\n \"Checking %(filename)s file upload parameters ... \"\n % {\"filename\": click.style(basename, bold=True)},\n nl=False,\n )\n\n context_msg = \"Failed to validate upload parameters!\"\n with handle_api_exceptions(\n ctx, opts=opts, context_msg=context_msg, reraise_on_error=skip_errors\n ):\n with maybe_spinner(opts):\n md5_checksum = validate_request_file_upload(\n owner=owner, repo=repo, filepath=filename\n )\n\n click.secho(\"OK\", fg=\"green\")\n\n return md5_checksum", "response": "Validate parameters for requesting a file upload."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nuploads a file via the API.", "response": "def upload_file(ctx, opts, owner, repo, filepath, skip_errors, md5_checksum):\n \"\"\"Upload a package file via the API.\"\"\"\n filename = click.format_filename(filepath)\n basename = os.path.basename(filename)\n\n click.echo(\n \"Requesting file upload for %(filename)s ... \"\n % {\"filename\": click.style(basename, bold=True)},\n nl=False,\n )\n\n context_msg = \"Failed to request file upload!\"\n with handle_api_exceptions(\n ctx, opts=opts, context_msg=context_msg, reraise_on_error=skip_errors\n ):\n with maybe_spinner(opts):\n identifier, upload_url, upload_fields = request_file_upload(\n owner=owner, repo=repo, filepath=filename, md5_checksum=md5_checksum\n )\n\n click.secho(\"OK\", fg=\"green\")\n\n context_msg = \"Failed to upload file!\"\n with handle_api_exceptions(ctx, opts=opts, context_msg=context_msg):\n filesize = utils.get_file_size(filepath=filename)\n\n label = \"Uploading %(filename)s:\" % {\n \"filename\": click.style(basename, bold=True)\n }\n\n with click.progressbar(\n length=filesize,\n label=label,\n fill_char=click.style(\"#\", fg=\"green\"),\n empty_char=click.style(\"-\", fg=\"red\"),\n ) as pb:\n\n def progress_callback(monitor):\n pb.update(monitor.bytes_read)\n\n api_upload_file(\n upload_url=upload_url,\n upload_fields=upload_fields,\n filepath=filename,\n callback=progress_callback,\n )\n\n return identifier"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef validate_create_package(\n ctx, opts, owner, repo, package_type, skip_errors, **kwargs\n):\n \"\"\"Check new package parameters via the API.\"\"\"\n click.echo(\n \"Checking %(package_type)s package upload parameters ... \"\n % {\"package_type\": click.style(package_type, bold=True)},\n nl=False,\n )\n\n context_msg = \"Failed to validate upload parameters!\"\n with handle_api_exceptions(\n ctx, opts=opts, context_msg=context_msg, reraise_on_error=skip_errors\n ):\n with maybe_spinner(opts):\n api_validate_create_package(\n package_format=package_type, owner=owner, repo=repo, **kwargs\n )\n\n click.secho(\"OK\", fg=\"green\")\n return True", "response": "Check new package parameters via the API."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create_package(ctx, opts, owner, repo, package_type, skip_errors, **kwargs):\n click.echo(\n \"Creating a new %(package_type)s package ... \"\n % {\"package_type\": click.style(package_type, bold=True)},\n nl=False,\n )\n\n context_msg = \"Failed to create package!\"\n with handle_api_exceptions(\n ctx, opts=opts, context_msg=context_msg, reraise_on_error=skip_errors\n ):\n with maybe_spinner(opts):\n slug_perm, slug = api_create_package(\n package_format=package_type, owner=owner, repo=repo, **kwargs\n )\n\n click.secho(\"OK\", fg=\"green\")\n\n click.echo(\n \"Created: %(owner)s/%(repo)s/%(slug)s (%(slug_perm)s)\"\n % {\n \"owner\": click.style(owner, fg=\"magenta\"),\n \"repo\": click.style(repo, fg=\"magenta\"),\n \"slug\": click.style(slug, fg=\"green\"),\n \"slug_perm\": click.style(slug_perm, bold=True),\n }\n )\n\n return slug_perm, slug", "response": "Create a new package via the API."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nwaiting for a package to be synced.", "response": "def wait_for_package_sync(\n ctx, opts, owner, repo, slug, wait_interval, skip_errors, attempts=3\n):\n \"\"\"Wait for a package to synchronise (or fail).\"\"\"\n # pylint: disable=too-many-locals\n attempts -= 1\n click.echo()\n label = \"Synchronising %(package)s:\" % {\"package\": click.style(slug, fg=\"green\")}\n\n status_str = \"Waiting\"\n stage_str = None\n\n def display_status(current):\n \"\"\"Display current sync status.\"\"\"\n # pylint: disable=unused-argument\n if not stage_str:\n return status_str\n return click.style(\n \"%(status)s / %(stage)s\" % {\"status\": status_str, \"stage\": stage_str},\n fg=\"cyan\",\n )\n\n context_msg = \"Failed to synchronise file!\"\n with handle_api_exceptions(\n ctx, opts=opts, context_msg=context_msg, reraise_on_error=skip_errors\n ):\n last_progress = 0\n with click.progressbar(\n length=100,\n label=label,\n fill_char=click.style(\"#\", fg=\"green\"),\n empty_char=click.style(\"-\", fg=\"red\"),\n item_show_func=display_status,\n ) as pb:\n while True:\n res = get_package_status(owner, repo, slug)\n ok, failed, progress, status_str, stage_str, reason = res\n delta = progress - last_progress\n if delta > 0:\n last_progress = progress\n pb.update(delta)\n if ok or failed:\n break\n time.sleep(wait_interval)\n\n if ok:\n click.secho(\"Package synchronised successfully!\", fg=\"green\")\n return\n\n click.secho(\n \"Package failed to synchronise during stage: %(stage)s\"\n % {\"stage\": click.style(stage_str or \"Unknown\", fg=\"yellow\")},\n fg=\"red\",\n )\n\n if reason:\n click.secho(\n \"Reason given: %(reason)s\" % {\"reason\": click.style(reason, fg=\"yellow\")},\n fg=\"red\",\n )\n\n # pylint: disable=fixme\n # FIXME: The API should communicate \"no retry\" fails\n if \"package should be deleted\" in reason and attempts > 1:\n click.secho(\n \"This is not recoverable, so stopping further attempts!\", fg=\"red\"\n )\n click.echo()\n attempts = 0\n\n if attempts + 1 > 0:\n # Show attempts upto and including zero attempts left\n click.secho(\n \"Attempts left: %(left)s (%(action)s)\"\n % {\n \"left\": click.style(six.text_type(attempts), bold=True),\n \"action\": \"trying again\" if attempts > 0 else \"giving up\",\n }\n )\n click.echo()\n\n if attempts > 0:\n from .resync import resync_package\n\n resync_package(\n ctx=ctx,\n opts=opts,\n owner=owner,\n repo=repo,\n slug=slug,\n skip_errors=skip_errors,\n )\n\n wait_for_package_sync(\n ctx=ctx,\n opts=opts,\n owner=owner,\n repo=repo,\n slug=slug,\n wait_interval=wait_interval,\n skip_errors=skip_errors,\n attempts=attempts,\n )\n else:\n ctx.exit(1)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef upload_files_and_create_package(\n ctx,\n opts,\n package_type,\n owner_repo,\n dry_run,\n no_wait_for_sync,\n wait_interval,\n skip_errors,\n sync_attempts,\n **kwargs\n):\n \"\"\"Upload package files and create a new package.\"\"\"\n # pylint: disable=unused-argument\n owner, repo = owner_repo\n\n # 1. Validate package create parameters\n validate_create_package(\n ctx=ctx,\n opts=opts,\n owner=owner,\n repo=repo,\n package_type=package_type,\n skip_errors=skip_errors,\n **kwargs\n )\n\n # 2. Validate file upload parameters\n md5_checksums = {}\n for k, v in kwargs.items():\n if not v or not k.endswith(\"_file\"):\n continue\n\n md5_checksums[k] = validate_upload_file(\n ctx=ctx,\n opts=opts,\n owner=owner,\n repo=repo,\n filepath=v,\n skip_errors=skip_errors,\n )\n\n if dry_run:\n click.echo()\n click.secho(\"You requested a dry run so skipping upload.\", fg=\"yellow\")\n return\n\n # 3. Upload any arguments that look like files\n for k, v in kwargs.items():\n if not v or not k.endswith(\"_file\"):\n continue\n\n kwargs[k] = upload_file(\n ctx=ctx,\n opts=opts,\n owner=owner,\n repo=repo,\n filepath=v,\n skip_errors=skip_errors,\n md5_checksum=md5_checksums[k],\n )\n\n # 4. Create the package with package files and additional arguments\n _, slug = create_package(\n ctx=ctx,\n opts=opts,\n owner=owner,\n repo=repo,\n package_type=package_type,\n skip_errors=skip_errors,\n **kwargs\n )\n\n if no_wait_for_sync:\n return\n\n # 5. (optionally) Wait for the package to synchronise\n wait_for_package_sync(\n ctx=ctx,\n opts=opts,\n owner=owner,\n repo=repo,\n slug=slug,\n wait_interval=wait_interval,\n skip_errors=skip_errors,\n attempts=sync_attempts,\n )", "response": "Upload files and create a new package."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_push_handlers():\n # pylint: disable=fixme\n # HACK: hacky territory - Dynamically generate a handler for each of the\n # package formats, until we have slightly more clever 'guess type'\n # handling. :-)\n handlers = create_push_handlers.handlers = {}\n context = create_push_handlers.context = get_package_formats()\n\n for key, parameters in six.iteritems(context):\n kwargs = parameters.copy()\n\n # Remove standard arguments\n kwargs.pop(\"package_file\")\n if \"distribution\" in parameters:\n has_distribution_param = True\n kwargs.pop(\"distribution\")\n else:\n has_distribution_param = False\n\n has_additional_params = len(kwargs) > 0\n\n help_text = \"\"\"\n Push/upload a new %(type)s package upstream.\n \"\"\" % {\n \"type\": key.capitalize()\n }\n\n if has_additional_params:\n help_text += \"\"\"\n\n PACKAGE_FILE: The main file to create the package from.\n \"\"\"\n else:\n help_text += \"\"\"\n\n PACKAGE_FILE: Any number of files to create packages from. Each\n file will result in a separate package.\n \"\"\"\n\n if has_distribution_param:\n target_metavar = \"OWNER/REPO/DISTRO/RELEASE\"\n target_callback = validators.validate_owner_repo_distro\n help_text += \"\"\"\n\n OWNER/REPO/DISTRO/RELEASE: Specify the OWNER namespace (i.e.\n user or org), the REPO name where the package file will be uploaded\n to, and the DISTRO and RELEASE the package is for. All separated by\n a slash.\n\n Example: 'your-org/awesome-repo/ubuntu/xenial'.\n \"\"\"\n else:\n target_metavar = \"OWNER/REPO\"\n target_callback = validators.validate_owner_repo\n help_text += \"\"\"\n\n OWNER/REPO: Specify the OWNER namespace (i.e. user or org), and the\n REPO name where the package file will be uploaded to. All separated\n by a slash.\n\n Example: 'your-org/awesome-repo'.\n \"\"\"\n\n @push.command(name=key, help=help_text)\n @decorators.common_api_auth_options\n @decorators.common_cli_config_options\n @decorators.common_cli_output_options\n @decorators.common_package_action_options\n @decorators.initialise_api\n @click.argument(\"owner_repo\", metavar=target_metavar, callback=target_callback)\n @click.argument(\n \"package_file\",\n nargs=1 if has_additional_params else -1,\n type=ExpandPath(\n dir_okay=False, exists=True, writable=False, resolve_path=True\n ),\n )\n @click.option(\n \"-n\",\n \"--dry-run\",\n default=False,\n is_flag=True,\n help=\"Execute in dry run mode (don't upload anything.)\",\n )\n @click.option(\n \"-n\",\n \"--dry-run\",\n default=False,\n is_flag=True,\n help=\"Execute in dry run mode (don't upload anything.)\",\n )\n @click.pass_context\n def push_handler(ctx, *args, **kwargs):\n \"\"\"Handle upload for a specific package format.\"\"\"\n parameters = context.get(ctx.info_name)\n kwargs[\"package_type\"] = ctx.info_name\n\n owner_repo = kwargs.pop(\"owner_repo\")\n if \"distribution\" in parameters:\n kwargs[\"distribution\"] = \"/\".join(owner_repo[2:])\n owner_repo = owner_repo[0:2]\n kwargs[\"owner_repo\"] = owner_repo\n\n package_files = kwargs.pop(\"package_file\")\n if not isinstance(package_files, tuple):\n package_files = (package_files,)\n\n for package_file in package_files:\n kwargs[\"package_file\"] = package_file\n\n try:\n click.echo()\n upload_files_and_create_package(ctx, *args, **kwargs)\n except ApiException:\n click.secho(\"Skipping error and moving on.\", fg=\"yellow\")\n\n click.echo()\n\n # Add any additional arguments\n for k, info in six.iteritems(kwargs):\n option_kwargs = {}\n option_name_fmt = \"--%(key)s\"\n\n if k.endswith(\"_file\"):\n # Treat parameters that end with _file as uploadable filepaths.\n option_kwargs[\"type\"] = ExpandPath(\n dir_okay=False, exists=True, writable=False, resolve_path=True\n )\n elif info[\"type\"] == \"bool\":\n option_name_fmt = \"--%(key)s/--no-%(key)s\"\n option_kwargs[\"is_flag\"] = True\n else:\n option_kwargs[\"type\"] = str\n\n option_name = option_name_fmt % {\"key\": k.replace(\"_\", \"-\")}\n decorator = click.option(\n option_name,\n required=info[\"required\"],\n help=info[\"help\"],\n **option_kwargs\n )\n push_handler = decorator(push_handler)\n\n handlers[key] = push_handler", "response": "Create a handler for uploading a new package from each of the package formats."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef memoize(func):\n func._result_cache = {} # pylint: disable-msg=W0212\n\n @wraps(func)\n def _memoized_func(*args, **kwargs):\n key = (args, tuple(sorted(kwargs.items())))\n if key in func._result_cache: # pylint: disable-msg=W0212\n return func._result_cache[key] # pylint: disable-msg=W0212\n else:\n result = func(*args, **kwargs)\n if isinstance(result, GeneratorType) or not isinstance(result, Hashable):\n raise TypeError(\"Can't memoize a generator or non-hashable object!\")\n func._result_cache[key] = result # pylint: disable-msg=W0212\n return result\n\n return _memoized_func", "response": "Memoize a function to return the cached result on subsequent calls."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nclear the memoized method or memoized property results for the given method names from the given object.", "response": "def clear_memoized_methods(obj, *method_names):\n \"\"\"\n Clear the memoized method or @memoizedproperty results for the given\n method names from the given object.\n\n >>> v = [0]\n >>> def inc():\n ... v[0] += 1\n ... return v[0]\n ...\n >>> class Foo(object):\n ... @memoizemethod\n ... def foo(self):\n ... return inc()\n ... @memoizedproperty\n ... def g(self):\n ... return inc()\n ...\n >>> f = Foo()\n >>> f.foo(), f.foo()\n (1, 1)\n >>> clear_memoized_methods(f, 'foo')\n >>> (f.foo(), f.foo(), f.g, f.g)\n (2, 2, 3, 3)\n >>> (f.foo(), f.foo(), f.g, f.g)\n (2, 2, 3, 3)\n >>> clear_memoized_methods(f, 'g', 'no_problem_if_undefined')\n >>> f.g, f.foo(), f.g\n (4, 2, 4)\n >>> f.foo()\n 2\n \"\"\"\n for key in list(getattr(obj, '_memoized_results', {}).keys()):\n # key[0] is the method name\n if key[0] in method_names:\n del obj._memoized_results[key]\n\n try:\n property_dict = obj._cache_\n except AttributeError:\n property_dict = obj._cache_ = {}\n\n for prop in method_names:\n inner_attname = '__%s' % prop\n if inner_attname in property_dict:\n del property_dict[inner_attname]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef aes_encrypt(base64_encryption_key, data):\n if isinstance(data, text_type):\n data = data.encode(\"UTF-8\")\n aes_key_bytes, hmac_key_bytes = _extract_keys(base64_encryption_key)\n data = _pad(data)\n iv_bytes = os.urandom(AES_BLOCK_SIZE)\n cipher = AES.new(aes_key_bytes, mode=AES.MODE_CBC, IV=iv_bytes)\n data = iv_bytes + cipher.encrypt(data) # prepend init vector\n hmac_signature = hmac.new(hmac_key_bytes, data, hashlib.sha256).digest()\n return as_base64(data + hmac_signature)", "response": "Encrypt data with AES - CBC and sign it with HMAC - SHA256"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef aes_decrypt(base64_encryption_key, base64_data):\n data = from_base64(base64_data)\n aes_key_bytes, hmac_key_bytes = _extract_keys(base64_encryption_key)\n data, hmac_signature = data[:-HMAC_SIG_SIZE], data[-HMAC_SIG_SIZE:]\n if hmac.new(hmac_key_bytes, data, hashlib.sha256).digest() != hmac_signature:\n raise AuthenticationError(\"HMAC authentication failed\")\n iv_bytes, data = data[:AES_BLOCK_SIZE], data[AES_BLOCK_SIZE:]\n cipher = AES.new(aes_key_bytes, AES.MODE_CBC, iv_bytes)\n data = cipher.decrypt(data)\n return _unpad(data)", "response": "Verify HMAC - SHA256 signature and decrypt data with AES - CBC"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef whoami(ctx, opts):\n click.echo(\"Retrieving your authentication status from the API ... \", nl=False)\n\n context_msg = \"Failed to retrieve your authentication status!\"\n with handle_api_exceptions(ctx, opts=opts, context_msg=context_msg):\n with maybe_spinner(opts):\n is_auth, username, email, name = get_user_brief()\n\n click.secho(\"OK\", fg=\"green\")\n click.echo(\"You are authenticated as:\")\n if not is_auth:\n click.secho(\"Nobody (i.e. anonymous user)\", fg=\"yellow\")\n else:\n click.secho(\n \"%(name)s (slug: %(username)s, email: %(email)s)\"\n % {\n \"name\": click.style(name, fg=\"cyan\"),\n \"username\": click.style(username, fg=\"magenta\"),\n \"email\": click.style(email, fg=\"green\"),\n }\n )", "response": "Retrieve your current authentication status."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nperforms a standardized pipline of operations across an iterable.", "response": "def _clean(self, value):\n\t\t\"\"\"Perform a standardized pipline of operations across an iterable.\"\"\"\n\t\t\n\t\tvalue = (str(v) for v in value)\n\t\t\n\t\tif self.strip:\n\t\t\tvalue = (v.strip() for v in value)\n\t\t\n\t\tif not self.empty:\n\t\t\tvalue = (v for v in value if v)\n\t\t\n\t\treturn value"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts the given string into a list of substrings.", "response": "def native(self, value, context=None):\n\t\t\"\"\"Convert the given string into a list of substrings.\"\"\"\n\t\t\n\t\tseparator = self.separator.strip() if self.strip and hasattr(self.separator, 'strip') else self.separator\n\t\tvalue = super().native(value, context)\n\t\t\n\t\tif value is None:\n\t\t\treturn self.cast()\n\t\t\n\t\tif hasattr(value, 'split'):\n\t\t\tvalue = value.split(separator)\n\t\t\n\t\tvalue = self._clean(value)\n\t\t\n\t\ttry:\n\t\t\treturn self.cast(value) if self.cast else value\n\t\texcept Exception as e:\n\t\t\traise Concern(\"{0} caught, failed to perform array transform: {1}\", e.__class__.__name__, str(e))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconstructs a string - like representation for an iterable of string - like objects.", "response": "def foreign(self, value, context=None):\n\t\t\"\"\"Construct a string-like representation for an iterable of string-like objects.\"\"\"\n\t\t\n\t\tif self.separator is None:\n\t\t\tseparator = ' '\n\t\telse:\n\t\t\tseparator = self.separator.strip() if self.strip and hasattr(self.separator, 'strip') else self.separator\n\t\t\n\t\tvalue = self._clean(value)\n\t\t\n\t\ttry:\n\t\t\tvalue = separator.join(value)\n\t\texcept Exception as e:\n\t\t\traise Concern(\"{0} caught, failed to convert to string: {1}\", e.__class__.__name__, str(e))\n\t\t\n\t\treturn super().foreign(value)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef tob32(val):\n ret = bytearray(4)\n ret[0] = (val>>24)&M8\n ret[1] = (val>>16)&M8\n ret[2] = (val>>8)&M8\n ret[3] = val&M8\n return ret", "response": "Return provided 32 bit value as a string of four bytes."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef bintoihex(buf, spos=0x0000):\n c = 0\n olen = len(buf)\n ret = \"\"\n # 16 byte lines\n while (c+0x10) <= olen:\n adr = c + spos\n l = ':10{0:04X}00'.format(adr)\n sum = 0x10+((adr>>8)&M8)+(adr&M8)\n for j in range(0,0x10):\n nb = buf[c+j]\n l += '{0:02X}'.format(nb)\n sum = (sum + nb)&M8\n l += '{0:02X}'.format((~sum+1)&M8)\n ret += l + '\\n'\n c += 0x10\n # remainder\n if c < olen:\n rem = olen-c\n sum = rem\n adr = c + spos\n l = ':{0:02X}{1:04X}00'.format(rem,adr) # rem < 0x10\n sum += ((adr>>8)&M8)+(adr&M8)\n for j in range(0,rem):\n nb = buf[c+j]\n l += '{0:02X}'.format(nb)\n sum = (sum + nb)&M8\n l += '{0:02X}'.format((~sum+1)&M8)\n ret += l + '\\n'\n ret += ':00000001FF\\n' # EOF\n return ret", "response": "Convert binary buffer to ihex and return as string."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef op_gen(mcode):\n gen = op_tbl[mcode[0]]\n ret = gen[0]\t# opcode\n nargs = len(gen)\n i = 1\n while i < nargs:\n if i < len(mcode):\t# or assume they are same len\n ret |= (mcode[i]&gen[i][0]) << gen[i][1]\n i += 1\n return ret", "response": "Generate a machine instruction using the op gen table."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nemits scan error and abort assembly.", "response": "def scanerror(self, msg):\n \"\"\"Emit scan error and abort assembly.\"\"\"\n error('scan error: ' + msg + ' on line {}'.format(self.sline))\n sys.exit(-1)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nemitting parse error and abort assembly.", "response": "def parseerror(self, msg, line=None):\n \"\"\"Emit parse error and abort assembly.\"\"\"\n if line is None:\n line = self.sline\n error('parse error: ' + msg + ' on line {}'.format(line))\n sys.exit(-2)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates payload for upload or check - upload operations.", "response": "def make_create_payload(**kwargs):\n \"\"\"Create payload for upload/check-upload operations.\"\"\"\n payload = {}\n # Add non-empty arguments\n for k, v in six.iteritems(kwargs):\n if v is not None:\n payload[k] = v\n\n return payload"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a new package in a repository.", "response": "def create_package(package_format, owner, repo, **kwargs):\n \"\"\"Create a new package in a repository.\"\"\"\n client = get_packages_api()\n\n with catch_raise_api_exception():\n upload = getattr(client, \"packages_upload_%s_with_http_info\" % package_format)\n\n data, _, headers = upload(\n owner=owner, repo=repo, data=make_create_payload(**kwargs)\n )\n\n ratelimits.maybe_rate_limit(client, headers)\n return data.slug_perm, data.slug"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef validate_create_package(package_format, owner, repo, **kwargs):\n client = get_packages_api()\n\n with catch_raise_api_exception():\n check = getattr(\n client, \"packages_validate_upload_%s_with_http_info\" % package_format\n )\n\n _, _, headers = check(\n owner=owner, repo=repo, data=make_create_payload(**kwargs)\n )\n\n ratelimits.maybe_rate_limit(client, headers)\n return True", "response": "Validate parameters for creating a package."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef copy_package(owner, repo, identifier, destination):\n client = get_packages_api()\n\n with catch_raise_api_exception():\n data, _, headers = client.packages_copy_with_http_info(\n owner=owner,\n repo=repo,\n identifier=identifier,\n data={\"destination\": destination},\n )\n\n ratelimits.maybe_rate_limit(client, headers)\n return data.slug_perm, data.slug", "response": "Copy a package to another repository."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmoves a package to another repository.", "response": "def move_package(owner, repo, identifier, destination):\n \"\"\"Move a package to another repository.\"\"\"\n client = get_packages_api()\n\n with catch_raise_api_exception():\n data, _, headers = client.packages_move_with_http_info(\n owner=owner,\n repo=repo,\n identifier=identifier,\n data={\"destination\": destination},\n )\n\n ratelimits.maybe_rate_limit(client, headers)\n return data.slug_perm, data.slug"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndelete a package in a repository.", "response": "def delete_package(owner, repo, identifier):\n \"\"\"Delete a package in a repository.\"\"\"\n client = get_packages_api()\n\n with catch_raise_api_exception():\n _, _, headers = client.packages_delete_with_http_info(\n owner=owner, repo=repo, identifier=identifier\n )\n\n ratelimits.maybe_rate_limit(client, headers)\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_package_status(owner, repo, identifier):\n client = get_packages_api()\n\n with catch_raise_api_exception():\n data, _, headers = client.packages_status_with_http_info(\n owner=owner, repo=repo, identifier=identifier\n )\n\n ratelimits.maybe_rate_limit(client, headers)\n\n # pylint: disable=no-member\n # Pylint detects the returned value as a tuple\n return (\n data.is_sync_completed,\n data.is_sync_failed,\n data.sync_progress,\n data.status_str,\n data.stage_str,\n data.status_reason,\n )", "response": "Get the status of a package in a repository."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nlist packages for a repository.", "response": "def list_packages(owner, repo, **kwargs):\n \"\"\"List packages for a repository.\"\"\"\n client = get_packages_api()\n\n api_kwargs = {}\n api_kwargs.update(utils.get_page_kwargs(**kwargs))\n api_kwargs.update(utils.get_query_kwargs(**kwargs))\n\n with catch_raise_api_exception():\n data, _, headers = client.packages_list_with_http_info(\n owner=owner, repo=repo, **api_kwargs\n )\n\n ratelimits.maybe_rate_limit(client, headers)\n page_info = PageInfo.from_headers(headers)\n return [x.to_dict() for x in data], page_info"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_package_formats():\n # pylint: disable=fixme\n # HACK: This obviously isn't great, and it is subject to change as\n # the API changes, but it'll do for now as a interim method of\n # introspection to get the parameters we need.\n def get_parameters(cls):\n \"\"\"Build parameters for a package format.\"\"\"\n params = {}\n\n # Create a dummy instance so we can check if a parameter is required.\n # As with the rest of this function, this is obviously hacky. We'll\n # figure out a way to pull this information in from the API later.\n dummy_kwargs = {k: \"dummy\" for k in cls.swagger_types}\n instance = cls(**dummy_kwargs)\n\n for k, v in six.iteritems(cls.swagger_types):\n attr = getattr(cls, k)\n docs = attr.__doc__.strip().split(\"\\n\")\n doc = (docs[1] if docs[1] else docs[0]).strip()\n\n try:\n setattr(instance, k, None)\n required = False\n except ValueError:\n required = True\n\n params[cls.attribute_map.get(k)] = {\n \"type\": v,\n \"help\": doc,\n \"required\": required,\n }\n\n return params\n\n return {\n key.replace(\"PackagesUpload\", \"\").lower(): get_parameters(cls)\n for key, cls in inspect.getmembers(cloudsmith_api.models)\n if key.startswith(\"PackagesUpload\")\n }", "response": "Get the list of available package formats and parameters."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting names for available package formats.", "response": "def get_package_format_names(predicate=None):\n \"\"\"Get names for available package formats.\"\"\"\n return [\n k\n for k, v in six.iteritems(get_package_formats())\n if not predicate or predicate(k, v)\n ]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef start(opts, bot, event):\n\n name = opts['--name']\n\n now = datetime.datetime.now()\n bot.timers[name] = now\n\n return bot.start_fmt.format(now)", "response": "Start a new timer."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_user_token(login, password):\n client = get_user_api()\n\n # Never use API key for the token endpoint\n config = cloudsmith_api.Configuration()\n set_api_key(config, None)\n\n with catch_raise_api_exception():\n data, _, headers = client.user_token_create_with_http_info(\n data={\"email\": login, \"password\": password}\n )\n\n ratelimits.maybe_rate_limit(client, headers)\n return data.token", "response": "Retrieve user token from the API via authentication."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_user_brief():\n client = get_user_api()\n\n with catch_raise_api_exception():\n data, _, headers = client.user_self_with_http_info()\n\n ratelimits.maybe_rate_limit(client, headers)\n return data.authenticated, data.slug, data.email, data.name", "response": "Retrieve brief for current user."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting the default filepath for the configuratin file.", "response": "def get_default_filepath(cls):\n \"\"\"Get the default filepath for the configuratin file.\"\"\"\n if not cls.config_files:\n return None\n if not cls.config_searchpath:\n return None\n filename = cls.config_files[0]\n filepath = cls.config_searchpath[0]\n return os.path.join(filepath, filename)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_default_file(cls, data=None, mode=None):\n filepath = cls.get_default_filepath()\n if not filepath:\n return False\n\n filename = os.path.basename(filepath)\n config = read_file(get_data_path(), filename)\n\n # Find and replace data in default config\n data = data or {}\n for k, v in six.iteritems(data):\n v = v or \"\"\n config = re.sub(\n r\"^(%(key)s) =[ ]*$\" % {\"key\": k},\n \"%(key)s = %(value)s\" % {\"key\": k, \"value\": v},\n config,\n flags=re.MULTILINE,\n )\n\n dirpath = os.path.dirname(filepath)\n if not os.path.exists(dirpath):\n os.makedirs(dirpath)\n with click.open_file(filepath, \"w+\") as f:\n f.write(config)\n\n if mode is not None:\n os.chmod(filepath, mode)\n\n return True", "response": "Create a config file and override data if specified."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef has_default_file(cls):\n for filename in cls.config_files:\n for searchpath in cls.config_searchpath:\n path = os.path.join(searchpath, filename)\n if os.path.exists(path):\n return True\n\n return False", "response": "Check if a configuration file exists."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load_config(cls, opts, path=None, profile=None):\n if path and os.path.exists(path):\n if os.path.isdir(path):\n cls.config_searchpath.insert(0, path)\n else:\n cls.config_files.insert(0, path)\n\n config = cls.read_config()\n values = config.get(\"default\", {})\n cls._load_values_into_opts(opts, values)\n\n if profile and profile != \"default\":\n values = config.get(\"profile:%s\" % profile, {})\n cls._load_values_into_opts(opts, values)\n\n return values", "response": "Load a configuration file into an options object."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef load_config_file(self, path, profile=None):\n config_cls = self.get_config_reader()\n return config_cls.load_config(self, path, profile=profile)", "response": "Load the standard config file."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nloads the credentials config file.", "response": "def load_creds_file(self, path, profile=None):\n \"\"\"Load the credentials config file.\"\"\"\n config_cls = self.get_creds_reader()\n return config_cls.load_config(self, path, profile=profile)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef api_headers(self, value):\n value = validators.validate_api_headers(\"api_headers\", value)\n self._set_option(\"api_headers\", value)", "response": "Set value for API headers."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef error_retry_codes(self, value):\n if isinstance(value, six.string_types):\n value = [int(x) for x in value.split(\",\")]\n self._set_option(\"error_retry_codes\", value)", "response": "Set value for error_retry_codes."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting value for an option.", "response": "def _set_option(self, name, value, allow_clear=False):\n \"\"\"Set value for an option.\"\"\"\n if not allow_clear:\n # Prevent clears if value was set\n try:\n current_value = self._get_option(name)\n if value is None and current_value is not None:\n return\n except AttributeError:\n pass\n\n self.opts[name] = value"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_rate_limits():\n client = get_rates_api()\n\n with catch_raise_api_exception():\n data, _, headers = client.rates_limits_list_with_http_info()\n\n ratelimits.maybe_rate_limit(client, headers)\n\n return {\n k: RateLimitsInfo.from_dict(v)\n for k, v in six.iteritems(data.to_dict().get(\"resources\", {}))\n }", "response": "Retrieve status and optionally rate limit information from the API."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef loads(self, value, context=None):\n\t\t\n\t\tif value == '' or (hasattr(value, 'strip') and value.strip() == ''):\n\t\t\treturn None\n\t\t\n\t\treturn self.native(value)", "response": "Attempt to load a string - based value into the native representation."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef dump(self, fh, value, context=None):\n\t\t\n\t\tvalue = self.dumps(value)\n\t\tfh.write(value)\n\t\treturn len(value)", "response": "Transform and write a string - based foreign value to the given file - like object."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef native(self, value, context=None):\n\t\t\n\t\tif self.strip and hasattr(value, 'strip'):\n\t\t\tvalue = value.strip()\n\t\t\n\t\tif self.none and value == '':\n\t\t\treturn None\n\t\t\n\t\tif self.encoding and isinstance(value, bytes):\n\t\t\treturn value.decode(self.encoding)\n\t\t\n\t\treturn value", "response": "Convert a value from a foriegn type ( i. e. web - safe to Python - native."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert a value from a foriegn type ( i. e. web - safe to Python - native.", "response": "def native(self, value, context=None):\n\t\t\"\"\"Convert a value from a foriegn type (i.e. web-safe) to Python-native.\"\"\"\n\t\t\n\t\tvalue = super().native(value, context)\n\t\t\n\t\tif value is None: return\n\t\t\n\t\ttry:\n\t\t\treturn self.ingress(value)\n\t\texcept Exception as e:\n\t\t\traise Concern(\"Unable to transform incoming value: {0}\", str(e))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef rates(ctx, opts):\n click.echo(\"Retrieving rate limits ... \", nl=False)\n\n context_msg = \"Failed to retrieve status!\"\n with handle_api_exceptions(ctx, opts=opts, context_msg=context_msg):\n with maybe_spinner(opts):\n resources_limits = get_rate_limits()\n\n click.secho(\"OK\", fg=\"green\")\n\n headers = [\"Resource\", \"Throttled\", \"Remaining\", \"Interval (Seconds)\", \"Reset\"]\n\n rows = []\n for resource, limits in six.iteritems(resources_limits):\n rows.append(\n [\n click.style(resource, fg=\"cyan\"),\n click.style(\n \"Yes\" if limits.throttled else \"No\",\n fg=\"red\" if limits.throttled else \"green\",\n ),\n \"%(remaining)s/%(limit)s\"\n % {\n \"remaining\": click.style(\n six.text_type(limits.remaining), fg=\"yellow\"\n ),\n \"limit\": click.style(six.text_type(limits.limit), fg=\"yellow\"),\n },\n click.style(six.text_type(limits.interval), fg=\"blue\"),\n click.style(six.text_type(limits.reset), fg=\"magenta\"),\n ]\n )\n\n if resources_limits:\n click.echo()\n utils.pretty_print_table(headers, rows)\n\n click.echo()\n\n num_results = len(resources_limits)\n list_suffix = \"resource%s\" % (\"s\" if num_results != 1 else \"\")\n utils.pretty_print_list_info(num_results=num_results, suffix=list_suffix)", "response": "Check current API rate limits."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchecking the status of the Cloudsmith service.", "response": "def service(ctx, opts):\n \"\"\"Check the status of the Cloudsmith service.\"\"\"\n click.echo(\"Retrieving service status ... \", nl=False)\n\n context_msg = \"Failed to retrieve status!\"\n with handle_api_exceptions(ctx, opts=opts, context_msg=context_msg):\n with maybe_spinner(opts):\n status, version = get_status(with_version=True)\n\n click.secho(\"OK\", fg=\"green\")\n\n config = cloudsmith_api.Configuration()\n\n click.echo()\n click.echo(\n \"The service endpoint is: %(endpoint)s\"\n % {\"endpoint\": click.style(config.host, bold=True)}\n )\n click.echo(\n \"The service status is: %(status)s\"\n % {\"status\": click.style(status, bold=True)}\n )\n click.echo(\n \"The service version is: %(version)s \"\n % {\"version\": click.style(version, bold=True)},\n nl=False,\n )\n\n api_version = get_api_version_info()\n\n if semver.compare(version, api_version) > 0:\n click.secho(\"(maybe out-of-date)\", fg=\"yellow\")\n\n click.echo()\n click.secho(\n \"The API library used by this CLI tool is built against \"\n \"service version: %(version)s\"\n % {\"version\": click.style(api_version, bold=True)},\n fg=\"yellow\",\n )\n else:\n click.secho(\"(up-to-date)\", fg=\"green\")\n\n click.echo()\n click.secho(\n \"The API library used by this CLI tool seems to be up-to-date.\", fg=\"green\"\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the total number of seconds in the current node.", "response": "def total_seconds(td):\n \"\"\"For those with older versions of Python, a pure-Python\n implementation of Python 2.7's :meth:`~datetime.timedelta.total_seconds`.\n\n Args:\n td (datetime.timedelta): The timedelta to convert to seconds.\n Returns:\n float: total number of seconds\n\n >>> td = timedelta(days=4, seconds=33)\n >>> total_seconds(td)\n 345633.0\n \"\"\"\n a_milli = 1000000.0\n td_ds = td.seconds + (td.days * 86400) # 24 * 60 * 60\n td_micro = td.microseconds + (td_ds * a_milli)\n return td_micro / a_milli"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts a datetime object to an integer timestamp suitable interoperation with time. time and na\u00efve other epoch - based timestamps.", "response": "def dt_to_timestamp(dt):\n \"\"\"Converts from a :class:`~datetime.datetime` object to an integer\n timestamp, suitable interoperation with :func:`time.time` and\n other `Epoch-based timestamps`.\n\n .. _Epoch-based timestamps: https://en.wikipedia.org/wiki/Unix_time\n\n >>> abs(round(time.time() - dt_to_timestamp(datetime.utcnow()), 2))\n 0.0\n\n ``dt_to_timestamp`` supports both timezone-aware and na\u00efve\n :class:`~datetime.datetime` objects. Note that it assumes na\u00efve\n datetime objects are implied UTC, such as those generated with\n :meth:`datetime.datetime.utcnow`. If your datetime objects are\n local time, such as those generated with\n :meth:`datetime.datetime.now`, first convert it using the\n :meth:`datetime.datetime.replace` method with ``tzinfo=``\n :class:`LocalTZ` object in this module, then pass the result of\n that to ``dt_to_timestamp``.\n \"\"\"\n if dt.tzinfo:\n td = dt - EPOCH_AWARE\n else:\n td = dt - EPOCH_NAIVE\n return total_seconds(td)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nparse the limited subset of ISO8601 - formatted time strings as .", "response": "def isoparse(iso_str):\n \"\"\"Parses the limited subset of `ISO8601-formatted time`_ strings as\n returned by :meth:`datetime.datetime.isoformat`.\n\n >>> epoch_dt = datetime.utcfromtimestamp(0)\n >>> iso_str = epoch_dt.isoformat()\n >>> print(iso_str)\n 1970-01-01T00:00:00\n >>> isoparse(iso_str)\n datetime.datetime(1970, 1, 1, 0, 0)\n\n >>> utcnow = datetime.utcnow()\n >>> utcnow == isoparse(utcnow.isoformat())\n True\n\n For further datetime parsing, see the `iso8601`_ package for strict\n ISO parsing and `dateutil`_ package for loose parsing and more.\n\n .. _ISO8601-formatted time: https://en.wikipedia.org/wiki/ISO_8601\n .. _iso8601: https://pypi.python.org/pypi/iso8601\n .. _dateutil: https://pypi.python.org/pypi/python-dateutil\n\n \"\"\"\n dt_args = [int(p) for p in _NONDIGIT_RE.split(iso_str)]\n return datetime(*dt_args)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef decimal_relative_time(d, other=None, ndigits=0, cardinalize=True):\n if other is None:\n other = datetime.utcnow()\n diff = other - d\n diff_seconds = total_seconds(diff)\n abs_diff = abs(diff)\n b_idx = bisect.bisect(_BOUND_DELTAS, abs_diff) - 1\n bbound, bunit, bname = _BOUNDS[b_idx]\n f_diff = diff_seconds / total_seconds(bunit)\n rounded_diff = round(f_diff, ndigits)\n if cardinalize:\n return rounded_diff, _cardinalize_time_unit(bname, abs(rounded_diff))\n return rounded_diff, bname", "response": "Returns a tuple representing the relative time difference between two log files or the current time."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a short string representation of the difference between two or one time objects or the current time.", "response": "def relative_time(d, other=None, ndigits=0):\n \"\"\"Get a string representation of the difference between two\n :class:`~datetime.datetime` objects or one\n :class:`~datetime.datetime` and the current time. Handles past and\n future times.\n\n Args:\n d (datetime): The first datetime object.\n other (datetime): An optional second datetime object. If\n unset, defaults to the current time as determined\n :meth:`datetime.utcnow`.\n ndigits (int): The number of decimal digits to round to,\n defaults to ``0``.\n Returns:\n A short English-language string.\n\n >>> now = datetime.utcnow()\n >>> relative_time(now, ndigits=1)\n '0 seconds ago'\n >>> relative_time(now - timedelta(days=1, seconds=36000), ndigits=1)\n '1.4 days ago'\n >>> relative_time(now + timedelta(days=7), now, ndigits=1)\n '1 week from now'\n\n \"\"\"\n drt, unit = decimal_relative_time(d, other, ndigits, cardinalize=True)\n phrase = 'ago'\n if drt < 0:\n phrase = 'from now'\n return '%g %s %s' % (abs(drt), unit, phrase)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef strpdate(string, format):\n whence = datetime.strptime(string, format)\n return whence.date()", "response": "Parse the date string according to the format in format."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nlisting packages in a repository.", "response": "def packages(ctx, opts, owner_repo, page, page_size, query):\n \"\"\"\n List packages for a repository.\n\n OWNER/REPO: Specify the OWNER namespace (i.e. user or org), and the\n REPO name to list packages for that namespace and repository. All separated\n by a slash.\n\n You can use the search query (-q|--query) to filter packages:\n\n - By name: 'my-package' (implicit) or 'name:my-package'\n\n - By filename: 'pkg.ext' (implicit) or 'filename:pkg.ext' (explicit)\n\n - By version: '1.0.0' (implicit) or 'version:1.0.0' (explicit)\n\n - By arch: 'x86_64' (implicit) or 'architecture:x86_64' (explicit)\n\n - By disto: 'el' (implicit) or 'distribution:el' (explicit)\n\n You can also modify the search terms:\n\n - '^foo' to anchor to start of term\n\n - 'foo$' to anchor to end of term\n\n - 'foo*bar' for fuzzy matching\n\n - '~foo' for negation of the term (explicit only, e.g. name:~foo)\n\n Multiple search terms are conjunctive (AND).\n\n Examples, to find packages named exactly foo, with a zip filename, that are\n NOT the x86 architecture, use something like this:\n\n --query 'name:^foo$ filename:.zip$ architecture:~x86'\n\n \"\"\"\n owner, repo = owner_repo\n\n # Use stderr for messages if the output is something else (e.g. # JSON)\n use_stderr = opts.output != \"pretty\"\n\n click.echo(\"Getting list of packages ... \", nl=False, err=use_stderr)\n\n context_msg = \"Failed to get list of packages!\"\n with handle_api_exceptions(ctx, opts=opts, context_msg=context_msg):\n with maybe_spinner(opts):\n packages_, page_info = list_packages(\n owner=owner, repo=repo, page=page, page_size=page_size, query=query\n )\n\n click.secho(\"OK\", fg=\"green\", err=use_stderr)\n\n if utils.maybe_print_as_json(opts, packages_, page_info):\n return\n\n headers = [\"Name\", \"Version\", \"Status\", \"Owner / Repository (Identifier)\"]\n rows = []\n for package in sorted(packages_, key=itemgetter(\"namespace\", \"slug\")):\n rows.append(\n [\n click.style(_get_package_name(package), fg=\"cyan\"),\n click.style(_get_package_version(package), fg=\"yellow\"),\n click.style(_get_package_status(package), fg=\"blue\"),\n \"%(owner_slug)s/%(repo_slug)s/%(slug)s\"\n % {\n \"owner_slug\": click.style(package[\"namespace\"], fg=\"magenta\"),\n \"repo_slug\": click.style(package[\"repository\"], fg=\"magenta\"),\n \"slug\": click.style(package[\"slug\"], fg=\"green\"),\n },\n ]\n )\n\n if packages_:\n click.echo()\n utils.pretty_print_table(headers, rows)\n\n click.echo()\n\n num_results = len(packages_)\n list_suffix = \"package%s visible\" % (\"s\" if num_results != 1 else \"\")\n utils.pretty_print_list_info(\n num_results=num_results, page_info=page_info, suffix=list_suffix\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nlists all repositories for a namespace.", "response": "def repos(ctx, opts, owner, page, page_size):\n \"\"\"\n List repositories for a namespace (owner).\n\n OWNER: Specify the OWNER namespace (i.e. user or org) to list the\n repositories for that namespace.\n\n If OWNER isn't specified it'll default to the currently authenticated user\n (if any). If you're unauthenticated, no results will be returned.\n \"\"\"\n # Use stderr for messages if the output is something else (e.g. # JSON)\n use_stderr = opts.output != \"pretty\"\n\n click.echo(\"Getting list of repositories ... \", nl=False, err=use_stderr)\n\n context_msg = \"Failed to get list of repositories!\"\n with handle_api_exceptions(ctx, opts=opts, context_msg=context_msg):\n with maybe_spinner(opts):\n repos_, page_info = list_repos(owner=owner, page=page, page_size=page_size)\n\n click.secho(\"OK\", fg=\"green\", err=use_stderr)\n\n if utils.maybe_print_as_json(opts, repos_, page_info):\n return\n\n headers = [\n \"Name\",\n \"Type\",\n \"Packages\",\n \"Groups\",\n \"Downloads\",\n \"Size\",\n \"Owner / Repository (Identifier)\",\n ]\n\n rows = []\n for repo in sorted(repos_, key=itemgetter(\"namespace\", \"slug\")):\n rows.append(\n [\n click.style(repo[\"name\"], fg=\"cyan\"),\n click.style(repo[\"repository_type_str\"], fg=\"yellow\"),\n click.style(six.text_type(repo[\"package_count\"]), fg=\"blue\"),\n click.style(six.text_type(repo[\"package_group_count\"]), fg=\"blue\"),\n click.style(six.text_type(repo[\"num_downloads\"]), fg=\"blue\"),\n click.style(six.text_type(repo[\"size_str\"]), fg=\"blue\"),\n \"%(owner_slug)s/%(slug)s\"\n % {\n \"owner_slug\": click.style(repo[\"namespace\"], fg=\"magenta\"),\n \"slug\": click.style(repo[\"slug\"], fg=\"green\"),\n },\n ]\n )\n\n if repos_:\n click.echo()\n utils.pretty_print_table(headers, rows)\n\n click.echo()\n\n num_results = len(repos_)\n list_suffix = \"repositor%s visible\" % (\"ies\" if num_results != 1 else \"y\")\n utils.pretty_print_list_info(\n num_results=num_results, page_info=page_info, suffix=list_suffix\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_package_status(package):\n status = package[\"status_str\"] or \"Unknown\"\n stage = package[\"stage_str\"] or \"Unknown\"\n if stage == \"Fully Synchronised\":\n return status\n return \"%(status)s / %(stage)s\" % {\"status\": status, \"stage\": stage}", "response": "Get the status for a package."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting a number string or sequence type into a pure boolean.", "response": "def boolify(value, nullable=False, return_string=False):\n \"\"\"Convert a number, string, or sequence type into a pure boolean.\n\n Args:\n value (number, string, sequence): pretty much anything\n\n Returns:\n bool: boolean representation of the given value\n\n Examples:\n >>> [boolify(x) for x in ('yes', 'no')]\n [True, False]\n >>> [boolify(x) for x in (0.1, 0+0j, True, '0', '0.0', '0.1', '2')]\n [True, False, True, False, False, True, True]\n >>> [boolify(x) for x in (\"true\", \"yes\", \"on\", \"y\")]\n [True, True, True, True]\n >>> [boolify(x) for x in (\"no\", \"non\", \"none\", \"off\", \"\")]\n [False, False, False, False, False]\n >>> [boolify(x) for x in ([], set(), dict(), tuple())]\n [False, False, False, False]\n >>> [boolify(x) for x in ([1], set([False]), dict({'a': 1}), tuple([2]))]\n [True, True, True, True]\n\n \"\"\"\n # cast number types naturally\n if isinstance(value, BOOL_COERCEABLE_TYPES):\n return bool(value)\n # try to coerce string into number\n val = text_type(value).strip().lower().replace('.', '', 1)\n if val.isnumeric():\n return bool(float(val))\n elif val in BOOLISH_TRUE:\n return True\n elif nullable and val in NULL_STRINGS:\n return None\n elif val in BOOLISH_FALSE:\n return False\n else: # must be False\n try:\n return bool(complex(val))\n except ValueError:\n if isinstance(value, string_types) and return_string:\n return value\n raise TypeCoercionError(value, \"The value %r cannot be boolified.\" % value)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef typify(value, type_hint=None):\n # value must be a string, or there at least needs to be a type hint\n if isinstance(value, string_types):\n value = value.strip()\n elif type_hint is None:\n # can't do anything because value isn't a string and there's no type hint\n return value\n\n # now we either have a stripped string, a type hint, or both\n # use the hint if it exists\n if isiterable(type_hint):\n if isinstance(type_hint, type) and issubclass(type_hint, Enum):\n try:\n return type_hint(value)\n except ValueError:\n return type_hint[value]\n type_hint = set(type_hint)\n if not (type_hint - NUMBER_TYPES_SET):\n return numberify(value)\n elif not (type_hint - STRING_TYPES_SET):\n return text_type(value)\n elif not (type_hint - {bool, NoneType}):\n return boolify(value, nullable=True)\n elif not (type_hint - (STRING_TYPES_SET | {bool})):\n return boolify(value, return_string=True)\n elif not (type_hint - (STRING_TYPES_SET | {NoneType})):\n value = text_type(value)\n return None if value.lower() == 'none' else value\n elif not (type_hint - {bool, int}):\n return typify_str_no_hint(text_type(value))\n else:\n raise NotImplementedError()\n elif type_hint is not None:\n # coerce using the type hint, or use boolify for bool\n try:\n return boolify(value) if type_hint == bool else type_hint(value)\n except ValueError as e:\n # ValueError: invalid literal for int() with base 10: 'nope'\n raise TypeCoercionError(value, text_type(e))\n else:\n # no type hint, but we know value is a string, so try to match with the regex patterns\n # if there's still no match, `typify_str_no_hint` will return `value`\n return typify_str_no_hint(value)", "response": "Takes a primitive value usually a string and attempts to make a more relevant type out of it."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nopens a disk - based carray or ctable object.", "response": "def open(rootdir, mode='a'):\n # ----------------------------------------------------------------------\n # https://github.com/Blosc/bcolz/blob/master/bcolz/toplevel.py#L104-L132\n # ----------------------------------------------------------------------\n \"\"\"\n open(rootdir, mode='a')\n\n Open a disk-based carray/ctable.\n This function could be used to open bcolz objects as bquery objects to\n perform queries on them.\n\n Parameters\n ----------\n rootdir : pathname (string)\n The directory hosting the carray/ctable object.\n mode : the open mode (string)\n Specifies the mode in which the object is opened. The supported\n values are:\n\n * 'r' for read-only\n * 'w' for emptying the previous underlying data\n * 'a' for allowing read/write on top of existing data\n\n Returns\n -------\n out : a carray/ctable object or IOError (if not objects are found)\n\n \"\"\"\n # First try with a carray\n rootsfile = os.path.join(rootdir, ROOTDIRS)\n if os.path.exists(rootsfile):\n return bquery.ctable(rootdir=rootdir, mode=mode)\n else:\n return bquery.carray(rootdir=rootdir, mode=mode)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndelete a package from a repository.", "response": "def delete(ctx, opts, owner_repo_package, yes):\n \"\"\"\n Delete a package from a repository.\n\n - OWNER/REPO/PACKAGE: Specify the OWNER namespace (i.e. user or org), the\n REPO name where the package is stored, and the PACKAGE name (slug) of the\n package itself. All separated by a slash.\n\n Example: 'your-org/awesome-repo/better-pkg'.\n \"\"\"\n owner, repo, slug = owner_repo_package\n\n delete_args = {\n \"owner\": click.style(owner, bold=True),\n \"repo\": click.style(repo, bold=True),\n \"package\": click.style(slug, bold=True),\n }\n\n prompt = \"delete the %(package)s from %(owner)s/%(repo)s\" % delete_args\n if not utils.confirm_operation(prompt, assume_yes=yes):\n return\n\n click.echo(\n \"Deleting %(package)s from %(owner)s/%(repo)s ... \" % delete_args, nl=False\n )\n\n context_msg = \"Failed to delete the package!\"\n with handle_api_exceptions(ctx, opts=opts, context_msg=context_msg):\n with maybe_spinner(opts):\n delete_package(owner=owner, repo=repo, identifier=slug)\n\n click.secho(\"OK\", fg=\"green\")"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nensures that owner and repo identifier is formatted correctly.", "response": "def validate_owner_repo_identifier(ctx, param, value):\n \"\"\"Ensure that owner/repo/identifier is formatted correctly.\"\"\"\n # pylint: disable=unused-argument\n form = \"OWNER/REPO/IDENTIFIER\"\n return validators.validate_slashes(param, value, minimum=3, maximum=3, form=form)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef common_entitlements_options(f):\n\n @click.option(\n \"--show-tokens\",\n default=False,\n is_flag=True,\n help=\"Show entitlement token string contents in output.\",\n )\n @click.pass_context\n @functools.wraps(f)\n def wrapper(ctx, *args, **kwargs):\n # pylint: disable=missing-docstring\n return ctx.invoke(f, *args, **kwargs)\n\n return wrapper", "response": "Add common options for entitlement commands."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef list_entitlements_options(f):\n\n @common_entitlements_options\n @decorators.common_cli_config_options\n @decorators.common_cli_list_options\n @decorators.common_cli_output_options\n @decorators.common_api_auth_options\n @decorators.initialise_api\n @click.argument(\n \"owner_repo\", metavar=\"OWNER/REPO\", callback=validators.validate_owner_repo\n )\n @click.pass_context\n @functools.wraps(f)\n def wrapper(ctx, *args, **kwargs):\n # pylint: disable=missing-docstring\n return ctx.invoke(f, *args, **kwargs)\n\n return wrapper", "response": "Options for list entitlements subcommand."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef list_entitlements(ctx, opts, owner_repo, page, page_size, show_tokens):\n owner, repo = owner_repo\n\n # Use stderr for messages if the output is something else (e.g. # JSON)\n use_stderr = opts.output != \"pretty\"\n\n click.echo(\n \"Getting list of entitlements for the %(repository)s \"\n \"repository ... \" % {\"repository\": click.style(repo, bold=True)},\n nl=False,\n err=use_stderr,\n )\n\n context_msg = \"Failed to get list of entitlements!\"\n with handle_api_exceptions(ctx, opts=opts, context_msg=context_msg):\n with maybe_spinner(opts):\n entitlements_, page_info = api.list_entitlements(\n owner=owner,\n repo=repo,\n page=page,\n page_size=page_size,\n show_tokens=show_tokens,\n )\n\n click.secho(\"OK\", fg=\"green\", err=use_stderr)\n\n print_entitlements(opts=opts, data=entitlements_, page_info=page_info)", "response": "List the entitlements for a repository."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef print_entitlements(opts, data, page_info=None, show_list_info=True):\n if utils.maybe_print_as_json(opts, data, page_info):\n return\n\n headers = [\"Name\", \"Token\", \"Created / Updated\", \"Identifier\"]\n\n rows = []\n for entitlement in sorted(data, key=itemgetter(\"name\")):\n rows.append(\n [\n click.style(\n \"%(name)s (%(type)s)\"\n % {\n \"name\": click.style(entitlement[\"name\"], fg=\"cyan\"),\n \"type\": \"user\" if entitlement[\"user\"] else \"token\",\n }\n ),\n click.style(entitlement[\"token\"], fg=\"yellow\"),\n click.style(entitlement[\"updated_at\"], fg=\"blue\"),\n click.style(entitlement[\"slug_perm\"], fg=\"green\"),\n ]\n )\n\n if data:\n click.echo()\n utils.pretty_print_table(headers, rows)\n\n if not show_list_info:\n return\n\n click.echo()\n\n num_results = len(data)\n list_suffix = \"entitlement%s\" % (\"s\" if num_results != 1 else \"\")\n utils.pretty_print_list_info(num_results=num_results, suffix=list_suffix)", "response": "Print entitlements as a table or output in another format."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a new entitlement in a repository.", "response": "def create(ctx, opts, owner_repo, show_tokens, name, token):\n \"\"\"\n Create a new entitlement in a repository.\n\n - OWNER/REPO: Specify the OWNER namespace (i.e. user or org), and the REPO\n name where you want to create an entitlement. All separated by a slash.\n\n Example: 'your-org/your-repo'\n\n Full CLI example:\n\n $ cloudsmith ents create your-org/your-repo --name 'Foobar'\n \"\"\"\n owner, repo = owner_repo\n\n # Use stderr for messages if the output is something else (e.g. # JSON)\n use_stderr = opts.output != \"pretty\"\n\n click.secho(\n \"Creating %(name)s entitlement for the %(repository)s \"\n \"repository ... \"\n % {\n \"name\": click.style(name, bold=True),\n \"repository\": click.style(repo, bold=True),\n },\n nl=False,\n err=use_stderr,\n )\n\n context_msg = \"Failed to create the entitlement!\"\n with handle_api_exceptions(ctx, opts=opts, context_msg=context_msg):\n with maybe_spinner(opts):\n entitlement = api.create_entitlement(\n owner=owner, repo=repo, name=name, token=token, show_tokens=show_tokens\n )\n\n click.secho(\"OK\", fg=\"green\", err=use_stderr)\n\n print_entitlements(opts=opts, data=[entitlement], show_list_info=False)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndeleting an entitlement from a repository.", "response": "def delete(ctx, opts, owner_repo_identifier, yes):\n \"\"\"\n Delete an entitlement from a repository.\n\n - OWNER/REPO/IDENTIFIER: Specify the OWNER namespace (i.e. user or org),\n and the REPO name that has an entitlement identified by IDENTIFIER. All\n separated by a slash.\n\n Example: 'your-org/your-repo/abcdef123456'\n\n Full CLI example:\n\n $ cloudsmith ents delete your-org/your-repo/abcdef123456\n \"\"\"\n owner, repo, identifier = owner_repo_identifier\n\n delete_args = {\n \"identifier\": click.style(identifier, bold=True),\n \"repository\": click.style(repo, bold=True),\n }\n\n prompt = (\n \"delete the %(identifier)s entitlement from the %(repository)s \"\n \"repository\" % delete_args\n )\n if not utils.confirm_operation(prompt, assume_yes=yes):\n return\n\n click.secho(\n \"Deleting %(identifier)s entitlement from the %(repository)s \"\n \"repository ... \" % delete_args,\n nl=False,\n )\n\n context_msg = \"Failed to delete the entitlement!\"\n with handle_api_exceptions(ctx, opts=opts, context_msg=context_msg):\n with maybe_spinner(opts):\n api.delete_entitlement(owner=owner, repo=repo, identifier=identifier)\n\n click.secho(\"OK\", fg=\"green\")"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update(ctx, opts, owner_repo_identifier, show_tokens, name, token):\n owner, repo, identifier = owner_repo_identifier\n\n # Use stderr for messages if the output is something else (e.g. # JSON)\n use_stderr = opts.output != \"pretty\"\n\n click.secho(\n \"Updating %(identifier)s entitlement for the %(repository)s \"\n \"repository ... \"\n % {\n \"identifier\": click.style(identifier, bold=True),\n \"repository\": click.style(repo, bold=True),\n },\n nl=False,\n err=use_stderr,\n )\n\n context_msg = \"Failed to update the entitlement!\"\n with handle_api_exceptions(ctx, opts=opts, context_msg=context_msg):\n with maybe_spinner(opts):\n entitlement = api.update_entitlement(\n owner=owner,\n repo=repo,\n identifier=identifier,\n name=name,\n token=token,\n show_tokens=show_tokens,\n )\n\n click.secho(\"OK\", fg=\"green\", err=use_stderr)\n\n print_entitlements(opts=opts, data=[entitlement], show_list_info=False)", "response": "Update a single resource in a repository."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef refresh(ctx, opts, owner_repo_identifier, show_tokens, yes):\n owner, repo, identifier = owner_repo_identifier\n\n refresh_args = {\n \"identifier\": click.style(identifier, bold=True),\n \"repository\": click.style(repo, bold=True),\n }\n\n # Use stderr for messages if the output is something else (e.g. # JSON)\n use_stderr = opts.output != \"pretty\"\n\n prompt = (\n \"refresh the %(identifier)s entitlement for the %(repository)s \"\n \"repository\" % refresh_args\n )\n if not utils.confirm_operation(prompt, assume_yes=yes, err=use_stderr):\n return\n\n click.secho(\n \"Refreshing %(identifier)s entitlement for the %(repository)s \"\n \"repository ... \" % refresh_args,\n nl=False,\n err=use_stderr,\n )\n\n context_msg = \"Failed to refresh the entitlement!\"\n with handle_api_exceptions(ctx, opts=opts, context_msg=context_msg):\n with maybe_spinner(opts):\n entitlement = api.refresh_entitlement(\n owner=owner, repo=repo, identifier=identifier, show_tokens=show_tokens\n )\n\n click.secho(\"OK\", fg=\"green\", err=use_stderr)\n\n print_entitlements(opts=opts, data=[entitlement], show_list_info=False)", "response": "Refresh an entitlement in a repository."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef ctime(message=None):\n \"Counts the time spent in some context\"\n t = time.time()\n yield\n if message:\n print message + \":\\t\",\n print round(time.time() - t, 4), \"sec\"", "response": "Counts the time spent in some context"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef rate_limit(client, headers, atexit=False):\n if not client or not headers:\n return False\n\n if not getattr(client.config, \"rate_limit\", False):\n return False\n\n rate_info = RateLimitsInfo.from_headers(headers)\n if not rate_info or not rate_info.interval:\n return False\n\n if rate_info.interval:\n cb = getattr(client.config, \"rate_limit_callback\", None)\n if cb and callable(cb):\n cb(rate_info, atexit=atexit)\n time.sleep(rate_info.interval)\n return True", "response": "Pause the process based on suggested rate interval."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nretrieves status and optionally version from the API.", "response": "def get_status(with_version=False):\n \"\"\"Retrieve status (and optionally) version from the API.\"\"\"\n client = get_status_api()\n\n with catch_raise_api_exception():\n data, _, headers = client.status_check_basic_with_http_info()\n\n ratelimits.maybe_rate_limit(client, headers)\n\n if with_version:\n return data.detail, data.version\n\n return data.detail"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef first(seq, key=lambda x: bool(x), default=None, apply=lambda x: x):\n return next((apply(x) for x in seq if key(x)), default() if callable(default) else default)", "response": "Give the first value that satisfies the key test."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncall each element of sequence to invoke the side effect.", "response": "def call_each(seq):\n \"\"\"Calls each element of sequence to invoke the side effect.\n\n Args:\n seq:\n\n Returns: None\n\n \"\"\"\n try:\n reduce(lambda _, y: y(), seq)\n except TypeError as e:\n if text_type(e) != \"reduce() of empty sequence with no initial value\":\n raise"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef transform_api_header_authorization(param, value):\n try:\n username, password = value.split(\":\", 1)\n except ValueError:\n raise click.BadParameter(\n \"Authorization header needs to be Authorization=username:password\",\n param=param,\n )\n\n value = \"%s:%s\" % (username.strip(), password)\n value = base64.b64encode(bytes(value.encode()))\n return \"Basic %s\" % value.decode(\"utf-8\")", "response": "Transform a username password value into a base64 string."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nvalidates that API headers are a CSV of k = v pairs.", "response": "def validate_api_headers(param, value):\n \"\"\"Validate that API headers is a CSV of k=v pairs.\"\"\"\n # pylint: disable=unused-argument\n if not value:\n return None\n\n headers = {}\n for kv in value.split(\",\"):\n try:\n k, v = kv.split(\"=\", 1)\n k = k.strip()\n\n for bad_header in BAD_API_HEADERS:\n if bad_header == k:\n raise click.BadParameter(\n \"%(key)s is not an allowed header\" % {\"key\": bad_header},\n param=param,\n )\n\n if k in API_HEADER_TRANSFORMS:\n transform_func = API_HEADER_TRANSFORMS[k]\n v = transform_func(param, v)\n except ValueError:\n raise click.BadParameter(\n \"Values need to be a CSV of key=value pairs\", param=param\n )\n\n headers[k] = v\n\n return headers"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef validate_slashes(param, value, minimum=2, maximum=None, form=None):\n try:\n value = value.split(\"/\")\n except ValueError:\n value = None\n\n if value:\n if len(value) < minimum:\n value = None\n elif maximum and len(value) > maximum:\n value = None\n\n if not value:\n form = form or \"/\".join(\"VALUE\" for _ in range(minimum))\n raise click.BadParameter(\n \"Must be in the form of %(form)s\" % {\"form\": form}, param=param\n )\n\n value = [v.strip() for v in value]\n if not all(value):\n raise click.BadParameter(\"Individual values cannot be blank\", param=param)\n\n return value", "response": "Ensure that parameter has slashes and minimum parts."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nensuring that owner and repo are formatted correctly.", "response": "def validate_owner_repo(ctx, param, value):\n \"\"\"Ensure that owner/repo is formatted correctly.\"\"\"\n # pylint: disable=unused-argument\n form = \"OWNER/REPO\"\n return validate_slashes(param, value, minimum=2, maximum=2, form=form)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef validate_owner_repo_package(ctx, param, value):\n # pylint: disable=unused-argument\n form = \"OWNER/REPO/PACKAGE\"\n return validate_slashes(param, value, minimum=3, maximum=3, form=form)", "response": "Ensure that owner. repo. package is formatted correctly."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nensuring that owner repo distro version is formatted correctly.", "response": "def validate_owner_repo_distro(ctx, param, value):\n \"\"\"Ensure that owner/repo/distro/version is formatted correctly.\"\"\"\n # pylint: disable=unused-argument\n form = \"OWNER/REPO/DISTRO[/RELEASE]\"\n return validate_slashes(param, value, minimum=3, maximum=4, form=form)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef validate_page(ctx, param, value):\n # pylint: disable=unused-argument\n if value == 0:\n raise click.BadParameter(\n \"Page is not zero-based, please set a value to 1 or higher.\", param=param\n )\n return value", "response": "Ensure that a valid value for page is chosen."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef validate_page_size(ctx, param, value):\n # pylint: disable=unused-argument\n if value == 0:\n raise click.BadParameter(\"Page size must be non-zero or unset.\", param=param)\n return value", "response": "Ensure that a valid value for page size is chosen."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget Account Asset Balance", "response": "def getbalance(self, **params):\n \"\"\"Get Account Asset Balance\n\n API Type\n --------\n HTTP Private API\n\n Docs\n ----\n https://lightning.bitflyer.jp/docs?lang=en#get-account-asset-balance\n \"\"\"\n if not all([self.api_key, self.api_secret]):\n raise AuthException()\n\n endpoint = \"/v1/me/getbalance\"\n return self.request(endpoint, params=params)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef tidy_all_the_variables(host, inventory_mgr):\n ''' removes all overridden and inherited variables from hosts\n and groups '''\n global _vars\n _vars = dict()\n _vars[host] = inventory_mgr.inventory.get_host_vars(host)\n for group in host.get_groups():\n remove_inherited_and_overridden_vars(_vars[host], group, inventory_mgr)\n remove_inherited_and_overridden_group_vars(group, inventory_mgr)\n return _vars", "response": "Removes all overridden and inherited variables from hosts\n and groups"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nmerging all entities by inventory source", "response": "def _plugins_inventory(self, entities):\n import os\n from ansible.plugins.loader import vars_loader\n from ansible.utils.vars import combine_vars\n ''' merges all entities by inventory source '''\n data = {}\n for inventory_dir in self.variable_manager._inventory._sources:\n if ',' in inventory_dir: # skip host lists\n continue\n elif not os.path.isdir(inventory_dir): # always pass 'inventory directory'\n inventory_dir = os.path.dirname(inventory_dir)\n\n for plugin in vars_loader.all():\n data = combine_vars(data, self._get_plugin_vars(plugin, inventory_dir, entities))\n return data"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef handle(self, *args, **options):\n # TODO: refactor these with .get\n if 'username' in options:\n self.username = options['username']\n else:\n self.username = None\n if 'password' in options:\n self.password = options['password']\n else:\n self.password = None\n\n self.xml_path = options.get('xml')\n self.url = options.get('url')\n try:\n blog_index = BlogIndexPage.objects.get(\n title__icontains=options['blog_index'])\n except BlogIndexPage.DoesNotExist:\n raise CommandError(\"Incorrect blog index title - have you created it?\")\n if self.url == \"just_testing\":\n with open('test-data.json') as test_json:\n posts = json.load(test_json)\n elif self.xml_path:\n try:\n import lxml\n from blog.wp_xml_parser import XML_parser\n except ImportError as e:\n print(\"You must have lxml installed to run xml imports.\"\n \" Run `pip install lxml`.\")\n raise e\n self.xml_parser = XML_parser(self.xml_path)\n posts = self.xml_parser.get_posts_data()\n else:\n posts = self.get_posts_data(self.url)\n self.should_import_comments = options.get('import_comments')\n self.create_blog_pages(posts, blog_index)", "response": "gets data from WordPress site"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate Image objects and transfer image files to media root", "response": "def create_images_from_urls_in_content(self, body):\n \"\"\"create Image objects and transfer image files to media root\"\"\"\n soup = BeautifulSoup(body, \"html5lib\")\n for img in soup.findAll('img'):\n old_url = img['src']\n if 'width' in img:\n width = img['width']\n if 'height' in img:\n height = img['height']\n else:\n width = 100\n height = 100\n path, file_ = os.path.split(img['src'])\n if not img['src']:\n continue # Blank image\n if img['src'].startswith('data:'):\n continue # Embedded image\n try:\n remote_image = urllib.request.urlretrieve(\n self.prepare_url(img['src']))\n except (urllib.error.HTTPError,\n urllib.error.URLError,\n UnicodeEncodeError,\n ValueError):\n print(\"Unable to import \" + img['src'])\n continue\n image = Image(title=file_, width=width, height=height)\n try:\n image.file.save(file_, File(open(remote_image[0], 'rb')))\n image.save()\n new_url = image.file.url\n body = body.replace(old_url, new_url)\n body = self.convert_html_entities(body)\n except TypeError:\n print(\"Unable to import image {}\".format(remote_image[0]))\n return body"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the comment object with this wordpress id", "response": "def lookup_comment_by_wordpress_id(self, comment_id, comments):\n \"\"\" Returns Django comment object with this wordpress id \"\"\"\n for comment in comments:\n if comment.wordpress_id == comment_id:\n return comment"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate blog entries from wordpress data", "response": "def create_blog_pages(self, posts, blog_index, *args, **options):\n \"\"\"create Blog post entries from wordpress data\"\"\"\n for post in posts:\n post_id = post.get('ID')\n title = post.get('title')\n if title:\n new_title = self.convert_html_entities(title)\n title = new_title\n slug = post.get('slug')\n description = post.get('description')\n if description:\n description = self.convert_html_entities(description)\n\n body = post.get('content')\n if not \"

\" in body:\n body = linebreaks(body)\n\n # get image info from content and create image objects\n body = self.create_images_from_urls_in_content(body)\n\n # author/user data\n author = post.get('author')\n user = self.create_user(author)\n categories = post.get('terms')\n # format the date\n date = post.get('date')[:10]\n try:\n new_entry = BlogPage.objects.get(slug=slug)\n new_entry.title = title\n new_entry.body = body\n new_entry.owner = user\n new_entry.save()\n except BlogPage.DoesNotExist:\n new_entry = blog_index.add_child(instance=BlogPage(\n title=title, slug=slug, search_description=\"description\",\n date=date, body=body, owner=user))\n featured_image = post.get('featured_image')\n if featured_image is not None:\n title = post['featured_image']['title']\n source = post['featured_image']['source']\n path, file_ = os.path.split(source)\n source = source.replace('stage.swoon', 'swoon')\n try:\n remote_image = urllib.request.urlretrieve(\n self.prepare_url(source))\n width = 640\n height = 290\n header_image = Image(title=title, width=width, height=height)\n header_image.file.save(\n file_, File(open(remote_image[0], 'rb')))\n header_image.save()\n except UnicodeEncodeError:\n header_image = None\n print('unable to set header image {}'.format(source))\n else:\n header_image = None\n new_entry.header_image = header_image\n new_entry.save()\n if categories:\n self.create_categories_and_tags(new_entry, categories)\n if self.should_import_comments:\n self.import_comments(post_id, slug)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate and stores a unique slug of value for an instance.", "response": "def unique_slugify(instance, value, slug_field_name='slug', queryset=None,\n slug_separator='-'):\n \"\"\"\n Calculates and stores a unique slug of ``value`` for an instance.\n\n ``slug_field_name`` should be a string matching the name of the field to\n store the slug in (and the field to check against for uniqueness).\n\n ``queryset`` usually doesn't need to be explicitly provided - it'll default\n to using the ``.all()`` queryset from the model's default manager.\n \"\"\"\n slug_field = instance._meta.get_field(slug_field_name)\n\n slug = getattr(instance, slug_field.attname)\n slug_len = slug_field.max_length\n\n # Sort out the initial slug, limiting its length if necessary.\n slug = slugify(value)\n if slug_len:\n slug = slug[:slug_len]\n slug = _slug_strip(slug, slug_separator)\n original_slug = slug\n\n # Create the queryset if one wasn't explicitly provided and exclude the\n # current instance from the queryset.\n if queryset is None:\n queryset = instance.__class__._default_manager.all()\n if instance.pk:\n queryset = queryset.exclude(pk=instance.pk)\n\n # Find a unique slug. If one matches, at '-2' to the end and try again\n # (then '-3', etc).\n next = 2\n while not slug or queryset.filter(**{slug_field_name: slug}):\n slug = original_slug\n end = '%s%s' % (slug_separator, next)\n if slug_len and len(slug) + len(end) > slug_len:\n slug = slug[:slug_len-len(end)]\n slug = _slug_strip(slug, slug_separator)\n slug = '%s%s' % (slug, end)\n next += 1\n\n setattr(instance, slug_field.attname, slug)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncleaning up a slug by removing separator characters that occur at the beginning and end of a slug.", "response": "def _slug_strip(value, separator='-'):\n \"\"\"\n Cleans up a slug by removing slug separator characters that occur at the\n beginning or end of a slug.\n\n If an alternate separator is used, it will also replace any instances of\n the default '-' separator with the new separator.\n \"\"\"\n separator = separator or ''\n if separator == '-' or not separator:\n re_sep = '-'\n else:\n re_sep = '(?:-|%s)' % re.escape(separator)\n # Remove multiple instances and if an alternate separator is provided,\n # replace the default '-' separator.\n if separator != re_sep:\n value = re.sub('%s+' % re_sep, separator, value)\n # Remove separator from the beginning and end of the slug.\n if separator:\n if separator != '-':\n re_sep = re.escape(separator)\n value = re.sub(r'^%s+|%s+$' % (re_sep, re_sep), '', value)\n return value"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nlimit choices in blog author field based on config settings", "response": "def limit_author_choices():\n \"\"\" Limit choices in blog author field based on config settings \"\"\"\n LIMIT_AUTHOR_CHOICES = getattr(settings, 'BLOG_LIMIT_AUTHOR_CHOICES_GROUP', None)\n if LIMIT_AUTHOR_CHOICES:\n if isinstance(LIMIT_AUTHOR_CHOICES, str):\n limit = Q(groups__name=LIMIT_AUTHOR_CHOICES)\n else:\n limit = Q()\n for s in LIMIT_AUTHOR_CHOICES:\n limit = limit | Q(groups__name=s)\n if getattr(settings, 'BLOG_LIMIT_AUTHOR_CHOICES_ADMIN', False):\n limit = limit | Q(is_staff=True)\n else:\n limit = {'is_staff': True}\n return limit"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_blog_context(context):\n context['authors'] = get_user_model().objects.filter(\n owned_pages__live=True,\n owned_pages__content_type__model='blogpage'\n ).annotate(Count('owned_pages')).order_by('-owned_pages__count')\n context['all_categories'] = BlogCategory.objects.all()\n context['root_categories'] = BlogCategory.objects.filter(\n parent=None,\n ).prefetch_related(\n 'children',\n ).annotate(\n blog_count=Count('blogpage'),\n )\n return context", "response": "Get context data useful on all blog related pages"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef remove_xmlns(xml_string):\n # splitting xml into sections, pre_chan is preamble before \n pre_chan, chan, post_chan= xml_string.partition('')\n # replace xmlns statements on preamble\n pre_chan = re.sub(r'xmlns:(?P