{"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfinds the VERSION_FILE and return its contents.", "response": "def get_version_from_file(path):\n \"\"\"Find the VERSION_FILE and return its contents.\n\n Returns\n -------\n version : string or None\n\n \"\"\"\n filename = os.path.join(path, VERSION_FILE)\n if not os.path.isfile(filename):\n # Look in the parent directory of path instead.\n filename = os.path.join(os.path.dirname(path), VERSION_FILE)\n if not os.path.isfile(filename):\n filename = ''\n\n if filename:\n with open(filename) as fh:\n version = fh.readline().strip()\n if version:\n return version"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nnormalising a version string according to PEP 440 if possible.", "response": "def normalised(version):\n \"\"\"Normalise a version string according to PEP 440, if possible.\"\"\"\n norm_version = pkg_resources.parse_version(version)\n if not isinstance(norm_version, tuple):\n # Let setuptools (>= 8) do the normalisation\n return str(norm_version)\n else:\n # Homegrown normalisation for older setuptools (< 8)\n public, sep, local = version.lower().partition('+')\n # Remove leading 'v' from public version\n if len(public) >= 2:\n if public[0] == 'v' and public[1] in '0123456789':\n public = public[1:]\n # Turn all chars except alphanumerics into periods in local version\n local = NON_ALPHANUMERIC.sub('.', local)\n return public + sep + local"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_version(path=None, module=None):\n # Check the module option first.\n version = get_version_from_module(module)\n if version:\n return normalised(version)\n\n # Turn path into a valid directory (default is current directory)\n if path is None:\n path = os.getcwd()\n path = os.path.abspath(path)\n if os.path.exists(path) and not os.path.isdir(path):\n path = os.path.dirname(path)\n if not os.path.isdir(path):\n raise ValueError('No such package source directory: %r' % (path,))\n\n # Check for an sdist in the process of being installed by pip.\n version = get_version_from_unpacked_sdist(path)\n if version:\n return normalised(version)\n\n # Check the SCM.\n scm, version = get_version_from_scm(path)\n if version:\n return normalised(version)\n\n # Check if there is a katversion file in the given path.\n version = get_version_from_file(path)\n if version:\n return normalised(version)\n\n # None of the above got a version so we will make one up based on the date.\n return normalised(date_version(scm))", "response": "Returns the version string of the current version of the current version of the current version of the current version."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _sane_version_list(version):\n v0 = str(version[0])\n if v0:\n # Test if the major is a number.\n try:\n v0 = v0.lstrip(\"v\").lstrip(\"V\")\n # Handle the common case where tags have v before major.\n v0 = int(v0)\n except ValueError:\n v0 = None\n\n if v0 is None:\n version = [0, 0] + version\n else:\n version[0] = v0\n\n try:\n # Test if the minor is a number.\n version[1] = int(version[1])\n except ValueError:\n # Insert Minor 0.\n version = [version[0], 0] + version[1:]\n\n return version", "response": "Ensure the major and minor numbers are int."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_version_list(path=None, module=None):\n major = 0\n minor = 0\n patch = '' # PEP440 calls this prerelease, postrelease or devrelease\n ver = get_version(path, module)\n if ver is not None:\n ver_segments = _sane_version_list(ver.split(\".\", 2))\n major = ver_segments[0]\n minor = ver_segments[1]\n patch = \".\".join(ver_segments[2:]) # Rejoin the .\n\n # Return None as first field, makes substitution easier in next step.\n return [None, major, minor, patch]", "response": "Return the version information as a tuple."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the build info tuple.", "response": "def build_info(name, path=None, module=None):\n \"\"\"Return the build info tuple.\"\"\"\n verlist = get_version_list(path, module)\n verlist[0] = name\n return tuple(verlist)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfinding all of the indices of particles of groupid", "response": "def find(self, groupid):\n \"\"\" return all of the indices of particles of groupid \"\"\"\n return self.indices[self.offset[groupid]\n :self.offset[groupid]+ self.length[groupid]]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef sum(self, weights=None):\n if weights is None:\n weights = self.data.weights\n return utils.bincount(self.labels, weights, self.N)", "response": "return the sum of weights of each object"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the center of each object", "response": "def center(self, weights=None):\n \"\"\" return the center of each object \"\"\"\n if weights is None:\n weights = self.data.weights\n mass = utils.bincount(self.labels, weights, self.N)\n cp = numpy.empty((len(mass), self.data.pos.shape[-1]), 'f8')\n for d in range(self.data.pos.shape[-1]):\n cp[..., d] = utils.bincount(self.labels, weights *\n self.data.pos[..., d], self.N)\n cp[..., d] /= mass\n return cp"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef getParamValues(self, paramName=None, paramId=None, useOnlyValids=True):\n \n if not paramName is None:\n if not paramId is None:\n if getParameterTypeNameFromID(paramId) != paramName:\n raise ValueError(\"Parameters paramId and paramName \"\n + \"passed to ParamSample.getParamValues() are incompatible.\")\n else:\n if paramId is None:\n raise ValueError(\"At least one of the attribute paramName and paramId \"\n + \"passed to ParamSample.getParamValues() most not be None.\")\n paramName = getParameterTypeNameFromID(paramId)\n \n df = self.sampleDF\n if useOnlyValids:\n df = df[df[\"isValid\"] == True]\n\n df.loc[:, \"paramNames\"] = [getParameterTypeNameFromID(param.typeId) for param in df[\"obj_parameter\"]]\n\n return df[df[\"paramNames\"] == paramName]", "response": "Return the rows of sampleDF that are associated to the parameter paramName."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef interpolate(self, interpValues):\n self.__operations.append([\"interpolate\", interpValues]) \n \n df = self.sampleDF\n self.interpValues = interpValues\n for interParamName, value in interpValues.items():\n self.__report += \"Interpolation of the parameters for independent variables '\" \\\n + interParamName + \"' at value \" + str(value) + \".\\n\" \n for ind, (paramTrace, resType) in enumerate(zip(df[\"obj_parameter\"], df[\"Result type\"])):\n if resType == \"numericalTrace\" and interParamName in paramTrace.indepNames:\n val = paramTrace.getInterp1dValues(value, interParamName, statsToReturn=[\"mean\"])\n if isinstance(val, list):\n if len(val) == 1:\n val = val[0]\n else:\n raise ValueError(\"This case has not been implemented yet.\")\n df.loc[ind, \"Values\"] = float(val)", "response": "This method is called by the interpolation functions in the base class to interpolate the parameters of the independant variables."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncomparing two Psisloo objects using pointwise approximate leave - one - out cross validation.", "response": "def loo_compare(psisloo1, psisloo2):\n \"\"\"\n Compares two models using pointwise approximate leave-one-out cross validation.\n \n For the method to be valid, the two models should have been fit on the same input data. \n\n Parameters\n -------------------\n psisloo1 : Psisloo object for model1\n psisloo2 : Psisloo object for model2\n\n Returns\n -------------------\n Dict with two values:\n\n diff: difference in elpd (estimated log predictive density) \n between two models, where a positive value indicates\n that model2 is a better fit than model1.\n\n se_diff: estimated standard error of the difference\n between model2 & model1.\n\n \"\"\"\n ## TODO: confirm that dimensions for psisloo1 & psisloo2 are the same\n loores = psisloo1.pointwise.join(\n psisloo2.pointwise,\n lsuffix = '_m1',\n rsuffix = '_m2')\n\n loores['pw_diff'] = loores.pointwise_elpd_m2 - loores.pointwise_elpd_m1\n\n sum_elpd_diff = loores.apply(numpy.sum).pw_diff\n sd_elpd_diff = loores.apply(numpy.std).pw_diff\n\n elpd_diff = {\n 'diff' : sum_elpd_diff,\n 'se_diff' : math.sqrt(len(loores.pw_diff)) * sd_elpd_diff\n }\n \n return elpd_diff"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nplot the pointwise importance - sampling indices of the observation units.", "response": "def plot(self):\n \"\"\" Graphical summary of pointwise pareto-k importance-sampling indices\n\n Pareto-k tail indices are plotted (on the y axis) for each observation unit (on the x axis)\n\n \"\"\"\n seaborn.pointplot(\n y = self.pointwise.pareto_k,\n x = self.pointwise.index,\n join = False)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nadding a new layer to the stack", "response": "def add_layer(self, formula='', thickness=np.NaN, density=np.NaN):\n \"\"\"provide another way to define the layers (stack)\n\n Parameters:\n ===========\n formula: string\n ex: 'CoAg2'\n ex: 'Al'\n thickness: float (in mm)\n density: float (g/cm3)\n \"\"\"\n if formula == '':\n return\n\n _new_stack = _utilities.formula_to_dictionary(formula=formula,\n thickness=thickness,\n density=density,\n database=self.database)\n # check if density has been defined\n self.__lock_density_if_defined(stack=_new_stack)\n\n new_stack = self.__update_stack_with_isotopes_infos(stack=_new_stack)\n self.stack = {**self.stack, **new_stack}\n\n # calculate stack_sigma, layer density, atoms_per_cm3 ...\n self.__math_on_stack()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_isotopic_ratio(self, compound='', element=''):\n _stack = self.stack\n compound = str(compound)\n\n if compound == '':\n _list_compounds = _stack.keys()\n list_all_dict = {}\n for _compound in _list_compounds:\n _compound = str(_compound)\n _list_element = _stack[_compound]['elements']\n list_all_dict[_compound] = {}\n for _element in _list_element:\n list_all_dict[_compound][_element] = self.get_isotopic_ratio(\n compound=_compound,\n element=_element)\n return list_all_dict\n\n # checking compound is valid\n list_compounds = _stack.keys()\n if compound not in list_compounds:\n list_compounds_joined = ', '.join(list_compounds)\n raise ValueError(\"Compound '{}' could not be find in {}\".format(compound, list_compounds_joined))\n\n # checking element is valid\n if element == '':\n # we assume that the element and compounds names matched\n element = compound\n list_element = _stack[compound].keys()\n if element not in list_element:\n list_element_joined = ', '.join(list_element)\n raise ValueError(\"Element '{}' should be any of those elements: {}\".format(element, list_element_joined))\n\n list_istopes = _stack[compound][element]['isotopes']['list']\n list_ratio = _stack[compound][element]['isotopes']['isotopic_ratio']\n iso_ratio = zip(list_istopes, list_ratio)\n\n _stoichiometric_ratio = {}\n for _iso, _ratio in iso_ratio:\n _stoichiometric_ratio[_iso] = _ratio\n\n return _stoichiometric_ratio", "response": "returns the list of isotopes for the element of the compound defined with their stoichiometric values"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_isotopic_ratio(self, compound='', element='', list_ratio=[]):\n _stack = self.stack\n\n list_compounds = _stack.keys()\n if compound not in _stack.keys():\n list_compounds_joined = ', '.join(list_compounds)\n raise ValueError(\"Compound '{}' could not be find in {}\".format(compound, list_compounds_joined))\n\n if element == '':\n # we assume that the element and compounds names matched\n element = compound\n list_element = _stack[compound].keys()\n if element not in list_element:\n list_element_joined = ', '.join(list_element)\n raise ValueError(\"Element '{}' should be any of those elements: {}\".format(element, list_element_joined))\n\n old_list_ratio = _stack[compound][element]['isotopes']['list']\n if not (len(old_list_ratio) == len(list_ratio)):\n raise ValueError(\"New list of ratio ({} elements) does not match old list size ({} elements!\".format(len(\n list_ratio), len(old_list_ratio)))\n\n self.stack[compound][element]['isotopes']['isotopic_ratio'] = list_ratio\n self.__update_molar_mass(compound=compound, element=element)\n self.__update_density(compound=compound, element=element)\n\n # update entire stack\n self.__math_on_stack()", "response": "This function updates the isotopic ratio of the current set of elements in the current set of elements."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_density(self, compound='', element=''):\n _stack = self.stack\n\n if compound == '':\n _list_compounds = _stack.keys()\n list_all_dict = {}\n for _compound in _list_compounds:\n _list_element = _stack[_compound]['elements']\n list_all_dict[_compound] = {}\n for _element in _list_element:\n list_all_dict[_compound][_element] = self.get_density(\n compound=_compound,\n element=_element)\n return list_all_dict\n\n # checking compound is valid\n list_compounds = _stack.keys()\n if compound not in list_compounds:\n list_compounds_joined = ', '.join(list_compounds)\n raise ValueError(\"Compound '{}' could not be find in {}\".format(compile, list_compounds_joined))\n\n # checking element is valid\n if element == '':\n # we assume that the element and compounds names matched\n element = compound\n list_element = _stack[compound].keys()\n if element not in list_element:\n list_element_joined = ', '.join(list_element)\n raise ValueError(\"Element '{}' should be any of those elements: {}\".format(element, list_element_joined))\n\n return _stack[compound][element]['density']['value']", "response": "returns the list of isotopes for the element of the compound defined with their density"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef __math_on_stack(self, used_lock=False):\n\n # populate stack_sigma (Sigma vs Energy for every element)\n self.__get_sigmas()\n\n # populate compound density (if none provided)\n self.__update_layer_density()\n\n # populate compound molar mass\n # self.__update_layer_molar_mass() ### included in __calculate_atoms_per_cm3\n\n # populate atoms_per_cm3\n self.__calculate_atoms_per_cm3(used_lock=used_lock)\n\n # calculate transmission and attenuation\n self.__calculate_transmission_attenuation()", "response": "This method is called by the module that is going to be run on the stack."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nlocking the density if the density has been defined during initialization.", "response": "def __lock_density_if_defined(self, stack: dict):\n \"\"\"lock (True) the density lock if the density has been been defined during initialization\n Store the resulting dictionary into density_lock\n\n Parameters:\n ===========\n stack: dictionary (optional)\n if not provided, the entire stack will be used\n \"\"\"\n\n if self.stack == {}:\n density_lock = {}\n else:\n density_lock = self.density_lock\n\n for _compound in stack.keys():\n _density = stack[_compound]['density']['value']\n if np.isnan(_density):\n density_lock[_compound] = False\n else:\n density_lock[_compound] = True\n self.density_lock = density_lock"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncalculates for each element the atoms per cm3", "response": "def __calculate_atoms_per_cm3(self, used_lock=False):\n \"\"\"calculate for each element, the atoms per cm3\"\"\"\n stack = self.stack\n _density_lock = self.density_lock\n\n for _name_of_compound in stack.keys():\n if used_lock and _density_lock[_name_of_compound]:\n continue\n molar_mass_layer, atoms_per_cm3_layer = _utilities.get_atoms_per_cm3_of_layer(\n compound_dict=stack[_name_of_compound])\n # Update layer molar mass\n stack[_name_of_compound]['molar_mass'] = {'value': molar_mass_layer,\n 'units': 'g/mol'}\n # Update atoms per cm3\n stack[_name_of_compound]['atoms_per_cm3'] = atoms_per_cm3_layer\n for _index, _name_of_ele in enumerate(stack[_name_of_compound]['elements']):\n stack[_name_of_compound][_name_of_ele]['atoms_per_cm3'] = atoms_per_cm3_layer * \\\n stack[_name_of_compound][\n 'stoichiometric_ratio'][_index]\n self.stack = stack"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef __update_stack_with_isotopes_infos(self, stack: dict):\n for _key in stack:\n _elements = stack[_key]['elements']\n for _element in _elements:\n _dict = _utilities.get_isotope_dicts(element=_element, database=self.database)\n stack[_key][_element] = _dict\n\n stack = self.__fill_missing_keys(stack=stack)\n return stack", "response": "update the stack with isotopes file names mass and atomic_ratio from each element in stack"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef __update_layer_density(self):\n _stack = self.stack\n _density_lock = self.density_lock\n list_compound = _stack.keys()\n for _key in list_compound:\n if _density_lock[_key]:\n continue\n\n _list_ratio = _stack[_key]['stoichiometric_ratio']\n _list_density = []\n for _element in _stack[_key]['elements']:\n _list_density.append(_stack[_key][_element]['density']['value'])\n _compound_density = _utilities.get_compound_density(list_density=_list_density,\n list_ratio=_list_ratio)\n\n _stack[_key]['density']['value'] = _compound_density\n self.stack = _stack", "response": "calculate or update the layer density"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef __update_molar_mass(self, compound='', element=''):\n _molar_mass_element = 0\n list_ratio = self.stack[compound][element]['isotopes']['isotopic_ratio']\n list_mass = self.stack[compound][element]['isotopes']['mass']['value']\n ratio_mass = zip(list_ratio, list_mass)\n for _ratio, _mass in ratio_mass:\n _molar_mass_element += np.float(_ratio) * np.float(_mass)\n self.stack[compound][element]['molar_mass']['value'] = _molar_mass_element", "response": "Update the molar mass of the element given due to stoichiometric changes"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef plot(self, y_axis='attenuation', x_axis='energy',\n logx=False, logy=False,\n mixed=True, all_layers=False, all_elements=False,\n all_isotopes=False, items_to_plot=None,\n time_unit='us', offset_us=0., source_to_detector_m=16.,\n time_resolution_us=0.16, t_start_us=1,\n plotly=False, ax_mpl=None,\n fmt='-', ms='2', lw='1.5', alpha=1):\n # offset delay values is normal 2.99 us with NONE actual MCP delay settings\n \"\"\"display the transmission or attenuation of compound, element and/or isotopes specified\n\n Parameters:\n ===========\n :param x_axis: x type for export. Must be either ['energy'|'lambda'|'time'|'number']\n :type x_axis: str\n :param y_axis: y type for export. Must be either ['transmission'|'attenuation'|'sigma'|'sigma_raw'|'miu_per_cm']\n :type y_axis: str\n :param logx: True -> display x in log scale\n :type logx: boolean.\n :param logy: True -> display y in log scale\n :type logy: boolean.\n :param mixed: boolean. True -> display the total of each layer\n False -> not displayed\n :param all_layers: boolean. True -> display all layers\n False -> not displayed\n :param all_elements: boolean. True -> display all elements signal\n False -> not displayed\n :param all_isotopes: boolean. True -> display all isotopes signal\n False -> not displayed\n :param items_to_plot: array that describes what to plot\n ex:\n [['CoAg','Ag','107-Ag'], ['CoAg']]\n if the dictionary is empty, everything is exported\n :param time_unit: string. Must be either ['s'|'us'|'ns']\n Note: this will be used only when x_axis='time'\n :param offset_us: default: 0\n Note: only used when x_axis='number' or 'time'\n :param source_to_detector_m: Note: this will be used only when x_axis='number' or 'time'\n :param time_resolution_us: Note: this will be used only when x_axis='number'\n :param t_start_us: when is the first acquisition occurred. default: 1\n Note: this will be used only when x_axis='number'\n :param plotly: control to use plotly to display or not.\n :type plotly: bool\n :param ax_mpl: matplotlib.axes to plot against\n :type ax_mpl: matplotlib.axes\n :param fmt: matplotlib.axes.plot kwargs\n :type fmt: str\n :param ms: matplotlib.axes.plot kwargs\n :type ms: float\n :param lw: matplotlib.axes.plot kwargs\n :type lw: float\n :param alpha: matplotlib.axes.plot kwargs\n :type alpha: float\n\n \"\"\"\n if x_axis not in x_type_list:\n raise ValueError(\"Please specify the x-axis type using one from '{}'.\".format(x_type_list))\n if time_unit not in time_unit_list:\n raise ValueError(\"Please specify the time unit using one from '{}'.\".format(time_unit_list))\n if y_axis not in y_type_list:\n raise ValueError(\"Please specify the y-axis type using one from '{}'.\".format(y_type_list))\n # figure size\n # plt.figure(figsize=(8, 8))\n\n # stack from self\n _stack_signal = self.stack_signal\n _stack = self.stack\n\n _stack_sigma = self.stack_sigma\n _x_axis = self.total_signal['energy_eV']\n x_axis_label = None\n\n # Creating the matplotlib graph..\n if ax_mpl is None:\n fig_mpl, ax_mpl = plt.subplots()\n\n \"\"\"X-axis\"\"\"\n # determine values and labels for x-axis with options from\n # 'energy(eV)' & 'lambda(A)' & 'time(us)' & 'image number(#)'\n if x_axis == 'energy':\n x_axis_label = 'Energy (eV)'\n if x_axis == 'lambda':\n x_axis_label = u\"Wavelength (\\u212B)\"\n _x_axis = _utilities.ev_to_angstroms(array=_x_axis)\n if x_axis == 'time':\n if time_unit == 's':\n x_axis_label = 'Time (s)'\n _x_axis = _utilities.ev_to_s(array=_x_axis,\n source_to_detector_m=source_to_detector_m,\n offset_us=offset_us)\n if time_unit == 'us':\n x_axis_label = 'Time (us)'\n _x_axis = 1e6 * _utilities.ev_to_s(array=_x_axis,\n source_to_detector_m=source_to_detector_m,\n offset_us=offset_us)\n if time_unit == 'ns':\n x_axis_label = 'Time (ns)'\n _x_axis = 1e9 * _utilities.ev_to_s(array=_x_axis,\n source_to_detector_m=source_to_detector_m,\n offset_us=offset_us)\n print(\"'{}' was obtained with the following:\\nsource_to_detector_m={}\\noffset_us={}\"\n .format(x_axis_label, source_to_detector_m, offset_us))\n\n if x_axis == 'number':\n x_axis_label = 'Image number (#)'\n _x_axis = _utilities.ev_to_image_number(array=_x_axis,\n source_to_detector_m=source_to_detector_m,\n offset_us=offset_us,\n time_resolution_us=time_resolution_us,\n t_start_us=t_start_us)\n print(\"'{}' was obtained with the following:\\nsource_to_detector_m={}\\noffset_us={}\\ntime_resolution_us={}\"\n .format(x_axis_label, source_to_detector_m, offset_us, time_resolution_us))\n if x_axis_label is None:\n raise ValueError(\"x_axis_label does NOT exist, please check.\")\n\n \"\"\"Y-axis\"\"\"\n # determine to plot transmission or attenuation\n # determine to put transmission or attenuation words for y-axis\n y_axis_tag = y_axis\n if y_axis == 'transmission':\n y_axis_label = 'Neutron Transmission'\n elif y_axis == 'attenuation':\n y_axis_label = 'Neutron Attenuation'\n elif y_axis == 'sigma':\n y_axis_tag = 'sigma_b'\n y_axis_label = 'Cross-section (barns)'\n elif y_axis == 'sigma_raw':\n y_axis_tag = 'sigma_b_raw'\n y_axis_label = 'Cross-section (barns)'\n else:\n y_axis_tag = 'miu_per_cm'\n y_axis_label = \"Attenuation coefficient (cm\\u207B\\u00B9)\"\n\n if y_axis_tag[:5] == 'sigma':\n mixed = False\n all_layers = False\n print(\"'y_axis='sigma'' is selected. Auto force 'mixed=False', 'all_layers=False'\")\n if y_axis_tag[-3:] == 'raw':\n all_elements = False\n print(\"'y_axis='sigma_raw'' is selected. Auto force 'all_elements=False'\")\n\n if y_axis_tag == 'miu_per_cm':\n mixed = False\n print(\"'y_axis='miu_per_cm'' is selected. Auto force 'mixed=False'\")\n\n # Plotting begins\n if mixed:\n _y_axis = self.total_signal[y_axis_tag]\n ax_mpl.plot(_x_axis, _y_axis, fmt, ms=ms, lw=lw, alpha=alpha, label=\"Total\")\n\n if all_layers:\n for _compound in _stack.keys():\n _y_axis = _stack_signal[_compound][y_axis_tag]\n ax_mpl.plot(_x_axis, _y_axis, fmt, ms=ms, lw=lw, alpha=alpha, label=_compound)\n\n if all_elements:\n for _compound in _stack.keys():\n for _element in _stack[_compound]['elements']:\n if y_axis_tag[:5] != 'sigma':\n _y_axis = _stack_signal[_compound][_element][y_axis_tag]\n ax_mpl.plot(_x_axis, _y_axis, fmt, ms=ms, lw=lw, alpha=alpha,\n label=\"{}/{}\".format(_compound, _element))\n else:\n _y_axis = _stack_sigma[_compound][_element]['sigma_b']\n ax_mpl.plot(_x_axis, _y_axis, fmt, ms=ms, lw=lw, alpha=alpha,\n label=\"{}/{}\".format(_compound, _element))\n\n if all_isotopes:\n for _compound in _stack.keys():\n for _element in _stack[_compound]['elements']:\n for _isotope in _stack[_compound][_element]['isotopes']['list']:\n if y_axis_tag[:5] != 'sigma':\n _y_axis = _stack_signal[_compound][_element][_isotope][y_axis_tag]\n ax_mpl.plot(_x_axis, _y_axis, fmt, ms=ms, lw=lw, alpha=alpha,\n label=\"{}/{}/{}\".format(_compound, _element, _isotope))\n else:\n _y_axis = _stack_sigma[_compound][_element][_isotope][y_axis_tag]\n ax_mpl.plot(_x_axis, _y_axis, fmt, ms=ms, lw=lw, alpha=alpha,\n label=\"{}/{}/{}\".format(_compound, _element, _isotope))\n\n \"\"\"Y-axis for specified items_to_plot\"\"\"\n if items_to_plot is not None:\n for _path_to_plot in items_to_plot:\n _path_to_plot = list(_path_to_plot)\n if y_axis_tag[:5] != 'sigma':\n _live_path = _stack_signal\n else:\n _len_of_path = len(_path_to_plot)\n if y_axis_tag[-3:] == 'raw':\n if _len_of_path < 3:\n raise ValueError(\"'y_axis={}' is not supported for layer or element levels '{}'.\".format(\n y_axis_tag, _path_to_plot[-1]))\n else:\n if _len_of_path < 2:\n raise ValueError(\"'y_axis={}' is not supported for layer level '{}'.\".format(\n y_axis_tag, _path_to_plot[-1]))\n\n _live_path = _stack_sigma\n _label = \"/\".join(_path_to_plot)\n while _path_to_plot:\n _item = _path_to_plot.pop(0)\n _live_path = _live_path[_item]\n _y_axis = _live_path[y_axis_tag]\n ax_mpl.plot(_x_axis, _y_axis, fmt, ms=ms, lw=lw, alpha=alpha, label=_label)\n\n if y_axis_tag[:5] != 'sigma' and y_axis_tag != 'miu_per_cm':\n ax_mpl.set_ylim(-0.01, 1.01)\n if logy is True:\n ax_mpl.set_yscale('log')\n if logx is True:\n ax_mpl.set_xscale('log')\n ax_mpl.set_xlabel(x_axis_label)\n ax_mpl.set_ylabel(y_axis_label)\n if not plotly:\n ax_mpl.legend(loc='best')\n # plt.tight_layout()\n return ax_mpl\n else:\n fig_mpl = ax_mpl.get_figure()\n plotly_fig = tls.mpl_to_plotly(fig_mpl)\n plotly_fig.layout.showlegend = True\n return plotly_fig", "response": "Plot the transmission or attenuation of the compound element and or isotope of the current language."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef export(self, output_type='df', filename=None, x_axis='energy', y_axis='attenuation', mixed=True,\n all_layers=False, all_elements=False, all_isotopes=False, items_to_export=None,\n offset_us=0., source_to_detector_m=16.,\n t_start_us=1, time_resolution_us=0.16, time_unit='us'):\n \"\"\"\n output x and y values to clipboard or .csv file\n output the transmission or attenuation or sigma of compound, element and/or isotopes specified\n 'sigma_b' exported for each isotope is the product resulted from (sigma * isotopic ratio)\n 'atoms_per_cm3' of each element is also exported in 'sigma' mode based on molar mass within stack.\n\n :param output_type: export type : ['df', 'csv', 'clip']\n :type output_type: str\n :param mixed: True -> display the total of each layer\n False -> not displayed\n :type mixed: boolean\n :param filename: string. filename (with .csv suffix) you would like to save as\n None -> export to clipboard\n :type filename: string\n :param x_axis: string. x type for export. Must in ['energy', 'lambda', 'time', 'number']\n :param y_axis: string. y type for export. Must in ['transmission', 'attenuation', 'sigma', 'sigma_raw', 'miu_per_cm']\n :param all_layers: boolean. True -> export all layers\n False -> not export\n :param all_elements: boolean. True -> export all elements signal\n False -> not export\n :param all_isotopes: boolean. True -> export all isotopes signal\n False -> not export\n :param items_to_export: array that describes what to export\n ex:\n [['CoAg','Ag','107-Ag'], ['CoAg']]\n if the dictionary is empty, everything is exported\n :param time_unit: string. Must be either 's' or 'us' or 'ns'\n Note: this will be used only when x_axis='time'\n :param offset_us: default: 0\n Note: only used when x_axis='number' or 'time'\n :param source_to_detector_m: Note: this will be used only when x_axis='number' or 'time'\n :param time_resolution_us: Note: this will be used only when x_axis='number'\n :param t_start_us: when is the first acquisition occurred. default: 1\n Note: this will be used only when x_axis='number'\n\n :return: simulated resonance signals or sigma in the form of 'clipboard' or '.csv file' or 'pd.DataFrame'\n \"\"\"\n if x_axis not in x_type_list:\n raise ValueError(\"Please specify the x-axis type using one from '{}'.\".format(x_type_list))\n if time_unit not in time_unit_list:\n raise ValueError(\"Please specify the time unit using one from '{}'.\".format(time_unit_list))\n if y_axis not in y_type_list:\n raise ValueError(\"Please specify the y-axis type using one from '{}'.\".format(y_type_list))\n if output_type not in export_type_list:\n raise ValueError(\"Please specify export type using one from '{}'.\".format(export_type_list))\n # stack from self\n _stack_signal = self.stack_signal\n _stack = self.stack\n\n _x_axis = self.total_signal['energy_eV']\n x_axis_label = None\n df = pd.DataFrame()\n\n \"\"\"X-axis\"\"\"\n # determine values and labels for x-axis with options from\n # 'energy(eV)' & 'lambda(A)' & 'time(us)' & 'image number(#)'\n if x_axis == 'energy':\n x_axis_label = 'Energy (eV)'\n if x_axis == 'lambda':\n x_axis_label = u\"Wavelength (\\u212B)\"\n _x_axis = _utilities.ev_to_angstroms(array=_x_axis)\n if x_axis == 'time':\n if time_unit == 's':\n x_axis_label = 'Time (s)'\n _x_axis = _utilities.ev_to_s(array=_x_axis,\n source_to_detector_m=source_to_detector_m,\n offset_us=offset_us)\n if time_unit == 'us':\n x_axis_label = 'Time (us)'\n _x_axis = 1e6 * _utilities.ev_to_s(array=_x_axis,\n source_to_detector_m=source_to_detector_m,\n offset_us=offset_us)\n if time_unit == 'ns':\n x_axis_label = 'Time (ns)'\n _x_axis = 1e9 * _utilities.ev_to_s(array=_x_axis,\n source_to_detector_m=source_to_detector_m,\n offset_us=offset_us)\n print(\"'{}' was obtained with the following:\\nsource_to_detector_m={}\\noffset_us={}\"\n .format(x_axis_label, source_to_detector_m, offset_us))\n\n if x_axis == 'number':\n x_axis_label = 'Image number (#)'\n _x_axis = _utilities.ev_to_image_number(array=_x_axis,\n source_to_detector_m=source_to_detector_m,\n offset_us=offset_us,\n time_resolution_us=time_resolution_us,\n t_start_us=t_start_us)\n print(\"'{}' was obtained with the following:\\nsource_to_detector_m={}\\noffset_us={}\\ntime_resolution_us={}\"\n .format(x_axis_label, source_to_detector_m, offset_us, time_resolution_us))\n\n if x_axis_label is None:\n raise ValueError(\"x_axis_label does NOT exist, please check.\")\n df[x_axis_label] = _x_axis\n\n \"\"\"Y-axis\"\"\"\n if y_axis[:5] != 'sigma':\n # export transmission or attenuation or miu_per_cm\n y_axis_tag = y_axis\n if y_axis_tag == 'miu_per_cm':\n mixed = False\n print(\"'y_axis='miu_per_cm'' is selected. Auto force 'mixed=False'\")\n if mixed:\n _y_axis = self.total_signal[y_axis_tag]\n df['Total_' + y_axis_tag] = _y_axis\n if items_to_export is None:\n # export based on specified level : layer|element|isotope\n if all_layers:\n for _compound in _stack.keys():\n _y_axis = _stack_signal[_compound][y_axis_tag]\n df[_compound] = _y_axis\n\n if all_elements:\n for _compound in _stack.keys():\n for _element in _stack[_compound]['elements']:\n _y_axis = _stack_signal[_compound][_element][y_axis_tag]\n df[_compound + '/' + _element] = _y_axis\n\n if all_isotopes:\n for _compound in _stack.keys():\n for _element in _stack[_compound]['elements']:\n for _isotope in _stack[_compound][_element]['isotopes']['list']:\n _y_axis = _stack_signal[_compound][_element][_isotope][y_axis_tag]\n df[_compound + '/' + _element + '/' + _isotope] = _y_axis\n else:\n # export specified transmission or attenuation\n for _path_to_export in items_to_export:\n _path_to_export = list(_path_to_export)\n _live_path = _stack_signal\n _label = \"/\".join(_path_to_export)\n while _path_to_export:\n _item = _path_to_export.pop(0)\n _live_path = _live_path[_item]\n _y_axis = _live_path[y_axis_tag]\n df[_label] = _y_axis\n else:\n # export sigma\n if y_axis == 'sigma':\n y_axis_tag = 'sigma_b'\n else:\n y_axis_tag = 'sigma_b_raw'\n # y_axis_tag = 'sigma_b_raw'\n _stack_sigma = self.stack_sigma\n if items_to_export is None:\n for _compound in _stack.keys():\n for _element in _stack[_compound]['elements']:\n _y_axis = _stack_sigma[_compound][_element]['sigma_b'] # No 'sigma_b_raw' at this level\n df[_compound + '/' + _element + '/atoms_per_cm3'] = _stack[_compound][_element]['atoms_per_cm3']\n df[_compound + '/' + _element] = _y_axis\n if all_isotopes:\n for _isotope in _stack[_compound][_element]['isotopes']['list']:\n _y_axis = _stack_sigma[_compound][_element][_isotope][y_axis_tag]\n df[_compound + '/' + _element + '/' + _isotope] = _y_axis\n else:\n # export specified sigma\n for _path_to_export in items_to_export:\n if y_axis_tag[-3:] == 'raw':\n if len(_path_to_export) < 3:\n raise ValueError(\n \"Getting raw sigma of '{}' at layer or element level is not supported. \"\n \"If it is a single element layer, please follow \"\n \"['layer', 'element', 'isotope'] format.\".format(_path_to_export[0]))\n else:\n if len(_path_to_export) < 2:\n raise ValueError(\n \"Getting weighted sigma of '{}' at layer level is not supported. \"\n \"If it is a single element layer, please follow \"\n \"['layer', 'element'] format.\".format(_path_to_export[0]))\n _path_to_export = list(_path_to_export)\n _live_path = _stack_sigma\n _label = \"/\".join(_path_to_export)\n while _path_to_export:\n _item = _path_to_export.pop(0)\n _live_path = _live_path[_item]\n _y_axis = _live_path[y_axis_tag]\n df[_label] = _y_axis\n\n if len(df.columns) <= 1:\n raise ValueError(\"No y values have been selected to export!\")\n if output_type == 'csv':\n if filename is None:\n filename = 'data.csv'\n if '.csv' not in filename:\n filename += '.csv'\n df.to_csv(filename, index=False)\n print(\"Exporting to file ('./{}') completed.\".format(filename))\n elif output_type == 'clip':\n df.to_clipboard(excel=True, index=False)\n print('Exporting to clipboard completed.')\n else: # output_type == 'df'\n return df", "response": "export x and y values to clipboard or. csv file"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nbuilding a yaml file", "response": "def main():\n \"\"\"Builds a yaml file\"\"\"\n parser = argparse.ArgumentParser(description='Compose a yaml file.')\n parser.add_argument(\n 'root',\n type=argparse.FileType('r'),\n help='The root yaml file to compose.'\n )\n\n args = parser.parse_args()\n\n result = yaml.load(args.root, Loader=ComposeLoader)\n\n print(yaml.dump(result))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _parse(self, text, mode):\n if text is None or not text.strip():\n return None\n\n # first try to parse as an ISO8601 date, if it doesn't work we'll try other options\n if len(text) >= 16:\n try:\n parsed = iso8601.parse_date(text, default_timezone=None)\n if not parsed.tzinfo:\n parsed = self._timezone.localize(parsed)\n\n return parsed\n except iso8601.ParseError:\n pass\n\n # split the text into numerical and text tokens\n tokens = regex.findall(r'([0-9]+|[^\\W\\d]+)', text, flags=regex.MULTILINE | regex.UNICODE | regex.V0)\n\n # get the possibilities for each token\n token_possibilities = []\n for token in tokens:\n possibilities = self._get_token_possibilities(token, mode)\n if len(possibilities) > 0:\n token_possibilities.append(possibilities)\n\n # see what valid sequences we can make\n sequences = self._get_possible_sequences(mode, len(token_possibilities), self._date_style)\n\n for sequence in sequences:\n match = OrderedDict()\n\n for c in range(len(sequence)):\n component = sequence[c]\n value = token_possibilities[c].get(component, None)\n match[component] = value\n\n if value is None:\n break\n else:\n # try to make a valid result from this and return if successful\n obj = self._make_result(match, self._now, self._timezone)\n if obj is not None:\n return obj\n\n return None", "response": "Parses the text into a date datetime or time depending on what information is available."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_possible_sequences(cls, mode, length, date_style):\n sequences = []\n date_sequences = cls.DATE_SEQUENCES_DAY_FIRST if date_style == DateStyle.DAY_FIRST else cls.DATE_SEQUENCES_MONTH_FIRST\n\n if mode == Mode.DATE or mode == Mode.AUTO:\n for seq in date_sequences:\n if len(seq) == length:\n sequences.append(seq)\n\n elif mode == Mode.TIME:\n for seq in cls.TIME_SEQUENCES:\n if len(seq) == length:\n sequences.append(seq)\n\n if mode == Mode.DATETIME or mode == Mode.AUTO:\n for date_seq in date_sequences:\n for time_seq in cls.TIME_SEQUENCES:\n if len(date_seq) + len(time_seq) == length:\n sequences.append(date_seq + time_seq)\n\n return sequences", "response": "Gets the possible component sequences in the given mode."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _make_result(cls, values, now, timezone):\n date = None\n time = None\n\n if Component.MONTH in values:\n year = cls._year_from_2digits(values.get(Component.YEAR, now.year), now.year)\n month = values[Component.MONTH]\n day = values.get(Component.DAY, 1)\n try:\n date = datetime.date(year, month, day)\n except ValueError:\n return None # not a valid date\n\n if (Component.HOUR in values and Component.MINUTE in values) or Component.HOUR_AND_MINUTE in values:\n if Component.HOUR_AND_MINUTE in values:\n combined = values[Component.HOUR_AND_MINUTE]\n hour = combined // 100\n minute = combined - (hour * 100)\n second = 0\n nano = 0\n else:\n hour = values[Component.HOUR]\n minute = values[Component.MINUTE]\n second = values.get(Component.SECOND, 0)\n nano = values.get(Component.NANO, 0)\n\n if hour < 12 and values.get(Component.AM_PM) == cls.PM:\n hour += 12\n elif hour == 12 and values.get(Component.AM_PM) == cls.AM:\n hour -= 12\n\n try:\n time = datetime.time(hour, minute, second, microsecond=nano // 1000)\n except ValueError:\n return None # not a valid time\n\n if Component.OFFSET in values:\n timezone = pytz.FixedOffset(values[Component.OFFSET] // 60)\n\n if date is not None and time is not None:\n return timezone.localize(datetime.datetime.combine(date, time))\n elif date is not None:\n return date\n elif time is not None:\n return time\n else:\n return None", "response": "Makes a date datetime or time object from a map of component values."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _year_from_2digits(short_year, current_year):\n if short_year < 100:\n short_year += current_year - (current_year % 100)\n if abs(short_year - current_year) >= 50:\n if short_year < current_year:\n return short_year + 100\n else:\n return short_year - 100\n return short_year", "response": "Converts a relative 2 - digit year to an absolute 4 - digit year."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nprepares the black list and white lists for a given project and patchset.", "response": "def prepare_patchset(project, patchset, binaries, ips, urls):\n \"\"\" Create black/white lists and default / project waivers\n and iterates over patchset file \"\"\"\n # Get Various Lists / Project Waivers\n lists = get_lists.GetLists()\n\n # Get file name black list and project waivers\n file_audit_list, file_audit_project_list = lists.file_audit_list(project)\n\n # Get file content black list and project waivers\n flag_list, ignore_list = lists.file_content_list(project)\n\n # Get URL Ignore Lists\n url_ignore = lists.url_ignore(project)\n\n # Get URL Ignore Lists\n ip_ignore = lists.ip_ignore(project)\n\n # Get File Ignore Lists\n file_ignore = lists.file_ignore()\n\n # Get Directory Ignore Lists\n ignore_directories = lists.ignore_directories(project)\n\n if binaries or ips or urls:\n try:\n apikey = os.environ[\"VT_KEY\"]\n except KeyError:\n logger.error(\"Please set your virustotal.com API key as an environment variable\")\n sys.exit(1)\n try:\n vt_rate_type = config.get('config', 'vt_rate_type')\n except six.moves.configparser.NoSectionError:\n logger.error(\"A config section is required for vt_rate_type with a public | private option \")\n sys.exit(1)\n\n patten = re.compile(r'\\bpublic\\b|\\bprivate\\b')\n if not patten.match(vt_rate_type):\n logger.error(\"Unrecognized %s option for vt_rate_type\", vt_rate_type)\n sys.exit(1)\n else:\n apikey = \"\"\n\n # Open patch set to get file list\n try:\n fo = open(patchset, 'r')\n lines = fo.readlines()\n except IOError:\n logger.error('%s does not exist', patchset)\n sys.exit(1)\n\n for line in lines:\n patch_file = line.strip('\\n')\n # Perform binary and file / content checks\n scan_patch(project, patch_file, binaries, ips, urls, file_audit_list,\n file_audit_project_list, flag_list, ignore_list,\n file_ignore, ignore_directories, url_ignore, ip_ignore, apikey)\n\n # Process final result\n process_failure(project)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nscans patch file for changes and return a list of successive items", "response": "def scan_patch(project, patch_file, binaries, ips, urls, file_audit_list,\n file_audit_project_list, flag_list, ignore_list, file_ignore,\n ignore_directories, url_ignore, ip_ignore, apikey):\n\n \"\"\"\n Scan actions for each commited file in patch set\n \"\"\"\n global failure\n split_path = patch_file.split(project + '/', 1)[-1]\n\n if not any(x in split_path for x in ignore_directories):\n if is_binary(patch_file) and binaries:\n hashlist = get_lists.GetLists()\n binary_hash = hashlist.binary_hash(project, split_path)\n\n with open(patch_file, 'rb') as afile:\n hasher = hashlib.sha256()\n buf = afile.read()\n hasher.update(buf)\n sha256hash = hasher.hexdigest()\n\n if sha256hash in binary_hash:\n\n logger.info('Found matching file hash for: %s',\n patch_file)\n else:\n logger.info('sha256hash: %s', sha256hash)\n logger.error('Non Whitelisted Binary file: %s',\n patch_file)\n\n scan_binary(patch_file, project, sha256hash, apikey)\n\n failure = True\n\n with open(reports_dir + \"binaries-\" + project + \".log\", \"a\") \\\n as gate_report:\n gate_report.write('Non Whitelisted Binary file: {0}\\n'.\n format(patch_file))\n else:\n # Check file names / extensions\n if file_audit_list.search(patch_file) and not \\\n file_audit_project_list.search(patch_file):\n match = file_audit_list.search(patch_file)\n logger.error('Blacklisted file: %s', patch_file)\n logger.error('Matched String: %s', match.group())\n failure = True\n with open(reports_dir + \"file-names_\" + project + \".log\", \"a\") \\\n as gate_report:\n gate_report.write('Blacklisted file: {0}\\n'.\n format(patch_file))\n gate_report.write('Matched String: {0}'.\n format(match.group()))\n\n # Open file to check for blacklisted content\n if not is_binary(patch_file):\n try:\n fo = open(patch_file, 'r')\n lines = fo.readlines()\n file_exists = True\n except IOError:\n file_exists = False\n\n if file_exists and not patch_file.endswith(tuple(file_ignore)):\n for line in lines:\n # Find IP Addresses and send for report to Virus Total\n if ips:\n ipaddr = re.findall(r'(?:\\d{1,3}\\.)+(?:\\d{1,3})', line)\n if ipaddr:\n ipaddr = ipaddr[0]\n if re.search(ip_ignore, ipaddr):\n logger.info('%s is in IP ignore list.', ipaddr)\n else:\n try:\n ipaddress.ip_address(ipaddr).is_global\n scan_ipaddr(ipaddr, apikey)\n except:\n pass # Ok to pass here, as this captures the odd string which is not an IP Address\n\n # Check for URLs and send for report to Virus Total\n if urls:\n url = re.search(\"(?Phttps?://[^\\s]+)\", line) or re.search(\"(?Pwww[^\\s]+)\", line)\n if url:\n url = url.group(\"url\")\n if re.search(url_ignore, url):\n logger.info('%s is in URL ignore list.', url)\n else:\n scan_url(url, apikey)\n\n # Perform search within text files\n for key, value in flag_list.items():\n regex = value['regex']\n desc = value['desc']\n if re.search(regex, line) and not re.search(\n ignore_list, line):\n logger.error('File contains violation: %s', patch_file)\n logger.error('Flagged Content: %s', line.rstrip())\n logger.error('Rationale: %s', desc.rstrip())\n failure = True\n with open(reports_dir + \"contents_\" + project + \".log\", \"a\") as gate_report:\n gate_report.write('File contains violation: {0}\\n'.format(patch_file))\n gate_report.write('Flagged Content: {0}'.format(line))\n gate_report.write('Matched Regular Exp: {0}\\n'.format(regex))\n gate_report.write('Rationale: {0}\\n'.format(desc.rstrip()))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nscans binary files for a given project and sha256hash.", "response": "def scan_binary(patch_file, project, sha256hash, apikey):\n \"\"\"\n Sends Binary (sha256hash) to Virus Total API\n \"\"\"\n v_api = virus_total.VirusTotal()\n\n while True:\n binary_report = v_api.binary_report(sha256hash, apikey)\n response_code = binary_report['response_code']\n\n # report does not exist, need to scan\n if response_code == 0:\n logger.info('Performing new scan of %s.', patch_file)\n scan_file = v_api.scan_file(patch_file, apikey)\n logger.info('VirusTotal Response: %s', scan_file['verbose_msg'])\n logger.info('Report will be rendered at: %s', scan_file['permalink'])\n binary_report = v_api.binary_report(sha256hash, apikey)\n\n # Item is still queued\n if response_code == -2:\n logger.info('Report job still queued..')\n\n if response_code == 1:\n logger.info('Report found, job complete.')\n break\n\n positives = binary_report['positives']\n\n if positives == 0:\n negative_report(binary_report, sha256hash, project, patch_file)\n else:\n positive_report(binary_report, sha256hash, project, patch_file)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef negative_report(binary_report, sha256hash, project, patch_file):\n report_url = binary_report['permalink']\n scan_date = binary_report['scan_date']\n logger.info('File scan date for %s shows a clean status on: %s', patch_file, scan_date)\n logger.info('Full report avaliable here: %s', report_url)\n logger.info('The following sha256 hash can be used in your %s.yaml file to suppress this scan:', project)\n logger.info('%s', sha256hash)\n with open(reports_dir + \"binaries-\" + project + \".log\", \"a\") as gate_report:\n gate_report.write('Non Whitelisted Binary: {}\\n'.format(patch_file))\n gate_report.write('File scan date for {} shows a clean status on {}\\n'.format(patch_file, scan_date))\n gate_report.write('The following sha256 hash can be used in your {}.yaml file to suppress this scan:\\n'.format(project))\n gate_report.write('{}\\n'.format(sha256hash))", "response": "Write negative binary report to log file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef positive_report(binary_report, sha256hash, project, patch_file):\n failure = True\n report_url = binary_report['permalink']\n scan_date = binary_report['scan_date']\n logger.error(\"Virus Found!\")\n logger.info('File scan date for %s shows a infected status on: %s', patch_file, scan_date)\n logger.info('Full report avaliable here: %s', report_url)", "response": "Check if a Positive match is found and if so show a full report"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef scan_ipaddr(ipaddr, apikey):\n logger.info('Query VirusTotal API for Public IP Found: %s', ipaddr)\n v_api = virus_total.VirusTotal()\n scan_ip = v_api.send_ip(ipaddr, apikey)\n response_code = scan_ip['response_code']\n verbose_msg = scan_ip['verbose_msg']\n urls = scan_ip['detected_urls']\n\n if urls:\n failure = True\n logger.error('%s has been known to resolve to malicious urls', ipaddr)\n for url in urls:\n logger.error('%s on date: %s', url['url'], url['scan_date'])\n else:\n logger.info('%s has no record of resolving to malicious urls', ipaddr)", "response": "Scan VirusTotal API for a given IP Address."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nscanning a URL for VirusTotal reports.", "response": "def scan_url(url, apikey):\n \"\"\"\n If URL is found, scan it\n \"\"\"\n logger.info('Found what I believe is a URL: %s', url)\n v_api = virus_total.VirusTotal()\n while True:\n url_report = v_api.url_report(url, apikey)\n response_code = url_report['response_code']\n\n # report does not exist, need to scan\n if response_code == -2:\n logger.info('Report job still queued..')\n\n if response_code == 0:\n logger.info('No report for %s', url)\n break\n\n if response_code == 1:\n logger.info('Report found, job complete for %s.', url)\n break\n\n try:\n positives = url_report['positives']\n if positives > 0:\n for site, results in url_report['scans'].items():\n if results['detected']:\n detected = True\n failure = True\n logger.error(\"%s is recorded as a %s by %s\", url, results['result'], site)\n if detected:\n logger.error(\"Full report available here: %s\", url_report['permalink'])\n else:\n logger.info(\"%s is recorded as a clean\", url)\n except:\n pass"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef process_failure(project):\n if failure:\n lists = get_lists.GetLists()\n report_url = lists.report_url(project)\n if report_url:\n print(report_url)\n sys.exit(1)", "response": "Exit if a failure occurs."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncomputes average gradient norm of an image", "response": "def average_gradient(data, *kwargs):\n \"\"\" Compute average gradient norm of an image\n \"\"\"\n return np.average(np.array(np.gradient(data))**2)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef contrast_rms(data, *kwargs):\n av = np.average(data, *kwargs)\n mal = 1 / (data.shape[0] * data.shape[1])\n return np.sqrt(mal * np.sum(np.square(data - av)))", "response": "Compute the RMS contrast norm of an image"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncompute spectral contrast of image", "response": "def spectral(data, lambd, *kwargs):\n \"\"\" Compute spectral contrast of image\n\n Performs bandpass filtering in Fourier space according to optical\n limit of detection system, approximated by twice the wavelength.\n\n\n Parameters\n ----------\n data : 2d ndarray\n the image to compute the norm from\n lambd : float\n wavelength of the light in pixels\n\n \"\"\"\n # Set up fast fourier transform\n # if not data.dtype == np.dtype(np.complex):\n # data = np.array(data, dtype=np.complex)\n # fftplan = fftw3.Plan(data.copy(), None, nthreads = _ncores,\n # direction=\"forward\", flags=_fftwflags)\n # fftdata = np.zeros(data.shape, dtype=np.complex)\n # fftplan.guru_execute_dft(data, fftdata)\n # fftw.destroy_plan(fftplan)\n fftdata = np.fft.fftn(data)\n\n # Filter Fourier transform\n fftdata[0, 0] = 0\n kx = 2 * np.pi * np.fft.fftfreq(data.shape[0]).reshape(1, -1)\n ky = 2 * np.pi * np.fft.fftfreq(data.shape[1]).reshape(-1, 1)\n kmax = (2 * np.pi) / (2 * lambd)\n fftdata[np.where(kx**2 + ky**2 > kmax**2)] = 0\n\n spec = np.sum(np.log(1 + np.abs(fftdata))) / np.sqrt(np.prod(data.shape))\n\n return spec"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses file extension. - *.json: uncompressed, utf-8 encode json file - *.gz: compressed, utf-8 encode json file", "response": "def is_gzip_file(abspath):\n \"\"\"Parse file extension.\n\n - *.json: uncompressed, utf-8 encode json file\n - *.gz: compressed, utf-8 encode json file\n \"\"\"\n abspath = abspath.lower()\n _, ext = os.path.splitext(abspath)\n if ext in [\".gz\", \".zip\"]:\n is_gzip = True\n else:\n is_gzip = False\n return is_gzip"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nwrite string to text file.", "response": "def write(s, path, encoding=\"utf-8\"):\n \"\"\"Write string to text file.\n \"\"\"\n is_gzip = is_gzip_file(path)\n\n with open(path, \"wb\") as f:\n if is_gzip:\n f.write(zlib.compress(s.encode(encoding)))\n else:\n f.write(s.encode(encoding))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef read(path, encoding=\"utf-8\"):\n is_gzip = is_gzip_file(path)\n\n with open(path, \"rb\") as f:\n if is_gzip:\n return zlib.decompress(f.read()).decode(encoding)\n else:\n return f.read().decode(encoding)", "response": "Read string from text file."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef smartread(path):\n with open(path, \"rb\") as f:\n content = f.read()\n result = chardet.detect(content)\n return content.decode(result[\"encoding\"])", "response": "Read text from file automatically detect encoding. Chardet is required."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef to_utf8(path, output_path=None):\n if output_path is None:\n basename, ext = os.path.splitext(path)\n output_path = basename + \"-UTF8Encode\" + ext\n \n text = smartread(path)\n write(text, output_path)", "response": "Convert any text file to utf8 encoding."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nyield a generator of lines from file", "response": "def readlines(path, encoding=\"utf-8\", skiplines=None, nlines=None, strip='right'):\n \"\"\"skip n lines and fetch the next n lines.\n \n :param skiplines: default None, skip first n lines\n :param nlines: default None, yield next n lines\n :param strip: default None, available option 'left', 'right', 'both'\n \n **\u4e2d\u6587\u6587\u6863**\n \n \u8df3\u8fc7\u524d#skiplines\u884c, \u7136\u540e\u8bfb\u53d6#nlines\u884c\u3002\u53ef\u5bf9\u5b57\u7b26\u4e32\u8fdb\u884cstrip\u9884\u5904\u7406\u3002\n \"\"\"\n strip_method = str(strip).lower()\n if strip_method in _strip_method_mapping:\n strip_func = _strip_method_mapping[strip_method]\n else:\n raise ValueError(\"'strip' keyword has to be one of \"\n \"None, 'left', 'right', 'both'.\")\n \n with open(path, \"rb\") as file: \n if skiplines:\n for _ in range(skiplines):\n next(file)\n \n if nlines:\n for _ in range(nlines):\n yield strip_func(next(file).decode(encoding))\n else:\n for line in file:\n yield strip_func(line.decode(encoding))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreading a file into chunks", "response": "def readchunks(path, encoding=\"utf-8\", skiplines=None, chunksize=None, strip='right'):\n \"\"\"skip n lines and fetch the next n lines as a chunk, and repeat fetching.\n \n :param skiplines: default None, skip first n lines\n :param chunksize: default None (size-1 chunk), lines chunk size\n :param strip: default None, avaliable option 'left', 'right', 'both'\n \n **\u4e2d\u6587\u6587\u6863**\n \n \u8df3\u8fc7\u524d#skiplines\u884c, \u6bcf\u6b21\u8bfb\u53d6#chunksize\u884cyield\u3002\u53ef\u5bf9\u5b57\u7b26\u4e32\u8fdb\u884cstrip\u9884\u5904\u7406\u3002\n \"\"\"\n strip_method = str(strip).lower()\n if strip_method in _strip_method_mapping:\n strip_func = _strip_method_mapping[strip_method]\n else:\n raise ValueError(\"'strip' keyword has to be one of \"\n \"None, 'left', 'right', 'both'.\")\n \n with open(path, \"rb\") as file:\n if skiplines:\n for _ in range(skiplines):\n next(file)\n \n if chunksize is None:\n chunksize = 1\n elif not isinstance(chunksize, int_type): \n raise ValueError(\"'chunksize' has to be None or an integer.\")\n \n chunk = list()\n while 1:\n for _ in range(chunksize):\n chunk.append(strip_func(next(file).decode(encoding)))\n if len(chunk) < chunksize:\n break\n yield chunk\n chunk = list()\n yield chunk"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _coerce_to_supported_type(cls, value):\n if value is None:\n return \"\" # empty string rather than none\n elif isinstance(value, dict):\n if '*' in value:\n return value['*']\n elif '__default__' in value:\n return value['__default__']\n else:\n return json.dumps(value, separators=(',', ':')) # return serialized JSON if no default\n elif isinstance(value, bool):\n return value\n elif isinstance(value, float) or isinstance(value, int):\n return Decimal(value)\n else:\n return value", "response": "Coerce the value to a supported data type."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nevaluate a template string and returns a tuple of the evaluated template string and a list of evaluation errors", "response": "def evaluate_template(self, template, context, url_encode=False, strategy=EvaluationStrategy.COMPLETE):\n \"\"\"\n Evaluates a template string, e.g. \"Hello @contact.name you have @(contact.reports * 2) reports\"\n :param template: the template string\n :param context: the evaluation context\n :param url_encode: whether or not values should be URL encoded\n :return: a tuple of the evaluated template and a list of evaluation errors\n \"\"\"\n input_chars = list(template)\n output_chars = []\n errors = []\n state = State.BODY\n current_expression_chars = []\n current_expression_terminated = False\n parentheses_level = 0\n\n # determines whether the given character is a word character, i.e. \\w in a regex\n is_word_char = lambda c: c and (c.isalnum() or c == '_')\n\n for pos, ch in enumerate(input_chars):\n # in order to determine if the b in a.b terminates an identifier, we have to peek two characters ahead as it\n # could be a.b. (b terminates) or a.b.c (b doesn't terminate)\n next_ch = input_chars[pos + 1] if (pos < (len(input_chars) - 1)) else None\n next_next_ch = input_chars[pos + 2] if (pos < (len(input_chars) - 2)) else None\n\n if state == State.BODY:\n if ch == self._expression_prefix and (is_word_char(next_ch) or next_ch == '('):\n state = State.PREFIX\n current_expression_chars = [ch]\n elif ch == self._expression_prefix and next_ch == self._expression_prefix:\n state = State.ESCAPED_PREFIX\n else:\n output_chars.append(ch)\n\n elif state == State.PREFIX:\n if is_word_char(ch):\n # we're parsing an expression like @XXX\n state = State.IDENTIFIER\n elif ch == '(':\n # we're parsing an expression like @(1 + 2)\n state = State.BALANCED\n parentheses_level += 1\n\n current_expression_chars.append(ch)\n\n elif state == State.IDENTIFIER:\n current_expression_chars.append(ch)\n\n elif state == State.BALANCED:\n if ch == '(':\n parentheses_level += 1\n elif ch == ')':\n parentheses_level -= 1\n elif ch == '\"':\n state = State.STRING_LITERAL\n\n current_expression_chars.append(ch)\n\n # expression terminates if parentheses balance\n if parentheses_level == 0:\n current_expression_terminated = True\n\n elif state == State.STRING_LITERAL:\n if ch == '\"':\n state = State.BALANCED\n current_expression_chars.append(ch)\n\n elif state == State.ESCAPED_PREFIX:\n state = State.BODY\n output_chars.append(ch)\n\n # identifier can terminate expression in 3 ways:\n # 1. next char is null (i.e. end of the input)\n # 2. next char is not a word character or period\n # 3. next char is a period, but it's not followed by a word character\n if state == State.IDENTIFIER:\n if not next_ch or (not is_word_char(next_ch) and next_ch != '.') or (next_ch == '.' and not is_word_char(next_next_ch)):\n current_expression_terminated = True\n\n if current_expression_terminated:\n expression = ''.join(current_expression_chars)\n output_chars.append(self._resolve_expression_block(expression, context, url_encode, strategy, errors))\n current_expression_chars = []\n current_expression_terminated = False\n state = State.BODY\n\n # if last expression didn't terminate - add to output as is\n if not current_expression_terminated and current_expression_chars:\n output_chars.append(''.join(current_expression_chars))\n\n output = ''.join(output_chars) # joining is fastest way to build strings in Python\n return output, errors"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _resolve_expression_block(self, expression, context, url_encode, strategy, errors):\n try:\n body = expression[1:] # strip prefix\n\n # if expression doesn't start with ( then check it's an allowed top level context reference\n if not body.startswith('('):\n top_level = body.split('.')[0].lower()\n if top_level not in self._allowed_top_levels:\n return expression\n\n evaluated = self.evaluate_expression(body, context, strategy)\n\n # convert result to string\n result = conversions.to_string(evaluated, context)\n\n return urlquote(result) if url_encode else result\n except EvaluationError as e:\n logger.debug(\"EvaluationError: %s\" % str(e))\n\n # if we can't evaluate expression, include it as is in the output\n errors.append(str(e))\n return expression", "response": "Resolves an expression block found in the template e. g. @(\\. \\ d + \\. \\ d + \\. \\ d + \\. \\ d + \\. \\ d + \\. \\ d + \\. \\ d + \\. \\ d + \\. \\ d + \\. \\ d + \\. \\ d + \\. \\ d + \\. \\ d + \\. \\ d + \\. \\ d + \\. \\ d + \\. \\ d + \\."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nevaluate a single expression and returns the value of the expression.", "response": "def evaluate_expression(self, expression, context, strategy=EvaluationStrategy.COMPLETE):\n \"\"\"\n Evaluates a single expression, e.g. \"contact.reports * 2\"\n :param expression: the expression string\n :param context: the evaluation context\n :param strategy: the evaluation strategy\n :return: the evaluated expression value\n \"\"\"\n from .gen.ExcellentLexer import ExcellentLexer\n from .gen.ExcellentParser import ExcellentParser\n\n stream = InputStream(expression)\n lexer = ExcellentLexer(stream)\n tokens = CommonTokenStream(lexer)\n\n parser = ExcellentParser(tokens)\n parser._errHandler = BailErrorStrategy()\n\n try:\n tree = parser.parse()\n\n if logger.isEnabledFor(logging.DEBUG):\n logger.debug(\"Expression '%s' parsed as %s\" % (expression, tree.toStringTree()))\n except ParseCancellationException as ex:\n message = None\n if ex.args and isinstance(ex.args[0], NoViableAltException):\n token = ex.args[0].offendingToken\n if token is not None and token.type != ExcellentParser.EOF:\n message = \"Expression error at: %s\" % token.text\n\n if message is None:\n message = \"Expression is invalid\"\n\n raise EvaluationError(message, ex)\n\n if strategy == EvaluationStrategy.RESOLVE_AVAILABLE:\n resolved = self._resolve_available(tokens, context)\n if resolved is not None:\n return resolved\n\n visitor = ExcellentVisitor(self._function_manager, context)\n return visitor.visit(tree)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _resolve_available(self, tokens, context):\n from .gen.ExcellentParser import ExcellentParser\n\n has_missing = False\n output_components = []\n\n for t in range(len(tokens.tokens) - 1): # we can ignore the final EOF token\n token = tokens.get(t)\n next_token = tokens.get(t + 1)\n\n # if token is a NAME not followed by ( then it's a context reference\n if token.type == ExcellentParser.NAME and next_token.type != ExcellentParser.LPAREN:\n try:\n output_components.append(context.resolve_variable(token.text))\n except EvaluationError:\n has_missing = True\n output_components.append(token)\n else:\n output_components.append(token)\n\n # if we don't have missing context references, perform evaluation as normal\n if not has_missing:\n return None\n\n # re-combine the tokens and context values back into an expression\n output = [self._expression_prefix]\n\n for output_component in output_components:\n if isinstance(output_component, Token):\n comp_val = output_component.text\n else:\n comp_val = conversions.to_repr(output_component, context)\n output.append(comp_val)\n\n return ''.join(output)", "response": "Resolves the available context references and returns a partially evaluated expression."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn an expression that is the result of evaluating the function.", "response": "def visitFunctionCall(self, ctx):\n \"\"\"\n expression : fnname LPAREN parameters? RPAREN\n \"\"\"\n func_name = ctx.fnname().getText()\n\n if ctx.parameters() is not None:\n parameters = self.visit(ctx.parameters())\n else:\n parameters = []\n\n return self._functions.invoke_function(self._eval_context, func_name, parameters)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a negative integer representation of the expression.", "response": "def visitNegation(self, ctx):\n \"\"\"\n expression: MINUS expression\n \"\"\"\n return -conversions.to_decimal(self.visit(ctx.expression()), self._eval_context)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the value of the expression EXPONENT expression as a decimal number.", "response": "def visitExponentExpression(self, ctx):\n \"\"\"\n expression: expression EXPONENT expression\n \"\"\"\n arg1 = conversions.to_decimal(self.visit(ctx.expression(0)), self._eval_context)\n arg2 = conversions.to_decimal(self.visit(ctx.expression(1)), self._eval_context)\n return conversions.to_decimal(decimal_pow(arg1, arg2), ctx)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef visitMultiplicationOrDivisionExpression(self, ctx):\n is_mul = ctx.TIMES() is not None\n\n arg1 = conversions.to_decimal(self.visit(ctx.expression(0)), self._eval_context)\n arg2 = conversions.to_decimal(self.visit(ctx.expression(1)), self._eval_context)\n\n if not is_mul and arg2 == Decimal(0):\n raise EvaluationError(\"Division by zero\")\n\n return arg1 * arg2 if is_mul else arg1 / arg2", "response": "Multiplication or division expression"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the value of the addition or subtraction expression as a number.", "response": "def visitAdditionOrSubtractionExpression(self, ctx):\n \"\"\"\n expression: expression (PLUS | MINUS) expression\n \"\"\"\n is_add = ctx.PLUS() is not None\n arg1 = self.visit(ctx.expression(0))\n arg2 = self.visit(ctx.expression(1))\n\n # first try as decimals\n try:\n _arg1 = conversions.to_decimal(arg1, self._eval_context)\n _arg2 = conversions.to_decimal(arg2, self._eval_context)\n return _arg1 + _arg2 if is_add else _arg1 - _arg2\n except EvaluationError:\n pass\n\n # then as date + something\n try:\n _arg1 = conversions.to_date_or_datetime(arg1, self._eval_context)\n\n if isinstance(arg2, datetime.time):\n # upgrade our date to datetime\n _arg1 = conversions.to_datetime(_arg1, self._eval_context)\n\n # convert time value to a duration\n _arg2 = datetime.timedelta(hours=arg2.hour, minutes=arg2.minute, seconds=arg2.second, microseconds=arg2.microsecond)\n else:\n _arg2 = datetime.timedelta(days=conversions.to_integer(arg2, self._eval_context))\n\n return _arg1 + _arg2 if is_add else _arg1 - _arg2\n\n except EvaluationError as ex:\n raise EvaluationError(\"Expression could not be evaluated as decimal or date arithmetic\", ex)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a boolean indicating if the value of a comparison expression is greater than or equal to a value of a key.", "response": "def visitComparisonExpression(self, ctx):\n \"\"\"\n expression: expression (LTE | LT | GTE | GT) expression\n \"\"\"\n arg1, arg2 = conversions.to_same(self.visit(ctx.expression(0)), self.visit(ctx.expression(1)), self._eval_context)\n\n if isinstance(arg1, str):\n # string comparison is case-insensitive\n compared = (arg1.lower() > arg2.lower()) - (arg1.lower() < arg2.lower())\n else:\n compared = (arg1 > arg2) - (arg1 < arg2)\n\n if ctx.LTE() is not None:\n return compared <= 0\n elif ctx.LT() is not None:\n return compared < 0\n elif ctx.GTE() is not None:\n return compared >= 0\n else: # GT\n return compared > 0"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef visitConcatenation(self, ctx):\n arg1 = conversions.to_string(self.visit(ctx.expression(0)), self._eval_context)\n arg2 = conversions.to_string(self.visit(ctx.expression(1)), self._eval_context)\n return arg1 + arg2", "response": "return an expression AMPERSAND expression"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the value of the variable referenced by the given context identifier.", "response": "def visitContextReference(self, ctx):\n \"\"\"\n expression: NAME\n \"\"\"\n identifier = ctx.NAME().getText()\n return self._eval_context.resolve_variable(identifier)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef load_cache(self):\n with open(self.cache_path, \"rb\") as f:\n print(\"Loading cached Zotero data...\")\n cache = pickle.load(f)\n self._references = cache[self.CACHE_REFERENCE_LIST]\n self.reference_types = cache[self.CACHE_REFERENCE_TYPES]\n self.reference_templates = cache[self.CACHE_REFERENCE_TEMPLATES]\n print(\"Cached Zotero data loaded.\")", "response": "Load the cached Zotero data."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nload the distant Zotero data.", "response": "def load_distant(self):\n \"\"\"Load the distant Zotero data.\"\"\"\n print(\"Loading distant Zotero data...\")\n self._references = self.get_references()\n self.reference_types = self.get_reference_types()\n self.reference_templates = self.get_reference_templates(self.reference_types)\n print(\"Distant Zotero data loaded.\")\n self.cache()"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef cache(self):\n with open(self.cache_path, \"wb\") as f:\n cache = {self.CACHE_REFERENCE_LIST: self._references,\n self.CACHE_REFERENCE_TYPES: self.reference_types,\n self.CACHE_REFERENCE_TEMPLATES: self.reference_templates}\n pickle.dump(cache, f)", "response": "Cache the Zotero data."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nvalidate and create the reference in Zotero and return the created item.", "response": "def create_distant_reference(self, ref_data):\n \"\"\"Validate and create the reference in Zotero and return the created item.\"\"\"\n self.validate_reference_data(ref_data)\n creation_status = self._zotero_lib.create_items([ref_data])\n try:\n created_item = creation_status[\"successful\"][\"0\"]\n return created_item\n except KeyError as e:\n print(creation_status)\n raise CreateZoteroItemError from e"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreplace the reference in the reference list with the reference ref.", "response": "def update_local_reference(self, index, ref):\n \"\"\"Replace the reference in the reference list and cache it.\"\"\"\n self._references[index] = ref\n self.cache()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nvalidate and update the distant reference in Zotero.", "response": "def update_distant_reference(self, ref):\n \"\"\"Validate and update the reference in Zotero.\n\n Existing fields not present will be left unmodified.\n \"\"\"\n self.validate_reference_data(ref[\"data\"])\n self._zotero_lib.update_item(ref)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nvalidates the reference data.", "response": "def validate_reference_data(self, ref_data):\n \"\"\"Validate the reference data.\n\n Zotero.check_items() caches data after the first API call.\n \"\"\"\n try:\n self._zotero_lib.check_items([ref_data])\n except InvalidItemFields as e:\n raise InvalidZoteroItemError from e"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_reference_types(self):\n item_types = self._zotero_lib.item_types()\n return sorted([x[\"itemType\"] for x in item_types])", "response": "Return the reference types."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the reference templates for the types as an ordered dictionary.", "response": "def get_reference_templates(self, ref_types):\n \"\"\"Return the reference templates for the types as an ordered dictionary.\"\"\"\n return OrderedDict([(x, self.get_reference_template(x)) for x in ref_types])"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the reference template for the type as an ordered dictionary.", "response": "def get_reference_template(self, ref_type):\n \"\"\"Return the reference template for the type as an ordered dictionary.\n\n Zotero.item_template() caches data after the first API call.\n \"\"\"\n template = self._zotero_lib.item_template(ref_type)\n return OrderedDict(sorted(template.items(), key=lambda x: x[0]))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the value of the field in extra otherwise empty.", "response": "def reference_extra_field(self, field, index):\n \"\"\"Return the value of the field in 'extra', otherwise ''.\"\"\"\n ref_data = self.reference_data(index)\n extra_fields = ref_data[\"extra\"].split(\"\\n\")\n field_id = field + \":\"\n matched = next((x for x in extra_fields if x.startswith(field_id)), None)\n if matched:\n return matched.replace(field_id, \"\", 1).strip()\n else:\n return \"\""} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the reference ID for the specified index.", "response": "def reference_id(self, index):\n \"\"\"Return the reference ID (locally defined).\"\"\"\n # TODO Include ISBN and ISSN?\n doi = self.reference_doi(index)\n if doi:\n return doi\n else:\n pmid = self.reference_pmid(index)\n if pmid:\n return \"PMID_\" + pmid\n else:\n unpublished_id = self.reference_unpublished_id(index)\n if unpublished_id:\n return \"UNPUBLISHED_\" + unpublished_id\n return \"\""} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the reference DOI.", "response": "def reference_doi(self, index):\n \"\"\"Return the reference DOI.\"\"\"\n return self.reference_data(index).get(\"DOI\", self.reference_extra_field(\"DOI\", index))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn as a list the surnames of the reference creators ( locally defined.", "response": "def reference_creator_surnames(self, index):\n \"\"\"Return as a list the surnames of the reference creators (locally defined).\"\"\"\n # TODO Not true, ex: ISBN 978-1-4398-3778-8. Return all creator types?\n # Academic books published as a collection of chapters contributed by\n # different authors have editors but not authors at the level of the\n # book (as opposed to the level of a chapter).\n creators = self.reference_data(index)[\"creators\"]\n creator_types = [x[\"creatorType\"] for x in creators]\n # 'name' (not split) might be used instead of 'firstName' and 'lastName'.\n try:\n if \"author\" in creator_types:\n return [x[\"lastName\"] for x in creators if x[\"creatorType\"] == \"author\"]\n else:\n return [x[\"lastName\"] for x in creators]\n except KeyError:\n return []"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the reference publication year.", "response": "def reference_year(self, index):\n \"\"\"Return the reference publication year.\"\"\"\n # TODO Use meta:parsedDate field instead?\n ref_date = self.reference_date(index)\n try:\n # NB: datetime.year returns an int.\n return parse(ref_date).year\n except ValueError:\n matched = re.search(r\"\\d{4}\", ref_date)\n if matched:\n return int(matched.group())\n else:\n return \"\""} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef reference_journal(self, index):\n # TODO Change the column name 'Journal' to an other?\n ref_type = self.reference_type(index)\n if ref_type == \"journalArticle\":\n return self.reference_data(index)[\"publicationTitle\"]\n else:\n return \"({})\".format(ref_type)", "response": "Return the reference journal name."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef reference_index(self, ref_id):\n try:\n indexes = range(self.reference_count())\n return next(i for i in indexes if self.reference_id(i) == ref_id)\n except StopIteration as e:\n raise ReferenceNotFoundError(\"ID: \" + ref_id) from e", "response": "Return the first reference with this ID."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning for citation the creator surnames and the publication year.", "response": "def reference_creators_citation(self, ref_id):\n \"\"\"Return for citation the creator surnames (locally defined) and the publication year.\"\"\"\n # FIXME Delayed refactoring. Use an index instead of an ID.\n index = self.reference_index(ref_id)\n creators = self.reference_creator_surnames(index)\n creator_count = len(creators)\n if creator_count == 0:\n return \"\"\n year = self.reference_year(index)\n if creator_count == 1:\n return \"{} ({})\".format(creators[0], year)\n elif creator_count == 2:\n return \"{} and {} ({})\".format(creators[0], creators[1], year)\n else:\n return \"{} et al. ({})\".format(creators[0], year)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncomputes the similarity between two PDFs.", "response": "def computePDFSimilarity(paperId, userPDF):\n if not isPDFInDb(paperId):\n return None\n\n userPDF.save(\"temp.pdf\")\n # check_call is blocking\n check_call(['pdftotext', '-enc', 'UTF-8', \"temp.pdf\", \"temp.txt\"])\n os.remove(\"temp.pdf\")\n \n a = open(\"temp.txt\", 'r').read()\n b = open(join(dbPath, paperId) + \".txt\", 'r').read()\n\n import nltk, string\n from sklearn.feature_extraction.text import TfidfVectorizer\n\n stemmer = nltk.stem.porter.PorterStemmer()\n remove_punctuation_map = dict((ord(char), None) for char in string.punctuation)\n\n def stem_tokens(tokens):\n return [stemmer.stem(item) for item in tokens]\n\n '''remove punctuation, lowercase, stem'''\n def normalize(text):\n return stem_tokens(nltk.word_tokenize(text.lower().translate(remove_punctuation_map)))\n\n vectorizer = TfidfVectorizer(tokenizer=normalize, stop_words='english')\n\n def cosine_sim(text1, text2):\n tfidf = vectorizer.fit_transform([text1, text2])\n return ((tfidf * tfidf.T).A)[0,1]\n\n similarity = cosine_sim(a, b)\n\n os.remove(\"temp.txt\")\n\n return similarity"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef getChildren(root_id, maxDepth=100, relationshipType=\"subClassOf\", \n alwaysFetch=False):\n \"\"\"\n Accessing web-based ontology service is too long, so we cache the \n information in a pickle file and query the services only if the info\n has not already been cached. \n \"\"\"\n childrenDic = {}\n fileName = os.path.join(os.path.dirname(__file__), \"children.bin\") \n\n #### CHECK FOR CASE OF BBP TAGS\n if root_id[:4] == \"BBP_\":\n\n if not alwaysFetch:\n try:\n with open(fileName, \"rb\") as childrenFile:\n childrenDic = pickle.load(childrenFile)\n \n if root_id in childrenDic:\n return childrenDic[root_id] \n except:\n pass \n\n childrenDic[root_id] = getBBPChildren(root_id) \n \n if not alwaysFetch:\n try:\n with open(fileName, \"wb\") as childrenFile:\n pickle.dump(childrenDic, childrenFile)\n except:\n pass \n \n return childrenDic[root_id]\n\n ## TODO: should also check for BBP children of online onto terms\n\n if root_id in nlx2ks:\n root_id = nlx2ks[root_id]\n\n if not alwaysFetch:\n try:\n with open(fileName, \"rb\") as childrenFile:\n childrenDic = pickle.load(childrenFile)\n \n if root_id in childrenDic:\n return childrenDic[root_id] \n except:\n pass\n\n direction=\"INCOMING\"\n #neighbors = graph.getNeighbors(root_id, depth=maxDepth, \n # relationshipType=relationshipType, \n # direction=direction)\n \n baseKS = \"http://matrix.neuinfo.org:9000\"\n response = requests.get(baseKS + \"/scigraph/graph/neighbors/\" + \n root_id + \"?direction=\" + direction + \n \"&depth=\" + str(maxDepth) + \n \"&project=%2A&blankNodes=false&relationshipType=\" \n + relationshipType)\n\n if not response.ok:\n return {}\n \n neighbors = response.json()\n\n if neighbors is None:\n return {}\n\n nodes = neighbors[\"nodes\"]\n #for node in np.array(nodes):\n # node[\"lbl\"] = node[\"lbl\"].encode('utf-8').decode('utf-8')\n \n #try:\n # assert(np.all([not node[\"lbl\"] is None for node in np.array(nodes)]))\n #except AssertionError: \n # for node in np.array(nodes):\n # if node[\"lbl\"] is None:\n # print(node[\"id\"])\n # raise \n \n # TODO: replace by the commented line below. This patch is only to \n # accomodate for a current issue with the knowledge-space endpoint. \n #childrenDic[root_id] = OntoDic({node[\"id\"]:node[\"lbl\"] for node in np.array(nodes)}) \n childrenDic[root_id] = OntoDic({node[\"id\"]:node[\"lbl\"] for node in np.array(nodes) if not node[\"lbl\"] is None})\n \n if not alwaysFetch: \n try:\n with open(fileName, \"wb\") as childrenFile:\n pickle.dump(childrenDic, childrenFile)\n except:\n pass \n\n return childrenDic[root_id]", "response": "Get the list of children of a given root_id."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef install_plugins(plugins, app, install_type, config):\n try:\n disable_plugins = config.disable_plugins\n if not disable_plugins:\n disable_plugins = []\n except AttributeError:\n disable_plugins = []\n for plugin_name in plugins:\n plugin_group = f'backendai_{plugin_name}_v10'\n registry = PluginRegistry(plugin_name)\n for entrypoint in pkg_resources.iter_entry_points(plugin_group):\n if entrypoint.name in disable_plugins:\n continue\n log.info('Installing plugin: {}.{}', plugin_group, entrypoint.name)\n plugin_module = entrypoint.load()\n plugin = getattr(plugin_module, 'get_plugin')(config)\n registry.register(plugin)\n if install_type == 'attr':\n setattr(app, plugin_name, registry)\n elif install_type == 'dict':\n assert isinstance(app, typing.MutableMapping), \\\n (f\"app must be an instance of MutableMapping \"\n f\"for 'dict' install_type.\")\n app[plugin_name] = registry\n else:\n raise ValueError(f'Invalid install type: {install_type}')", "response": "Automatically installs plugins into the app."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nadding a new element to the set.", "response": "def addElement(self,etype='hex8',corners=[-1.0,-1.0,-1.0,1.,-1.0,-1.0,1.0,1.0,-1.0,-1.0,1.0,-1.0,-1.0,-1.0,1.0,1.0,-1.0,1.0,1.0,1.0,1.0,-1.0,1.0,1.0],name='new_elem'):\n '''\n corners - list of nodal coordinates properly ordered for element type (counter clockwise)\n '''\n lastelm = self.elements[-1][1]\n lastnode = self.nodes[-1][0]\n elm = [etype,lastelm+1]\n for i in range(old_div(len(corners),3)):\n elm.append(lastnode+1+i)\n \n self.elements.append(elm)\n self.elsets['e'+name] = {}\n self.elsets['e'+name][int(elm[1])] = True\n \n cnt = 1\n self.nsets['n'+name] = []\n for i in range(0,len(corners),3):\n self.nodes.append([lastnode+cnt, corners[i], corners[i+1], corners[i+2]])\n self.nsets['n'+name].append(lastnode+cnt)\n cnt += 1\n \n # if this is a quad4 or tri3 element make a surface set\n if etype == 'quad4' or etype == 'tri3':\n self.fsets['f'+name] = [[etype, MeshDef.facetID, lastnode+1, lastnode+2, lastnode+3, lastnode+4]]\n MeshDef.facetID += 1"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef getEdges(self, type, entail='true', limit=100, skip=0, callback=None, output='application/json'):\n\n kwargs = {'type':type, 'entail':entail, 'limit':limit, 'skip':skip, 'callback':callback}\n kwargs = {k:dumps(v) if type(v) is dict else v for k, v in kwargs.items()}\n param_rest = self._make_rest('type', **kwargs)\n url = self._basePath + ('/graph/edges/{type}').format(**kwargs)\n requests_params = {k:v for k, v in kwargs.items() if k != 'type'}\n return self._get('GET', url, requests_params, output)", "response": "Get the nodes connected by an edge type from the graph."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget neighbors from a CURIE.", "response": "def getNeighbors(self, id, depth=1, blankNodes='false', relationshipType=None, direction='BOTH', project='*', callback=None, output='application/json'):\n \"\"\" Get neighbors from: /graph/neighbors/{id}\n Arguments:\n id: This ID should be either a CURIE or an IRI\n depth: How far to traverse neighbors\n blankNodes: Traverse blank nodes\n relationshipType: Which relationship to traverse\n direction: Which direction to traverse: INCOMING, OUTGOING, BOTH (default). Only used if relationshipType is specified.\n project: Which properties to project. Defaults to '*'.\n callback: Name of the JSONP callback ('fn' by default). Supplying this parameter or\n requesting a javascript media type will cause a JSONP response to be\n rendered.\n outputs:\n application/json\n application/graphson\n application/xml\n application/graphml+xml\n application/xgmml\n text/gml\n text/csv\n text/tab-separated-values\n image/jpeg\n image/png\n \"\"\"\n\n kwargs = {'id':id, 'depth':depth, 'blankNodes':blankNodes, 'relationshipType':relationshipType, 'direction':direction, 'project':project, 'callback':callback}\n kwargs = {k:dumps(v) if type(v) is dict else v for k, v in kwargs.items()}\n param_rest = self._make_rest('id', **kwargs)\n url = self._basePath + ('/graph/neighbors/{id}').format(**kwargs)\n requests_params = {k:v for k, v in kwargs.items() if k != 'id'}\n return self._get('GET', url, requests_params, output)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nclasses Enrichment Service from: /analyzer/enrichment Arguments: sample: A list of CURIEs for nodes whose attributes are to be tested for enrichment. For example, a list of genes. ontologyClass: CURIE for parent ontology class for the attribute to be tested. For example, GO biological process path: A path expression that connects sample nodes to attribute class nodes callback: Name of the JSONP callback ('fn' by default). Supplying this parameter or requesting a javascript media type will cause a JSONP response to be rendered. outputs: application/json", "response": "def enrich(self, sample, ontologyClass, path, callback=None, output='application/json'):\n \"\"\" Class Enrichment Service from: /analyzer/enrichment\n Arguments:\n sample: A list of CURIEs for nodes whose attributes are to be tested for enrichment. For example, a list of genes.\n ontologyClass: CURIE for parent ontology class for the attribute to be tested. For example, GO biological process\n path: A path expression that connects sample nodes to attribute class nodes\n callback: Name of the JSONP callback ('fn' by default). Supplying this parameter or\n requesting a javascript media type will cause a JSONP response to be\n rendered.\n outputs:\n application/json\n \"\"\"\n\n kwargs = {'sample':sample, 'ontologyClass':ontologyClass, 'path':path, 'callback':callback}\n kwargs = {k:dumps(v) if type(v) is dict else v for k, v in kwargs.items()}\n param_rest = self._make_rest('path', **kwargs)\n url = self._basePath + ('/analyzer/enrichment').format(**kwargs)\n requests_params = {k:v for k, v in kwargs.items() if k != 'path'}\n return self._get('GET', url, requests_params, output)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nannotates text from the annotations API.", "response": "def annotatePost(self, content, includeCat=None, excludeCat=None, minLength=4, longestOnly='false', includeAbbrev='false', includeAcronym='false', includeNumbers='false', ignoreTag=None, stylesheet=None, scripts=None, targetId=None, targetClass=None):\n \"\"\" Annotate text from: /annotations\n Arguments:\n content: The content to annotate\n includeCat: A set of categories to include\n excludeCat: A set of categories to exclude\n minLength: The minimum number of characters in annotated entities\n longestOnly: Should only the longest entity be returned for an overlapping group\n includeAbbrev: Should abbreviations be included\n includeAcronym: Should acronyms be included\n includeNumbers: Should numbers be included\n ignoreTag: HTML tags that should not be annotated\n stylesheet: CSS stylesheets to add to the HEAD\n scripts: JavaScripts that should to add to the HEAD\n targetId: A set of element IDs to annotate\n targetClass: A set of CSS class names to annotate\n \"\"\"\n\n kwargs = {'content':content, 'includeCat':includeCat, 'excludeCat':excludeCat, 'minLength':minLength, 'longestOnly':longestOnly, 'includeAbbrev':includeAbbrev, 'includeAcronym':includeAcronym, 'includeNumbers':includeNumbers, 'ignoreTag':ignoreTag, 'stylesheet':stylesheet, 'scripts':scripts, 'targetId':targetId, 'targetClass':targetClass}\n kwargs = {k:dumps(v) if type(v) is dict else v for k, v in kwargs.items()}\n param_rest = self._make_rest('content', **kwargs)\n url = self._basePath + ('/annotations').format(**kwargs)\n requests_params = {k:v for k, v in kwargs.items() if k != 'content'}\n return self._get('POST', url, requests_params)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef getPos(self, text):\n\n kwargs = {'text':text}\n kwargs = {k:dumps(v) if type(v) is dict else v for k, v in kwargs.items()}\n param_rest = self._make_rest('text', **kwargs)\n url = self._basePath + ('/lexical/pos').format(**kwargs)\n requests_params = {k:v for k, v in kwargs.items() if k != 'text'}\n return self._get('GET', url, requests_params)", "response": "Get the POS of the text in the current language."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef findByPrefix(self, term, limit=20, searchSynonyms='true', searchAbbreviations='false', searchAcronyms='false', includeDeprecated='false', category=None, prefix=None):\n\n kwargs = {'term':term, 'limit':limit, 'searchSynonyms':searchSynonyms, 'searchAbbreviations':searchAbbreviations, 'searchAcronyms':searchAcronyms, 'includeDeprecated':includeDeprecated, 'category':category, 'prefix':prefix}\n kwargs = {k:dumps(v) if type(v) is dict else v for k, v in kwargs.items()}\n param_rest = self._make_rest('term', **kwargs)\n url = self._basePath + ('/vocabulary/autocomplete/{term}').format(**kwargs)\n requests_params = {k:v for k, v in kwargs.items() if k != 'term'}\n return self._get('GET', url, requests_params)", "response": "Find a concept by its prefix from the Vocabulary Autocomplete API."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nloads zip file and return complex field", "response": "def load_cell(fname=\"HL60_field.zip\"):\n \"Load zip file and return complex field\"\n here = op.dirname(op.abspath(__file__))\n data = op.join(here, \"data\")\n arc = zipfile.ZipFile(op.join(data, fname))\n for f in arc.filelist:\n with arc.open(f) as fd:\n if f.filename.count(\"imag\"):\n imag = np.loadtxt(fd)\n\n elif f.filename.count(\"real\"):\n real = np.loadtxt(fd)\n\n field = real + 1j * imag\n return field"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef bootstrap(nside, rand, nbar, *data):\n\n def split(data, indices, axis):\n \"\"\" This function splits array. It fixes the bug\n in numpy that zero length array are improperly handled.\n\n In the future this will be fixed.\n \"\"\"\n s = []\n s.append(slice(0, indices[0]))\n for i in range(len(indices) - 1):\n s.append(slice(indices[i], indices[i+1]))\n s.append(slice(indices[-1], None))\n\n rt = []\n for ss in s:\n ind = [slice(None, None, None) for i in range(len(data.shape))]\n ind[axis] = ss\n ind = tuple(ind)\n rt.append(data[ind])\n return rt\n\n def hpsplit(nside, data):\n # data is (RA, DEC)\n RA, DEC = data\n pix = radec2pix(nside, RA, DEC)\n n = numpy.bincount(pix)\n a = numpy.argsort(pix)\n data = numpy.array(data)[:, a]\n rt = split(data, n.cumsum(), axis=-1)\n return rt\n\n # mean area of sky.\n\n Abar = 41252.96 / nside2npix(nside)\n rand = hpsplit(nside, rand)\n if len(data) > 0:\n data = [list(i) for i in zip(*[hpsplit(nside, d1) for d1 in data])]\n else:\n data = [[] for i in range(len(rand))]\n\n heap = []\n j = 0\n for r, d in zip(rand, data):\n if len(r[0]) == 0: continue\n a = 1.0 * len(r[0]) / nbar\n j = j + 1\n if len(heap) == 0:\n heapq.heappush(heap, (a, j, r, d))\n else:\n a0, j0, r0, d0 = heapq.heappop(heap)\n if a0 + a < Abar:\n a0 += a\n d0 = [\n numpy.concatenate((d0[i], d[i]), axis=-1)\n for i in range(len(d))\n ]\n r0 = numpy.concatenate((r0, r), axis=-1)\n else:\n heapq.heappush(heap, (a, j, r, d))\n heapq.heappush(heap, (a0, j0, r0, d0))\n\n for i in range(len(heap)):\n area, j, r, d = heapq.heappop(heap)\n rt = [area, r] + d\n yield rt", "response": "This function will bootstrap the data based on the sky coverage of a random sample."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nload project specific flag list file", "response": "def load_project_flag_list_file(self, project_exceptions, project):\n \"\"\" Loads project specific lists \"\"\"\n if self.loaded:\n return\n exception_file = None\n for item in project_exceptions:\n if project in item:\n exception_file = item.get(project)\n if exception_file is not None:\n try:\n with open(exception_file, 'r') as f:\n ex = yaml.safe_load(f)\n except IOError:\n logger.error('File not found: %s', exception_file)\n sys.exit(0)\n for key in ex:\n if key in fl:\n fl[key][project] = _merge(fl[key][project], ex.get(key, None)) \\\n if project in fl[key] else ex.get(key, None)\n self.loaded = True\n else:\n logger.info('%s not found in %s', project, ignore_list)\n logger.info('No project specific exceptions will be applied')"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngather sha256 hashes from binary lists", "response": "def binary_hash(self, project, patch_file):\n \"\"\" Gathers sha256 hashes from binary lists \"\"\"\n global il\n exception_file = None\n try:\n project_exceptions = il.get('project_exceptions')\n except KeyError:\n logger.info('project_exceptions missing in %s for %s', ignore_list, project)\n\n for project_files in project_exceptions:\n if project in project_files:\n exception_file = project_files.get(project)\n with open(exception_file, 'r') as f:\n bl = yaml.safe_load(f)\n\n for key, value in bl.items():\n if key == 'binaries':\n if patch_file in value:\n hashvalue = value[patch_file]\n return hashvalue\n else:\n for key, value in il.items():\n if key == 'binaries':\n if patch_file in value:\n hashvalue = value[patch_file]\n return hashvalue\n else:\n hashvalue = \"\"\n return hashvalue\n else:\n logger.info('%s not found in %s', project, ignore_list)\n logger.info('No project specific exceptions will be applied')\n hashvalue = \"\"\n return hashvalue"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef file_audit_list(self, project):\n project_list = False\n self.load_project_flag_list_file(il.get('project_exceptions'), project)\n try:\n default_list = set((fl['file_audits']['file_names']))\n except KeyError:\n logger.error('Key Error processing file_names list values')\n try:\n project_list = set((fl['file_audits'][project]['file_names']))\n logger.info('Loaded %s specific file_audits entries', project)\n except KeyError:\n logger.info('No project specific file_names section for project %s', project)\n\n file_names_re = re.compile(\"|\".join(default_list),\n flags=re.IGNORECASE)\n\n if project_list:\n file_names_proj_re = re.compile(\"|\".join(project_list),\n flags=re.IGNORECASE)\n return file_names_re, file_names_proj_re\n else:\n file_names_proj_re = re.compile(\"\")\n return file_names_re, file_names_proj_re", "response": "Gathers file name lists"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngathering a list of directories to ignore", "response": "def ignore_directories(self, project):\n \"\"\" Gathers a list of directories to ignore \"\"\"\n project_list = False\n try:\n ignore_directories = il['ignore_directories']\n except KeyError:\n logger.error('Key Error processing ignore_directories list values')\n\n try:\n project_exceptions = il.get('project_exceptions')\n for item in project_exceptions:\n if project in item:\n exception_file = item.get(project)\n with open(exception_file, 'r') as f:\n test_list = yaml.safe_load(f)\n project_list = test_list['ignore_directories']\n except KeyError:\n logger.info('No ignore_directories for %s', project)\n\n if project_list:\n ignore_directories = ignore_directories + project_list\n return ignore_directories\n else:\n return ignore_directories"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef url_ignore(self, project):\n project_list = False\n try:\n url_ignore = il['url_ignore']\n except KeyError:\n logger.error('Key Error processing url_ignore list values')\n\n try:\n project_exceptions = il.get('project_exceptions')\n for item in project_exceptions:\n if project in item:\n exception_file = item.get(project)\n with open(exception_file, 'r') as f:\n url_list = yaml.safe_load(f)\n project_list = url_list['url_ignore']\n except KeyError:\n logger.info('No url_ignore for %s', project)\n\n if project_list:\n url_ignore = url_ignore + project_list\n url_ignore_re = re.compile(\"|\".join(url_ignore), flags=re.IGNORECASE)\n return url_ignore_re\n else:\n url_ignore_re = re.compile(\"|\".join(url_ignore), flags=re.IGNORECASE)\n return url_ignore_re", "response": "Gathers a list of URLs to ignore for a project"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef ip_ignore(self, project):\n project_list = False\n try:\n ip_ignore = il['ip_ignore']\n except KeyError:\n logger.error('Key Error processing ip_ignore list values')\n\n try:\n project_exceptions = il.get('project_exceptions')\n for item in project_exceptions:\n if project in item:\n exception_file = item.get(project)\n with open(exception_file, 'r') as f:\n ip_list = yaml.safe_load(f)\n project_list = ip_list['ip_ignore']\n except KeyError:\n logger.info('No ip_ignore for %s', project)\n\n if project_list:\n ip_ignore = ip_ignore + project_list\n ip_ignore_re = re.compile(\"|\".join(ip_ignore), flags=re.IGNORECASE)\n return ip_ignore_re\n else:\n ip_ignore_re = re.compile(\"|\".join(ip_ignore), flags=re.IGNORECASE)\n return ip_ignore_re", "response": "Gathers a list of URLs to ignore for a project"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef download_from_github(fname, path):\n base_url = 'https://github.com/ornlneutronimaging/ImagingReso/blob/master/ImagingReso/reference_data/'\n # Add GitHub junk to the file name for downloading.\n f = fname + '?raw=true'\n url = base_url + f\n block_size = 16384\n req = urlopen(url)\n\n # Get file size from header\n if sys.version_info[0] < 3:\n file_size = int(req.info().getheaders('Content-Length')[0])\n else:\n file_size = req.length\n # downloaded = 0\n\n # Check if file already downloaded\n if os.path.exists(fname):\n if os.path.getsize(fname) == file_size:\n print(\"Skipping downloading '{}'\".format(fname))\n else:\n overwrite = input(\"File size changed, overwrite '{}'? ([y]/n) \".format(fname))\n if overwrite.lower().startswith('n'):\n print(\"Local file '{}' kept without overwriting.\".format(fname))\n\n # Copy file to disk\n print(\"Downloading '{}'... \".format(fname))\n with open(fname, 'wb') as fh:\n while True:\n chunk = req.read(block_size)\n if not chunk:\n break\n fh.write(chunk)\n # downloaded += len(chunk)\n print('')\n print('Download completed.')\n print(\"Unzipping '{}'... \".format(fname))\n\n _database_zip = zipfile.ZipFile(fname)\n _database_zip.extractall(path=path)\n print(\"'{}' has been unzipped and database '{}' is ready to use.\".format(fname, fname.replace('.zip', '')))\n\n os.remove(fname)\n print(\"'{}' has been deleted\".format(fname))", "response": "Download database from GitHub and save it to path."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_list_element_from_database(database='ENDF_VII'):\n _file_path = os.path.abspath(os.path.dirname(__file__))\n _ref_data_folder = os.path.join(_file_path, 'reference_data')\n _database_folder = os.path.join(_ref_data_folder, database)\n\n if not os.path.exists(_ref_data_folder):\n os.makedirs(_ref_data_folder)\n print(\"Folder to store database files has been created: '{}'\".format(_ref_data_folder))\n\n if not os.path.exists(_database_folder):\n print(\"First time using database '{}'? \".format(database))\n print(\"I will retrieve and store a local copy of database'{}': \".format(database))\n download_from_github(fname=database + '.zip', path=_ref_data_folder)\n\n # if '/_elements_list.csv' NOT exist\n if not os.path.exists(_database_folder + '/_elements_list.csv'):\n # glob all .csv files\n _list_files = glob.glob(_database_folder + '/*.csv')\n\n # glob all .h5 files if NO .csv file exist\n if not _list_files:\n _list_files = glob.glob(_database_folder + '/*.h5')\n\n # test if files globed\n _empty_list_boo = not _list_files\n if _empty_list_boo is True:\n raise ValueError(\"'{}' does not contain any '*.csv' or '*.h5' file.\".format(_database_folder))\n\n # convert path/to/file to filename only\n _list_short_filename_without_extension = [os.path.splitext(os.path.basename(_file))[0] for _file in _list_files]\n\n # isolate element names and output as list\n if '-' in _list_short_filename_without_extension[0]:\n _list_element = list(set([_name.split('-')[0] for _name in _list_short_filename_without_extension]))\n else:\n _list_letter_part = list(\n set([re.split(r'(\\d+)', _name)[0] for _name in _list_short_filename_without_extension]))\n _list_element = []\n for each_letter_part in _list_letter_part:\n if len(each_letter_part) <= 2:\n _list_element.append(each_letter_part)\n # save to current dir\n _list_element.sort()\n df_to_save = pd.DataFrame()\n df_to_save['elements'] = _list_element\n df_to_save.to_csv(_database_folder + '/_elements_list.csv')\n # print(\"NOT FOUND '{}'\".format(_database_folder + '/_elements_list.csv'))\n # print(\"SAVED '{}'\".format(_database_folder + '/_elements_list.csv'))\n\n # '/_elements_list.csv' exist\n else:\n df_to_read = pd.read_csv(_database_folder + '/_elements_list.csv')\n _list_element = list(df_to_read['elements'])\n # print(\"FOUND '{}'\".format(_database_folder + '/_elements_list.csv'))\n # print(\"READ '{}'\".format(_database_folder + '/_elements_list.csv'))\n\n return _list_element", "response": "returns a string array of all the element from the database"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef is_element_in_database(element='', database='ENDF_VII'):\n if element == '':\n return False\n\n list_entry_from_database = get_list_element_from_database(database=database)\n if element in list_entry_from_database:\n return True\n return False", "response": "returns True if the element is found in the database False otherwise"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfunction to create a dictionary from a formula", "response": "def formula_to_dictionary(formula='', thickness=np.NaN, density=np.NaN, database='ENDF_VII'):\n \"\"\"create dictionary based on formula given\n \n Parameters:\n ===========\n formula: string\n ex: 'AgCo2'\n ex: 'Ag'\n thickness: float (in mm) default is np.NaN\n density: float (in g/cm3) default is np.NaN\n database: string (default is ENDV_VIII). Database where to look for elements\n \n Raises:\n =======\n ValueError if one of the element is missing from the database\n \n Return:\n =======\n the dictionary of the elements passed\n ex: {'AgCo2': {'elements': ['Ag','Co'],\n 'stoichiometric_ratio': [1,2],\n 'thickness': {'value': thickness,\n 'units': 'mm'},\n 'density': {'value': density,\n 'units': 'g/cm3'},\n 'molar_mass': {'value': np.nan,\n 'units': 'g/mol'},\n }\n \"\"\"\n\n if '.' in formula:\n raise ValueError(\"formula '{}' is invalid, containing symbol '{}' !\".format(formula, '.'))\n _formula_parsed = re.findall(r'([A-Z][a-z]*)(\\d*)', formula)\n if len(_formula_parsed) == 0:\n raise ValueError(\"formula '{}' is invalid !\".format(formula))\n\n # _dictionary = {}\n _elements_array = []\n _atomic_ratio_array = []\n for _element in _formula_parsed:\n [_single_element, _atomic_ratio] = list(_element)\n if not is_element_in_database(element=_single_element, database=database):\n raise ValueError(\"element '{}' not found in the database '{}'!\".format(_single_element, database))\n\n if _atomic_ratio == '':\n _atomic_ratio = 1\n\n _atomic_ratio_array.append(int(_atomic_ratio))\n _elements_array.append(_single_element)\n _dict = {formula: {'elements': _elements_array,\n 'stoichiometric_ratio': _atomic_ratio_array,\n 'thickness': {'value': thickness,\n 'units': 'mm'},\n 'density': {'value': density,\n 'units': 'g/cm3'},\n 'molar_mass': {'value': np.nan,\n 'units': 'g/mol'}\n }\n }\n return _dict"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_isotope_dicts(element='', database='ENDF_VII'):\n _file_path = os.path.abspath(os.path.dirname(__file__))\n _database_folder = os.path.join(_file_path, 'reference_data', database)\n _element_search_path = os.path.join(_database_folder, element + '-*.csv')\n\n list_files = glob.glob(_element_search_path)\n if not list_files:\n raise ValueError(\"File names contains NO '-', the name should in the format of 'Cd-115_m1' or 'Cd-114'\")\n list_files.sort()\n isotope_dict = {'isotopes': {'list': [],\n 'file_names': [],\n 'density': {'value': np.NaN,\n 'units': 'g/cm3'},\n 'mass': {'value': [],\n 'units': 'g/mol',\n },\n 'isotopic_ratio': [], },\n 'density': {'value': np.NaN,\n 'units': 'g/cm3'},\n 'molar_mass': {'value': np.NaN,\n 'units': 'g/mol'},\n }\n\n # isotope_dict_mirror = {}\n _isotopes_list = []\n _isotopes_list_files = []\n _isotopes_mass = []\n _isotopes_density = []\n _isotopes_atomic_ratio = []\n _density = np.NaN\n _molar_mass = np.NaN\n\n for file in list_files:\n # Obtain element, z number from the basename\n _basename = os.path.basename(file)\n filename = os.path.splitext(_basename)[0]\n if '-' in filename:\n [_name, _number] = filename.split('-')\n if '_' in _number:\n [aaa, meta] = _number.split('_')\n _number = aaa[:]\n else:\n _split_list = re.split(r'(\\d+)', filename)\n if len(_split_list) == 2:\n [_name, _number] = _split_list\n else:\n _name = _split_list[0]\n _number = _split_list[1]\n if _number == '0':\n _number = '12'\n _symbol = _number + '-' + _name\n isotope = str(_symbol)\n\n _isotopes_list.append(isotope)\n _isotopes_list_files.append(_basename)\n _isotopes_mass.append(get_mass(isotope))\n _isotopes_atomic_ratio.append(get_abundance(isotope))\n _isotopes_density.append(get_density(isotope))\n _density = get_density(element)\n _molar_mass = get_mass(element)\n\n isotope_dict['isotopes']['list'] = _isotopes_list\n isotope_dict['isotopes']['file_names'] = _isotopes_list_files\n isotope_dict['isotopes']['mass']['value'] = _isotopes_mass\n isotope_dict['isotopes']['isotopic_ratio'] = _isotopes_atomic_ratio\n isotope_dict['isotopes']['density']['value'] = _isotopes_density\n isotope_dict['density']['value'] = _density\n isotope_dict['molar_mass']['value'] = _molar_mass\n return isotope_dict", "response": "returns a dictionary with list of isotopes found in database and name of database files"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_database_data(file_name=''):\n if not os.path.exists(file_name):\n raise IOError(\"File {} does not exist!\".format(file_name))\n df = pd.read_csv(file_name, header=1)\n return df", "response": "return the energy and sigma from the file_name\n \n "} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_interpolated_data(df: pd.DataFrame, e_min=np.nan, e_max=np.nan, e_step=np.nan):\n nbr_point = int((e_max - e_min) / e_step + 1)\n x_axis = np.linspace(e_min, e_max, nbr_point).round(6)\n y_axis_function = interp1d(x=df['E_eV'], y=df['Sig_b'], kind='linear')\n y_axis = y_axis_function(x_axis)\n\n return {'x_axis': x_axis, 'y_axis': y_axis}", "response": "return the interpolated x and y axis for the given x range e_min e_max with step defined\n "} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_sigma(database_file_name='', e_min=np.NaN, e_max=np.NaN, e_step=np.NaN, t_kelvin=None):\n\n file_extension = os.path.splitext(database_file_name)[1]\n\n if t_kelvin is None:\n # '.csv' files\n if file_extension != '.csv':\n raise IOError(\"Cross-section File type must be '.csv'\")\n else:\n _df = get_database_data(file_name=database_file_name)\n _dict = get_interpolated_data(df=_df, e_min=e_min, e_max=e_max,\n e_step=e_step)\n return {'energy_eV': _dict['x_axis'],\n 'sigma_b': _dict['y_axis']}\n else:\n raise ValueError(\"Doppler broadened cross-section in not yet supported in current version.\")", "response": "retrieve the Energy and sigma axis for the given isotope"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_atoms_per_cm3_of_layer(compound_dict: dict):\n # atoms_per_cm3 = {}\n\n _list_of_elements = compound_dict['elements']\n _stoichiometric_list = compound_dict['stoichiometric_ratio']\n\n _element_stoichio = zip(_list_of_elements, _stoichiometric_list)\n _molar_mass_sum = 0\n for _element, _stoichio in _element_stoichio:\n _molar_mass_sum += _stoichio * compound_dict[_element]['molar_mass']['value']\n\n atoms_per_cm3 = Avogadro * compound_dict['density']['value'] / _molar_mass_sum\n # _element_stoichio = zip(_list_of_elements, _stoichiometric_list)\n # for _element, _stoichio in _element_stoichio:\n # _step1 = (compound_dict['density']['value'] * _stoichio) / _molar_mass_sum\n # atoms_per_cm3[_element] = Avogadro * _step1\n\n return _molar_mass_sum, atoms_per_cm3", "response": "calculate the atoms per cm3 of the given compound"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef calculate_linear_attenuation_coefficient(atoms_per_cm3: np.float, sigma_b: np.array):\n miu_per_cm = 1e-24 * sigma_b * atoms_per_cm3\n return np.array(miu_per_cm)", "response": "calculate the linear attenuation coefficient using the formula\n "} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate the transmission signal using the formula", "response": "def calculate_trans(thickness_cm: np.float, miu_per_cm: np.array):\n \"\"\"calculate the transmission signal using the formula\n\n transmission = exp( - thickness_cm * atoms_per_cm3 * 1e-24 * sigma_b)\n\n Parameters:\n ===========\n thickness: float (in cm)\n atoms_per_cm3: float (number of atoms per cm3 of element/isotope)\n sigma_b: np.array of sigma retrieved from database\n\n Returns:\n ========\n transmission array\n \"\"\"\n transmission = np.exp(-thickness_cm * miu_per_cm)\n return np.array(transmission)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef calculate_transmission(thickness_cm: np.float, atoms_per_cm3: np.float, sigma_b: np.array):\n miu_per_cm = calculate_linear_attenuation_coefficient(atoms_per_cm3=atoms_per_cm3, sigma_b=sigma_b)\n transmission = calculate_trans(thickness_cm=thickness_cm, miu_per_cm=miu_per_cm)\n return miu_per_cm, transmission", "response": "calculate the transmission signal using the formula\n \n "} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts distance into new units", "response": "def set_distance_units(value=np.NaN, from_units='mm', to_units='cm'):\n \"\"\"convert distance into new units\n \n Parameters:\n ===========\n value: float. value to convert\n from_units: string. Must be 'mm', 'cm' or 'm'\n to_units: string. must be 'mm','cm' or 'm'\n \n Returns:\n ========\n converted value\n \n Raises:\n =======\n ValueError if from_units is not a valid unit (see above)\n ValueError if to_units is not a valid unit\n \"\"\"\n if from_units == to_units:\n return value\n\n if from_units == 'cm':\n if to_units == 'mm':\n coeff = 10\n elif to_units == 'm':\n coeff = 0.01\n else:\n raise ValueError(\"to_units not supported ['cm','m','mm']!\")\n elif from_units == 'mm':\n if to_units == 'cm':\n coeff = 0.1\n elif to_units == 'm':\n coeff = 0.001\n else:\n raise ValueError(\"to_units not supported ['cm','m','mm']!\")\n elif from_units == 'm':\n if to_units == 'mm':\n coeff = 1000\n elif to_units == 'cm':\n coeff = 100\n else:\n raise ValueError(\"to_units not supported ['cm','m','mm']!\")\n else:\n raise ValueError(\"to_units not supported ['cm','m','mm']!\")\n\n return coeff * value"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts energy to time", "response": "def ev_to_s(offset_us, source_to_detector_m, array):\n # delay values is normal 2.99 us with NONE actual MCP delay settings\n \"\"\"convert energy (eV) to time (us)\n\n Parameters:\n ===========\n array: array (in eV)\n offset_us: float. Delay of detector in us\n source_to_detector_m: float. Distance source to detector in m\n\n Returns:\n ========\n time: array in s \n \"\"\"\n # 1000 is used to convert eV to meV\n time_s = np.sqrt(81.787 / (array * 1000.)) * source_to_detector_m / 3956.\n time_record_s = time_s - offset_us * 1e-6\n return time_record_s"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef s_to_ev(offset_us, source_to_detector_m, array):\n lambda_a = 3956. * (array + offset_us * 1e-6) / source_to_detector_m\n return (81.787 / pow(lambda_a, 2)) / 1000.", "response": "convert time ( s ) to energy ( eV"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting energy in eV to image number", "response": "def ev_to_image_number(offset_us, source_to_detector_m, time_resolution_us, t_start_us, array):\n # delay values is normal 2.99 us with NONE actual MCP delay settings\n \"\"\"convert energy (eV) to image numbers (#)\n\n Parameters:\n ===========\n numpy array of energy in eV\n offset_us: float. Delay of detector in us\n source_to_detector_m: float. Distance source to detector in m\n\n Returns:\n ========\n image numbers: array of image number\n \"\"\"\n time_tot_us = np.sqrt(81.787 / (array * 1000)) * source_to_detector_m * 100 / 0.3956\n time_record_us = (time_tot_us + offset_us)\n image_number = (time_record_us - t_start_us) / time_resolution_us\n return image_number"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nclones a git repository inside a temporary folder then delete the folder.", "response": "def temp_repo(url, branch, commit=''):\n \"\"\" Clone a git repository inside a temporary folder, yield the folder then delete the folder.\n\n :param string url: url of the repo to clone.\n :param string branch: name of the branch to checkout to.\n :param string commit: Optional commit rev to checkout to. If mentioned, that take over the branch\n :return: yield the path to the temporary folder\n :rtype: string\n \"\"\"\n tmp_folder = tempfile.mkdtemp()\n git.Repo.clone_from(\n url, tmp_folder, branch=branch\n )\n if commit:\n git_cmd = git.Git(tmp_folder)\n git_cmd.checkout(commit)\n yield tmp_folder\n shutil.rmtree(tmp_folder)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef force_move(source, destination):\n if not os.path.exists(destination):\n raise RuntimeError(\n 'The code could not be moved to {destination} '\n 'because the folder does not exist'.format(destination=destination))\n\n destination_folder = os.path.join(destination, os.path.split(source)[-1])\n\n if os.path.exists(destination_folder):\n shutil.rmtree(destination_folder)\n shutil.move(source, destination)", "response": "Force the move of the source folder to the destination folder."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _run_command_inside_folder(command, folder):\n logger.debug(\"command: %s\", command)\n # avoid usage of shell = True\n # see https://docs.openstack.org/bandit/latest/plugins/subprocess_popen_with_shell_equals_true.html\n process = subprocess.Popen(command.split(), stdout=subprocess.PIPE, cwd=folder)\n stream_data = process.communicate()[0]\n logger.debug(\"%s stdout: %s (RC %s)\", command, stream_data, process.returncode)\n return process.returncode, stream_data", "response": "Run a command inside the folder."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nparses the given url and update the environment variable if required.", "response": "def parse_url(url):\n \"\"\" Parse the given url and update it with environment value if required.\n\n :param basestring url:\n :rtype: basestring\n :raise: KeyError if environment variable is needed but not found.\n \"\"\"\n # the url has to be a unicode by pystache's design, but the unicode concept has been rewamped in py3\n # we use a try except to make the code compatible with py2 and py3\n try:\n url = unicode(url)\n except NameError:\n url = url\n\n parsed = pystache.parse(url)\n # pylint: disable=protected-access\n variables = (element.key for element in parsed._parse_tree if isinstance(element, _EscapeNode))\n return pystache.render(url, {variable: os.environ[variable] for variable in variables})"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ninstall a third party odoo add - on to destination.", "response": "def install(self, destination):\n \"\"\" Install a third party odoo add-on\n\n :param string destination: the folder where the add-on should end up at.\n \"\"\"\n logger.info(\n \"Installing %s@%s to %s\",\n self.repo, self.commit if self.commit else self.branch, destination\n )\n with temp_repo(self.repo, self.branch, self.commit) as tmp:\n self._apply_patches(tmp)\n self._move_modules(tmp, destination)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nmoving modules froom the temp directory to the destination.", "response": "def _move_modules(self, temp_repo, destination):\n \"\"\"Move modules froom the temp directory to the destination.\n\n :param string temp_repo: the folder containing the code.\n :param string destination: the folder where the add-on should end up at.\n \"\"\"\n folders = self._get_module_folders(temp_repo)\n for folder in folders:\n force_move(folder, destination)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_module_folders(self, temp_repo):\n paths = (\n os.path.join(temp_repo, path) for path in os.listdir(temp_repo)\n if self._is_module_included(path)\n )\n return (path for path in paths if os.path.isdir(path))", "response": "Get a list of module paths contained in a temp directory."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nevaluates if the module must be included in the Odoo addons.", "response": "def _is_module_included(self, module):\n \"\"\"Evaluate if the module must be included in the Odoo addons.\n\n :param string module: the name of the module\n :rtype: bool\n \"\"\"\n if module in self.exclude_modules:\n return False\n\n if self.include_modules is None:\n return True\n\n return module in self.include_modules"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _move_modules(self, temp_repo, destination):\n tmp_addons = os.path.join(temp_repo, 'addons')\n tmp_odoo_addons = os.path.join(temp_repo, 'odoo/addons')\n folders = self._get_module_folders(tmp_addons)\n for folder in folders:\n force_move(folder, tmp_odoo_addons)\n\n tmp_odoo = os.path.join(temp_repo, 'odoo')\n force_move(tmp_odoo, destination)", "response": "Move modules from the temp directory to the destination."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nmerge code from the given repo url to the git repo contained in the given folder.", "response": "def apply(self, folder):\n \"\"\" Merge code from the given repo url to the git repo contained in the given folder.\n\n :param string folder: path of the folder where is the git repo cloned at.\n :raise: RuntimeError if the patch could not be applied.\n \"\"\"\n logger.info(\"Apply Patch %s@%s (commit %s)\", self.url, self.branch, self.commit)\n remote_name = 'patch'\n commands = [\n \"git remote add {} {}\".format(remote_name, self.url),\n \"git fetch {} {}\".format(remote_name, self.branch),\n 'git merge {} -m \"patch\"'.format(self.commit),\n \"git remote remove {}\".format(remote_name),\n ]\n for command in commands:\n return_code, stream_data = _run_command_inside_folder(command, folder)\n if return_code:\n msg = \"Could not apply patch from {}@{}: {}. Error: {}\".format(\n self.url, self.branch, command, stream_data)\n logger.error(msg)\n raise RuntimeError(msg)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef apply(self, folder):\n logger.info(\"Apply Patch File %s\", self.file_path)\n command = \"git apply {}\".format(self.file_path)\n return_code, stream_data = _run_command_inside_folder(command, folder)\n\n if return_code:\n msg = \"Could not apply patch file at {}. Error: {}\".format(self.file_path, stream_data)\n logger.error(msg)\n raise RuntimeError(msg)", "response": "Apply a git patch file to a folder."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating context to keep all needed variables in.", "response": "def _set_up_context(cls):\n \"\"\"Create context to keep all needed variables in.\"\"\"\n cls.context = AttributeDict()\n cls.context.new_meta = {}\n cls.context.new_transitions = {}\n cls.context.new_methods = {}"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _check_states_enum(cls):\n states_enum_name = cls.context.get_config('states_enum_name')\n try:\n cls.context['states_enum'] = getattr(\n cls.context.new_class, states_enum_name)\n except AttributeError:\n raise ValueError('No states enum given!')\n\n proper = True\n try:\n if not issubclass(cls.context.states_enum, Enum):\n proper = False\n except TypeError:\n proper = False\n\n if not proper:\n raise ValueError(\n 'Please provide enum instance to define available states.')", "response": "Check if states enum exists and is proper one."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _check_if_states_are_strings(cls):\n for item in list(cls.context.states_enum):\n if not isinstance(item.value, six.string_types):\n raise ValueError(\n 'Item {name} is not string. Only strings are allowed.'\n .format(name=item.name)\n )", "response": "Check if all states are strings."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _check_state_value(cls):\n state_value = cls.context.get_config('initial_state', None)\n state_value = state_value or getattr(\n cls.context.new_class, cls.context.state_name, None\n )\n\n if not state_value:\n raise ValueError(\n \"Empty state is disallowed, yet no initial state is given!\"\n )\n state_value = (\n cls.context\n .new_meta['translator']\n .translate(state_value)\n )\n cls.context.state_value = state_value", "response": "Check initial state value - if is proper and translate it."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding attributes common to all state machines.", "response": "def _add_standard_attributes(cls):\n \"\"\"Add attributes common to all state machines.\n\n These are methods for setting and checking state etc.\n\n \"\"\"\n setattr(\n cls.context.new_class,\n cls.context.new_meta['state_attribute_name'],\n cls.context.state_value)\n setattr(\n cls.context.new_class,\n cls.context.state_name,\n utils.state_property)\n\n setattr(cls.context.new_class, 'is_', utils.is_)\n setattr(cls.context.new_class, 'can_be_', utils.can_be_)\n setattr(cls.context.new_class, 'set_', utils.set_)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _generate_standard_transitions(cls):\n allowed_transitions = cls.context.get_config('transitions', {})\n for key, transitions in allowed_transitions.items():\n key = cls.context.new_meta['translator'].translate(key)\n\n new_transitions = set()\n for trans in transitions:\n if not isinstance(trans, Enum):\n trans = cls.context.new_meta['translator'].translate(trans)\n new_transitions.add(trans)\n\n cls.context.new_transitions[key] = new_transitions\n\n for state in cls.context.states_enum:\n if state not in cls.context.new_transitions:\n cls.context.new_transitions[state] = set()", "response": "Generate methods used for transitions."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngenerating standard setters getters and checkers.", "response": "def _generate_standard_methods(cls):\n \"\"\"Generate standard setters, getters and checkers.\"\"\"\n for state in cls.context.states_enum:\n getter_name = 'is_{name}'.format(name=state.value)\n cls.context.new_methods[getter_name] = utils.generate_getter(state)\n\n setter_name = 'set_{name}'.format(name=state.value)\n cls.context.new_methods[setter_name] = utils.generate_setter(state)\n\n checker_name = 'can_be_{name}'.format(name=state.value)\n checker = utils.generate_checker(state)\n cls.context.new_methods[checker_name] = checker\n\n cls.context.new_methods['actual_state'] = utils.actual_state\n cls.context.new_methods['as_enum'] = utils.as_enum\n cls.context.new_methods['force_set'] = utils.force_set"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _add_new_methods(cls):\n for name, method in cls.context.new_methods.items():\n if hasattr(cls.context.new_class, name):\n raise ValueError(\n \"Name collision in state machine class - '{name}'.\"\n .format(name)\n )\n\n setattr(cls.context.new_class, name, method)", "response": "Add all generated methods to result class."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncheck and set complete option.", "response": "def _set_complete_option(cls):\n \"\"\"Check and set complete option.\"\"\"\n get_config = cls.context.get_config\n complete = get_config('complete', None)\n if complete is None:\n conditions = [\n get_config('transitions', False),\n get_config('named_transitions', False),\n ]\n complete = not any(conditions)\n\n cls.context.new_meta['complete'] = complete"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the absolute path to the directory containing the package data.", "response": "def data_directory():\n \"\"\"Return the absolute path to the directory containing the package data.\"\"\"\n package_directory = os.path.abspath(os.path.dirname(__file__))\n return os.path.join(package_directory, \"data\")"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nselects a map by name or critiera", "response": "def selectMap(name=None, excludeName=False, closestMatch=True, **tags):\n \"\"\"select a map by name and/or critiera\"\"\"\n matches = filterMapAttrs(**tags)\n if not matches: raise c.InvalidMapSelection(\"could not find any matching maps given criteria: %s\"%tags)\n if name: # if name is specified, consider only the best-matching names only\n matches = filterMapNames(name, excludeRegex=excludeName, closestMatch=closestMatch, records=matches)\n try:\n if closestMatch: return random.choice(matches) # pick any map at random that matches all criteria\n elif matches: return matches\n except IndexError: pass # matches is empty still\n raise c.InvalidMapSelection(\"requested map '%s', but could not locate \"\\\n \"it within %s or its subdirectories. Submit the map to https://\"\\\n \"github.com/ttinies/sc2gameMapRepo/tree/master/sc2maptool/maps\"%(\n name, c.PATH_MAP_INSTALL))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef filterMapAttrs(records=getIndex(), **tags):\n if len(tags) == 0: return records # otherwise if unspecified, all given records match\n ret = []\n for record in records: # attempt to match attributes\n if matchRecordAttrs(record, tags):\n ret.append(record)\n return ret", "response": "returns a list of records that match the given tags"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef matchRecordAttrs(mapobj, attrs):\n for k,v in iteritems(attrs):\n try: val = getattr(mapobj, k)\n except AttributeError: # k isn't an attr of record\n if bool(v): return False # if k doesn't exist in mapobj but was required, no match\n else: continue # otherwise ignore attributes that aren't defined for the given map record\n if val != v: return False # if any criteria matches, it's considered a match\n return True", "response": "attempt to match given attributes against a single map object s attributes"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef filterMapNames(regexText, records=getIndex(), excludeRegex=False, closestMatch=True):\n bestScr = 99999 # a big enough number to not be a valid file system path\n regex = re.compile(regexText, flags=re.IGNORECASE)\n ret = []\n if excludeRegex: # match only records that do NOT contain regex\n if regexText and closestMatch: # then maps with fewer characters are better matches\n for m in list(records):\n if re.search(regex, m.name): continue # map must NOT contain specified phrase\n score = len(m.name) # the map with the smallest map name means it has the largets matching character percentage\n if score == bestScr:\n bestScr = score\n ret.append(m)\n elif score < bestScr: # new set of best maps\n bestScr = score\n ret = [m]\n else: # all maps that match regex are included\n for m in list(records):\n if re.search(regex, m.name): continue # map must NOT contain specified phrase\n ret.append(m) # any mapname containing regex matches\n else: # only match records that contain regex\n if regexText and closestMatch: # then maps with fewer characters are better matches\n for m in records:\n if not re.search(regex, m.name): continue # map must contain specified phrase if excludeRegex==True\n score = len(m.name) # the map with the smallest map name means it has the largets matching character percentage\n if score == bestScr:\n bestScr = score\n ret.append(m)\n elif score < bestScr: # new group of best maps\n bestScr = score\n ret = [m]\n else: # all maps that match regex are included\n for m in records:\n if not re.search(regex, m.name): continue # map must contain specified phrase if excludeRegex==True\n ret.append(m) # any mapname containing regex matches\n return ret", "response": "matches each record against regexText according to parameters\n returns a list of maps that match the regexText"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntry conversion of any value to a boolean", "response": "def to_boolean(value, ctx):\n \"\"\"\n Tries conversion of any value to a boolean\n \"\"\"\n if isinstance(value, bool):\n return value\n elif isinstance(value, int):\n return value != 0\n elif isinstance(value, Decimal):\n return value != Decimal(0)\n elif isinstance(value, str):\n value = value.lower()\n if value == 'true':\n return True\n elif value == 'false':\n return False\n elif isinstance(value, datetime.date) or isinstance(value, datetime.time):\n return True\n\n raise EvaluationError(\"Can't convert '%s' to a boolean\" % str(value))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ntrying conversion of any value to an integer", "response": "def to_integer(value, ctx):\n \"\"\"\n Tries conversion of any value to an integer\n \"\"\"\n if isinstance(value, bool):\n return 1 if value else 0\n elif isinstance(value, int):\n return value\n elif isinstance(value, Decimal):\n try:\n val = int(value.to_integral_exact(ROUND_HALF_UP))\n if isinstance(val, int):\n return val\n except ArithmeticError:\n pass\n elif isinstance(value, str):\n try:\n return int(value)\n except ValueError:\n pass\n\n raise EvaluationError(\"Can't convert '%s' to an integer\" % str(value))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntries conversion of any value to a decimal", "response": "def to_decimal(value, ctx):\n \"\"\"\n Tries conversion of any value to a decimal\n \"\"\"\n if isinstance(value, bool):\n return Decimal(1) if value else Decimal(0)\n elif isinstance(value, int):\n return Decimal(value)\n elif isinstance(value, Decimal):\n return value\n elif isinstance(value, str):\n try:\n return Decimal(value)\n except Exception:\n pass\n\n raise EvaluationError(\"Can't convert '%s' to a decimal\" % str(value))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef to_string(value, ctx):\n if isinstance(value, bool):\n return \"TRUE\" if value else \"FALSE\"\n elif isinstance(value, int):\n return str(value)\n elif isinstance(value, Decimal):\n return format_decimal(value)\n elif isinstance(value, str):\n return value\n elif type(value) == datetime.date:\n return value.strftime(ctx.get_date_format(False))\n elif isinstance(value, datetime.time):\n return value.strftime('%H:%M')\n elif isinstance(value, datetime.datetime):\n return value.astimezone(ctx.timezone).isoformat()\n\n raise EvaluationError(\"Can't convert '%s' to a string\" % str(value))", "response": "Converts any value to a string"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef to_date(value, ctx):\n if isinstance(value, str):\n temporal = ctx.get_date_parser().auto(value)\n if temporal is not None:\n return to_date(temporal, ctx)\n elif type(value) == datetime.date:\n return value\n elif isinstance(value, datetime.datetime):\n return value.date() # discard time\n\n raise EvaluationError(\"Can't convert '%s' to a date\" % str(value))", "response": "Tries conversion of any value to a date"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntry conversion of any value to a datetime", "response": "def to_datetime(value, ctx):\n \"\"\"\n Tries conversion of any value to a datetime\n \"\"\"\n if isinstance(value, str):\n temporal = ctx.get_date_parser().auto(value)\n if temporal is not None:\n return to_datetime(temporal, ctx)\n elif type(value) == datetime.date:\n return ctx.timezone.localize(datetime.datetime.combine(value, datetime.time(0, 0)))\n elif isinstance(value, datetime.datetime):\n return value.astimezone(ctx.timezone)\n\n raise EvaluationError(\"Can't convert '%s' to a datetime\" % str(value))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef to_date_or_datetime(value, ctx):\n if isinstance(value, str):\n temporal = ctx.get_date_parser().auto(value)\n if temporal is not None:\n return temporal\n elif type(value) == datetime.date:\n return value\n elif isinstance(value, datetime.datetime):\n return value.astimezone(ctx.timezone)\n\n raise EvaluationError(\"Can't convert '%s' to a date or datetime\" % str(value))", "response": "Tries conversion of any value to a date or datetime"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ntry conversion of any value to a time", "response": "def to_time(value, ctx):\n \"\"\"\n Tries conversion of any value to a time\n \"\"\"\n if isinstance(value, str):\n time = ctx.get_date_parser().time(value)\n if time is not None:\n return time\n elif isinstance(value, datetime.time):\n return value\n elif isinstance(value, datetime.datetime):\n return value.astimezone(ctx.timezone).time()\n\n raise EvaluationError(\"Can't convert '%s' to a time\" % str(value))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef to_same(value1, value2, ctx):\n if type(value1) == type(value2):\n return value1, value2\n\n try:\n # try converting to two decimals\n return to_decimal(value1, ctx), to_decimal(value2, ctx)\n except EvaluationError:\n pass\n\n try:\n # try converting to two dates\n d1, d2 = to_date_or_datetime(value1, ctx), to_date_or_datetime(value2, ctx)\n\n # if either one is a datetime, then the other needs to become a datetime\n if type(value1) != type(value2):\n d1, d2 = to_datetime(d1, ctx), to_datetime(d2, ctx)\n return d1, d2\n except EvaluationError:\n pass\n\n # try converting to two strings\n return to_string(value1, ctx), to_string(value2, ctx)", "response": "Converts two values to the most - likely type."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef to_repr(value, ctx):\n as_string = to_string(value, ctx)\n\n if isinstance(value, str) or isinstance(value, datetime.date) or isinstance(value, datetime.time):\n as_string = as_string.replace('\"', '\"\"') # escape quotes by doubling\n as_string = '\"%s\"' % as_string\n\n return as_string", "response": "Converts a value back to its representation form e. g. x -> x"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nformats a decimal number", "response": "def format_decimal(decimal):\n \"\"\"\n Formats a decimal number\n :param decimal: the decimal value\n :return: the formatted string value\n \"\"\"\n # strip trailing fractional zeros\n normalized = decimal.normalize()\n sign, digits, exponent = normalized.as_tuple()\n if exponent >= 1:\n normalized = normalized.quantize(1)\n\n return str(normalized)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncheck if I am running inside a Linux container.", "response": "def is_containerized() -> bool:\n '''\n Check if I am running inside a Linux container.\n '''\n try:\n cginfo = Path('/proc/self/cgroup').read_text()\n if '/docker/' in cginfo or '/lxc/' in cginfo:\n return True\n except IOError:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef detect_cloud() -> str:\n '''\n Detect the cloud provider where I am running on.\n '''\n # NOTE: Contributions are welcome!\n # Please add other cloud providers such as Rackspace, IBM BlueMix, etc.\n if sys.platform.startswith('linux'):\n # Google Cloud Platform or Amazon AWS (hvm)\n try:\n # AWS Nitro-based instances\n mb = Path('/sys/devices/virtual/dmi/id/board_vendor').read_text().lower()\n if 'amazon' in mb:\n return 'amazon'\n except IOError:\n pass\n try:\n bios = Path('/sys/devices/virtual/dmi/id/bios_version').read_text().lower()\n if 'google' in bios:\n return 'google'\n if 'amazon' in bios:\n return 'amazon'\n except IOError:\n pass\n # Microsoft Azure\n # https://gallery.technet.microsoft.com/scriptcenter/Detect-Windows-Azure-aed06d51\n # TODO: this only works with Debian/Ubuntu instances.\n # TODO: this does not work inside containers.\n try:\n dhcp = Path('/var/lib/dhcp/dhclient.eth0.leases').read_text()\n if 'unknown-245' in dhcp:\n return 'azure'\n # alternative method is to read /var/lib/waagent/GoalState.1.xml\n # but it requires sudo privilege.\n except IOError:\n pass\n else:\n log.warning('Cloud detection is implemented for Linux only yet.')\n return None", "response": "Detect the cloud provider where I am running on."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef addBlock(self,branch=None,btype=None,mtype=None,attributes=None):\n '''\n Add block definition to list of blocks in material\n Order for list entry: branch level (0=root), block type (material,solid,fluid,etc.), matid (integer if root, False otherwise),\n material name (string if root, False otherwise), material type, dictionary of attributes or False if none\n '''\n if branch == 0:\n attributes = self.attributes\n blk = {'branch': branch,\n 'btype': btype,\n 'mtype': mtype,\n 'attributes': attributes}\n \n self.blocks.append(blk)", "response": "Add block definition to list of blocks in material\n "} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef refocus(field, d, nm, res, method=\"helmholtz\", num_cpus=1, padding=True):\n # FFT of field\n fshape = len(field.shape)\n assert fshape in [1, 2], \"Dimension of `field` must be 1 or 2.\"\n\n func = fft_propagate\n names = func.__code__.co_varnames[:func.__code__.co_argcount]\n\n loc = locals()\n vardict = dict()\n\n for name in names:\n if name in loc:\n vardict[name] = loc[name]\n\n if padding:\n field = pad.pad_add(field)\n\n vardict[\"fftfield\"] = np.fft.fftn(field)\n\n refoc = func(**vardict)\n\n if padding:\n refoc = pad.pad_rem(refoc)\n\n return refoc", "response": "Refocus a 1D or 2D array of electric fields at d."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef refocus_stack(fieldstack, d, nm, res, method=\"helmholtz\",\n num_cpus=_cpu_count, copy=True, padding=True):\n \"\"\"Refocus a stack of 1D or 2D fields\n\n\n Parameters\n ----------\n fieldstack : 2d or 3d array\n Stack of 1D or 2D background corrected electric fields (Ex/BEx).\n The first axis iterates through the individual fields.\n d : float\n Distance to be propagated in pixels (negative for backwards)\n nm : float\n Refractive index of medium\n res : float\n Wavelenth in pixels\n method : str\n Defines the method of propagation;\n one of\n\n - \"helmholtz\" : the optical transfer function `exp(idk\u2098(M-1))`\n - \"fresnel\" : paraxial approximation `exp(idk\u00b2/k\u2098)`\n\n num_cpus : str\n Defines the number of CPUs to be used for refocusing.\n copy : bool\n If False, overwrites input stack.\n padding : bool\n Perform padding with linear ramp from edge to average\n to reduce ringing artifacts.\n\n .. versionadded:: 0.1.4\n\n Returns\n -------\n Electric field stack at `d`.\n \"\"\"\n\n func = refocus\n names = func.__code__.co_varnames[:func.__code__.co_argcount]\n\n loc = locals()\n vardict = dict()\n for name in names:\n if name in loc.keys():\n vardict[name] = loc[name]\n # default keyword arguments\n func_def = func.__defaults__[::-1]\n\n # child processes should only use one cpu\n vardict[\"num_cpus\"] = 1\n vardict[\"padding\"] = padding\n\n M = fieldstack.shape[0]\n stackargs = list()\n\n # Create individual arglists for all fields\n for m in range(M):\n kwarg = vardict.copy()\n kwarg[\"field\"] = fieldstack[m]\n # now we turn the kwarg into an arglist\n args = list()\n for i, a in enumerate(names[::-1]):\n # first set default\n if i < len(func_def):\n val = func_def[i]\n if a in kwarg:\n val = kwarg[a]\n args.append(val)\n stackargs.append(args[::-1])\n\n p = mp.Pool(num_cpus)\n result = p.map_async(_refocus_wrapper, stackargs).get()\n p.close()\n p.terminate()\n p.join()\n\n if copy:\n data = np.zeros(fieldstack.shape, dtype=result[0].dtype)\n else:\n data = fieldstack\n\n for m in range(M):\n data[m] = result[m]\n\n return data", "response": "Refocuses a stack of electric fields at d."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\npropagates a 1D or 2D Fourier transformed field at d nm res", "response": "def fft_propagate(fftfield, d, nm, res, method=\"helmholtz\",\n ret_fft=False):\n \"\"\"Propagates a 1D or 2D Fourier transformed field\n\n\n Parameters\n ----------\n fftfield : 1-dimensional or 2-dimensional ndarray\n Fourier transform of 1D Electric field component\n d : float\n Distance to be propagated in pixels (negative for backwards)\n nm : float\n Refractive index of medium\n res : float\n Wavelength in pixels\n method : str\n Defines the method of propagation;\n one of\n\n - \"helmholtz\" : the optical transfer function `exp(idk\u2098(M-1))`\n - \"fresnel\" : paraxial approximation `exp(idk\u00b2/k\u2098)`\n\n ret_fft : bool\n Do not perform an inverse Fourier transform and return the field\n in Fourier space.\n\n\n Returns\n -------\n Electric field at `d`. If `ret_fft` is True, then the\n Fourier transform of the electric field will be returned (faster).\n \"\"\"\n fshape = len(fftfield.shape)\n assert fshape in [1, 2], \"Dimension of `fftfield` must be 1 or 2.\"\n\n if fshape == 1:\n func = fft_propagate_2d\n else:\n func = fft_propagate_3d\n\n names = func.__code__.co_varnames[:func.__code__.co_argcount]\n\n loc = locals()\n vardict = dict()\n for name in names:\n vardict[name] = loc[name]\n\n return func(**vardict)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fft_propagate_2d(fftfield, d, nm, res, method=\"helmholtz\",\n ret_fft=False):\n \"\"\"Propagate a 1D Fourier transformed field in 2D\n\n\n Parameters\n ----------\n fftfield : 1d array\n Fourier transform of 1D Electric field component\n d : float\n Distance to be propagated in pixels (negative for backwards)\n nm : float\n Refractive index of medium\n res : float\n Wavelength in pixels\n method : str\n Defines the method of propagation;\n one of\n\n - \"helmholtz\" : the optical transfer function `exp(idk\u2098(M-1))`\n - \"fresnel\" : paraxial approximation `exp(idk\u00b2/k\u2098)`\n\n ret_fft : bool\n Do not perform an inverse Fourier transform and return the field\n in Fourier space.\n\n\n Returns\n -------\n Electric field at `d`. If `ret_fft` is True, then the\n Fourier transform of the electric field will be returned (faster).\n \"\"\"\n assert len(fftfield.shape) == 1, \"Dimension of `fftfield` must be 1.\"\n km = (2 * np.pi * nm) / res\n kx = np.fft.fftfreq(len(fftfield)) * 2 * np.pi\n\n # free space propagator is\n if method == \"helmholtz\":\n # exp(i*sqrt(km\u00b2-kx\u00b2)*d)\n # Also subtract incoming plane wave. We are only considering\n # the scattered field here.\n root_km = km**2 - kx**2\n rt0 = (root_km > 0)\n # multiply by rt0 (filter in Fourier space)\n fstemp = np.exp(1j * (np.sqrt(root_km * rt0) - km) * d) * rt0\n elif method == \"fresnel\":\n # exp(i*d*(km-kx\u00b2/(2*km))\n # fstemp = np.exp(-1j * d * (kx**2/(2*km)))\n fstemp = np.exp(-1j * d * (kx**2/(2*km)))\n else:\n raise ValueError(\"Unknown method: {}\".format(method))\n\n if ret_fft:\n return fftfield * fstemp\n else:\n return np.fft.ifft(fftfield * fstemp)", "response": "Propagate a 1D Fourier transformed field in 2D."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef fft_propagate_3d(fftfield, d, nm, res, method=\"helmholtz\",\n ret_fft=False):\n \"\"\"Propagate a 2D Fourier transformed field in 3D\n\n\n Parameters\n ----------\n fftfield : 2d array\n Fourier transform of 2D Electric field component\n d : float\n Distance to be propagated in pixels (negative for backwards)\n nm : float\n Refractive index of medium\n res : float\n Wavelength in pixels\n method : str\n Defines the method of propagation;\n one of\n\n - \"helmholtz\" : the optical transfer function `exp(idk\u2098(M-1))`\n - \"fresnel\" : paraxial approximation `exp(idk\u00b2/k\u2098)`\n\n ret_fft : bool\n Do not perform an inverse Fourier transform and return the field\n in Fourier space.\n\n\n Returns\n -------\n Electric field at `d`. If `ret_fft` is True, then the\n Fourier transform of the electric field will be returned (faster).\n \"\"\"\n assert len(fftfield.shape) == 2, \"Dimension of `fftfield` must be 2.\"\n # if fftfield.shape[0] != fftfield.shape[1]:\n # raise NotImplementedError(\"Field must be square shaped.\")\n # free space propagator is\n # exp(i*sqrt(km**2-kx**2-ky**2)*d)\n km = (2 * np.pi * nm) / res\n kx = (np.fft.fftfreq(fftfield.shape[0]) * 2 * np.pi).reshape(-1, 1)\n ky = (np.fft.fftfreq(fftfield.shape[1]) * 2 * np.pi).reshape(1, -1)\n if method == \"helmholtz\":\n # exp(i*sqrt(km\u00b2-kx\u00b2-ky\u00b2)*d)\n root_km = km**2 - kx**2 - ky**2\n rt0 = (root_km > 0)\n # multiply by rt0 (filter in Fourier space)\n fstemp = np.exp(1j * (np.sqrt(root_km * rt0) - km) * d) * rt0\n elif method == \"fresnel\":\n # exp(i*d*(km-(kx\u00b2+ky\u00b2)/(2*km))\n # fstemp = np.exp(-1j * d * (kx**2+ky**2)/(2*km))\n fstemp = np.exp(-1j * d * (kx**2 + ky**2)/(2*km))\n else:\n raise ValueError(\"Unknown method: {}\".format(method))\n # fstemp[np.where(np.isnan(fstemp))] = 0\n # Also subtract incoming plane wave. We are only considering\n # the scattered field here.\n if ret_fft:\n return fftfield * fstemp\n else:\n return np.fft.ifft2(fftfield * fstemp)", "response": "Propagate a 2D Fourier transformed field in 3D"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\npushing the object to the GIT repository.", "response": "def push(self):\n \"\"\"\n Adding the no_thin argument to the GIT push because we had some issues pushing previously.\n According to http://stackoverflow.com/questions/16586642/git-unpack-error-on-push-to-gerrit#comment42953435_23610917,\n \"a new optimization which causes git to send as little data as possible over the network caused this bug to manifest, \n so my guess is --no-thin just turns these optimizations off. From git push --help: \"A thin transfer significantly \n reduces the amount of sent data when the sender and receiver share many of the same objects in common.\" (--thin is the default).\"\n \"\"\"\n\n if not self.canRunRemoteCmd(): \n return None\n\n try:\n fetchInfo = self.repo.remotes.origin.push(no_thin=True)[0]\n except exc.GitCommandError as e:\n print(dir(e))\n print(e)\n raise\n\n\n if fetchInfo.flags & fetchInfo.ERROR:\n try:\n raise IOError(\"An error occured while trying to push the GIT repository from the server. Error flag: '\" + \n str(fetchInfo.flags) + \"', message: '\" + str(fetchInfo.note) + \"'.\")\n except:\n IOError(\"An error occured while trying to push the GIT repository from the server.\")\n return fetchInfo"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef autofocus_stack(fieldstack, nm, res, ival, roi=None,\n metric=\"average gradient\", padding=True,\n same_dist=False, ret_ds=False, ret_grads=False,\n num_cpus=_cpu_count, copy=True):\n \"\"\"Numerical autofocusing of a stack using the Helmholtz equation.\n\n\n Parameters\n ----------\n fieldstack : 2d or 3d ndarray\n Electric field is BG-Corrected, i.e. Field = EX/BEx\n nm : float\n Refractive index of medium.\n res : float\n Size of wavelength in pixels.\n ival : tuple of floats\n Approximate interval to search for optimal focus in px.\n metric : str\n see `autofocus_field`.\n padding : bool\n Perform padding with linear ramp from edge to average\n to reduce ringing artifacts.\n\n .. versionchanged:: 0.1.4\n improved padding value and padding location\n ret_dopt : bool\n Return optimized distance and gradient plotting data.\n same_dist : bool\n Refocus entire sinogram with one distance.\n red_ds : bool\n Return the autofocusing distances in pixels. Defaults to False.\n If sam_dist is True, still returns autofocusing distances\n of first pass. The used refocusing distance is the\n average.\n red_grads : bool\n Return the computed gradients as a list.\n copy : bool\n If False, overwrites input array.\n\n\n Returns\n -------\n The focused field (and the refocussing distance + data if d is None)\n \"\"\"\n dopt = list()\n grad = list()\n\n M = fieldstack.shape[0]\n\n # setup arguments\n stackargs = list()\n for s in range(M):\n stackargs.append([fieldstack[s].copy(copy), nm, res, ival,\n roi, metric, padding, True, True, 1])\n # perform first pass\n p = mp.Pool(num_cpus)\n result = p.map_async(_autofocus_wrapper, stackargs).get()\n p.close()\n p.terminate()\n p.join()\n # result = []\n # for arg in stackargs:\n # result += _autofocus_wrapper(arg)\n\n newstack = np.zeros(fieldstack.shape, dtype=fieldstack.dtype)\n\n for s in range(M):\n field, ds, gs = result[s]\n dopt.append(ds)\n grad.append(gs)\n newstack[s] = field\n\n # perform second pass if `same_dist` is True\n if same_dist:\n # find average dopt\n davg = np.average(dopt)\n newstack = refocus_stack(fieldstack, davg, nm, res,\n num_cpus=num_cpus, copy=copy,\n padding=padding)\n\n ret_list = [newstack]\n if ret_ds:\n ret_list += [dopt]\n if ret_grads:\n ret_list += [grad]\n\n if len(ret_list) == 1:\n return ret_list[0]\n else:\n return tuple(ret_list)", "response": "This function calculates the autofocusing of a single stack of sinograms."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef minimize_metric(field, metric_func, nm, res, ival, roi=None,\n coarse_acc=1, fine_acc=.005,\n return_gradient=True, padding=True):\n \"\"\"Find the focus by minimizing the `metric` of an image\n\n Parameters\n ----------\n field : 2d array\n electric field\n metric_func : callable\n some metric to be minimized\n ival : tuple of floats\n (minimum, maximum) of interval to search in pixels\n nm : float\n RI of medium\n res : float\n wavelength in pixels\n roi : rectangular region of interest (x1, y1, x2, y2)\n Region of interest of `field` for which the metric will be\n minimized. If not given, the entire `field` will be used.\n coarse_acc : float\n accuracy for determination of global minimum in pixels\n fine_acc : float\n accuracy for fine localization percentage of gradient change\n return_gradient:\n return x and y values of computed gradient\n padding : bool\n perform padding with linear ramp from edge to average\n to reduce ringing artifacts.\n\n .. versionchanged:: 0.1.4\n improved padding value and padding location\n \"\"\"\n if roi is not None:\n assert len(roi) == len(field.shape) * \\\n 2, \"ROI must match field dimension\"\n\n initshape = field.shape\n Fshape = len(initshape)\n propfunc = fft_propagate\n\n if roi is None:\n if Fshape == 2:\n roi = (0, 0, field.shape[0], field.shape[1])\n else:\n roi = (0, field.shape[0])\n\n roi = 1*np.array(roi)\n\n if padding:\n # Pad with correct complex number\n field = pad.pad_add(field)\n\n if ival[0] > ival[1]:\n ival = (ival[1], ival[0])\n # set coarse interval\n # coarse_acc = int(np.ceil(ival[1]-ival[0]))/100\n N = 100 / coarse_acc\n zc = np.linspace(ival[0], ival[1], N, endpoint=True)\n\n # compute fft of field\n fftfield = np.fft.fftn(field)\n\n # fftplan = fftw3.Plan(fftfield.copy(), None, nthreads = _ncores,\n # direction=\"backward\", flags=_fftwflags)\n\n # initiate gradient vector\n gradc = np.zeros(zc.shape)\n for i in range(len(zc)):\n d = zc[i]\n # fsp = propfunc(fftfield, d, nm, res, fftplan=fftplan)\n fsp = propfunc(fftfield, d, nm, res)\n if Fshape == 2:\n gradc[i] = metric_func(fsp[roi[0]:roi[2], roi[1]:roi[3]])\n else:\n gradc[i] = metric_func(fsp[roi[0]:roi[1]])\n\n minid = np.argmin(gradc)\n if minid == 0:\n zc -= zc[1] - zc[0]\n minid += 1\n if minid == len(zc) - 1:\n zc += zc[1] - zc[0]\n minid -= 1\n zf = 1*zc\n gradf = 1 * gradc\n\n numfine = 10\n mingrad = gradc[minid]\n\n while True:\n gradf = np.zeros(numfine)\n ival = (zf[minid - 1], zf[minid + 1])\n zf = np.linspace(ival[0], ival[1], numfine)\n for i in range(len(zf)):\n d = zf[i]\n fsp = propfunc(fftfield, d, nm, res)\n if Fshape == 2:\n gradf[i] = metric_func(fsp[roi[0]:roi[2], roi[1]:roi[3]])\n else:\n gradf[i] = metric_func(fsp[roi[0]:roi[1]])\n minid = np.argmin(gradf)\n if minid == 0:\n zf -= zf[1] - zf[0]\n minid += 1\n if minid == len(zf) - 1:\n zf += zf[1] - zf[0]\n minid -= 1\n if abs(mingrad - gradf[minid]) / 100 < fine_acc:\n break\n\n minid = np.argmin(gradf)\n fsp = propfunc(fftfield, zf[minid], nm, res)\n\n if padding:\n fsp = pad.pad_rem(fsp)\n\n if return_gradient:\n return fsp, zf[minid], [(zc, gradc), (zf, gradf)]\n return fsp, zf[minid]", "response": "Find the focus by minimizing a metric of an image."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef prepare_project(project, project_dir, binaries, ips, urls):\n\n # Get Various Lists / Project Waivers\n lists = get_lists.GetLists()\n\n # Get file name black list and project waivers\n file_audit_list, file_audit_project_list = lists.file_audit_list(project)\n\n # Get file content black list and project waivers\n flag_list, ignore_list = lists.file_content_list(project)\n\n # Get File Ignore Lists\n file_ignore = lists.file_ignore()\n\n ignore_directories = lists.ignore_directories(project)\n\n # Get URL Ignore Lists\n url_ignore = lists.url_ignore(project)\n\n # Get IP Ignore Lists\n ip_ignore = lists.ip_ignore(project)\n\n # Get Binary Ignore Lists\n hashlist = get_lists.GetLists()\n\n if binaries or ips or urls:\n try:\n apikey = os.environ[\"VT_KEY\"]\n except KeyError:\n logger.error(\"Please set your virustotal.com API key as an environment variable\")\n sys.exit(1)\n try:\n vt_rate_type = config.get('config', 'vt_rate_type')\n except six.moves.configparser.NoSectionError:\n logger.error(\"A config section is required for vt_rate_type with a public | private option\")\n sys.exit(1)\n\n patten = re.compile(r'\\bpublic\\b|\\bprivate\\b')\n if not patten.match(vt_rate_type):\n logger.error(\"Unrecognized %s option for vt_rate_type\", vt_rate_type)\n sys.exit(1)\n\n # Perform rudimentary scans\n scan_file(project, project_dir, binaries, ips, urls, file_audit_list,\n file_audit_project_list, flag_list, ignore_list, hashlist,\n file_ignore, ignore_directories, url_ignore, ip_ignore, apikey)", "response": "Prepares the project for rudimentary analysis"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nscans a file for a specific project.", "response": "def scan_file(project, project_dir, binaries, ips, urls, file_audit_list,\n file_audit_project_list, flag_list, ignore_list, hashlist,\n file_ignore, ignore_directories, url_ignore, ip_ignore, apikey):\n \"\"\"\n Main scan tasks begin\n \"\"\"\n logger.info(\"Commencing scan tasks..\")\n for root, dirs, files in os.walk(project_dir):\n # Filter out ignored directories from list.\n for items in files:\n full_path = os.path.join(root, items)\n split_path = full_path.split(project + '/', 1)[-1]\n # Check for Blacklisted file names\n if not any(x in split_path for x in ignore_directories):\n if file_audit_list.search(full_path) and not \\\n file_audit_project_list.search(full_path):\n match = file_audit_list.search(full_path)\n logger.error('Blacklisted filename: %s', full_path)\n logger.error('Matched String: %s', match.group())\n with open(reports_dir + \"file-names_\" + project + \".log\",\n \"a\") as gate_report:\n gate_report. \\\n write('Blacklisted filename: {0}\\n'.\n format(full_path))\n gate_report. \\\n write('Matched String: {0}'.\n format(match.group()))\n\n # Check if Binary is whitelisted\n if is_binary(full_path) and binaries:\n\n split_path = full_path.split(project + '/', 1)[-1]\n binary_hash = hashlist.binary_hash(project, split_path)\n with open(full_path, 'rb') as afile:\n hasher = hashlib.sha256()\n buf = afile.read()\n hasher.update(buf)\n sha256hash = hasher.hexdigest()\n if sha256hash in binary_hash:\n logger.info('Found matching file hash for: %s', full_path)\n logger.info('No further action needed for: %s', full_path)\n else:\n logger.info('Non Whitelisted Binary file: %s', full_path)\n scan_binary(full_path, split_path, project, sha256hash, apikey)\n\n else:\n if not items.endswith(tuple(file_ignore)) and not is_binary(full_path):\n try:\n fo = open(full_path, 'r')\n lines = fo.readlines()\n except IOError:\n logger.error('%s does not exist', full_path)\n\n for line in lines:\n # Find IP Addresses and send for report to Virus Total\n if ips:\n ipaddr = re.findall(r'(?:\\d{1,3}\\.)+(?:\\d{1,3})', line)\n if ipaddr:\n ipaddr = ipaddr[0]\n if re.search(ip_ignore, ipaddr):\n logger.info('%s is in IP ignore list.', ipaddr)\n else:\n try:\n ipaddress.ip_address(ipaddr).is_global\n scan_ipaddr(ipaddr, line, project, split_path, apikey)\n except:\n pass # Ok to pass here, as this captures the odd string which is nt an IP Address\n\n # Check for URLs and send for report to Virus Total\n if urls:\n url = re.search(\"(?Phttps?://[^\\s]+)\", line) or re.search(\"(?Pwww[^\\s]+)\", line)\n if url:\n url = url.group(\"url\")\n if re.search(url_ignore, url):\n logger.info('%s is in URL ignore list.', url)\n else:\n scan_url(url, line, project, split_path, apikey)\n\n # Check flagged content in files\n for key, value in flag_list.items():\n regex = value['regex']\n desc = value['desc']\n if re.search(regex, line) and not re.search(ignore_list, line):\n logger.error('File contains violation: %s', full_path)\n logger.error('Flagged Content: %s', line.rstrip())\n logger.error('Matched Regular Exp: %s', regex)\n logger.error('Rationale: %s', desc.rstrip())\n with open(reports_dir + \"contents-\" + project + \".log\", \"a\") as gate_report:\n gate_report.write('File contains violation: {0}\\n'.format(full_path))\n gate_report.write('Flagged Content: {0}'.format(line))\n gate_report.write('Matched Regular Exp: {0}'.format(regex))\n gate_report.write('Rationale: {0}\\n'.format(desc.rstrip()))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef scan_ipaddr(ipaddr, line, project, split_path, apikey):\n logger.info('Found what I believe is an IP Address: %s', line.strip())\n logger.info('File %s. Parsed IP Address: %s', split_path, ipaddr)\n with open(reports_dir + \"ips-\" + project + \".log\", \"a\") as gate_report:\n gate_report.write('File {} contains what I believe is an IP Address: {}\\n'.format(split_path, ipaddr))\n v_api = virus_total.VirusTotal()\n scan_ip = v_api.send_ip(ipaddr, apikey)\n response_code = scan_ip['response_code']\n verbose_msg = scan_ip['verbose_msg']\n urls = scan_ip['detected_urls']\n with open(reports_dir + \"ips-\" + project + \".log\", \"a\") as gate_report:\n if urls:\n logger.error('%s has been known to resolve to the following malicious urls:', ipaddr)\n gate_report.write('{} has been known to resolve to the following malicious urls:\\n'.format(ipaddr))\n\n for url in urls:\n logger.info('%s on date: %s', url['url'], url['scan_date'])\n gate_report.write('{} on {}\\n'.format(url['url'], url['scan_date']))\n sleep(0.2)\n else:\n logger.info('No malicious DNS history found for: %s', ipaddr)\n gate_report.write('No malicious DNS history found for: {}\\n'.format(ipaddr))", "response": "Scan the VirusTotal API for a single IP Address."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nscanning a URL and return a list of the VirusTotal objects.", "response": "def scan_url(url, line, project, split_path, apikey):\n \"\"\"\n If URL is found, scan it\n \"\"\"\n logger.info('File %s contains what I believe is a URL: %s', split_path, line.strip())\n logger.info('Scanning: %s', url)\n with open(reports_dir + \"urls-\" + project + \".log\", \"a\") as gate_report:\n gate_report.write('File {} contains what I believe is a URL: {}\\n'.format(split_path, url))\n\n v_api = virus_total.VirusTotal()\n while True:\n url_report = v_api.url_report(url, apikey)\n response_code = url_report['response_code']\n\n # report does not exist, need to scan\n if response_code == -2:\n logger.info('Report job still queued..')\n\n if response_code == 0:\n logger.info('No report for %s', url)\n break\n\n if response_code == 1:\n logger.info('Report found, job complete for %s.', url)\n break\n\n try:\n positives = url_report['positives']\n if positives > 0:\n for site, results in url_report['scans'].items():\n printed = False\n if results['detected']:\n with open(reports_dir + \"urls-\" + project + \".log\", \"a\") as gate_report:\n logger.error(\"%s is recorded as a %s by %s\", url, results['result'], site)\n gate_report.write('{} is recorded as a {} by {}\\n'.format(url, results['result'], site))\n if not printed:\n printed = True\n with open(reports_dir + \"urls-\" + project + \".log\", \"a\") as gate_report:\n logger.error(\"Full report available here: %s\", url_report['permalink'])\n gate_report.write('Full report available here: {}\\n'.format(url_report['permalink']))\n\n else:\n logger.info(\"%s is recorded as a clean\", url)\n with open(reports_dir + \"urls-\" + project + \".log\", \"a\") as gate_report:\n gate_report.write('{} is recorded as a clean\\n'.format(url))\n\n except:\n # No positives so we can pass this\n pass"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef getIndex(folderPath=None):\n try: return cache.structure\n except AttributeError: pass # if it doesn't exist, generate and cache the map file data\n if folderPath == None:\n from sc2maptool.startup import setup\n folderPath = setup()\n ############################################################################\n def folderSearch(path, attrList=[]):\n ret = []\n for item in glob(os.path.join(path, '*')):\n if item == os.sep: continue\n itemName = os.path.basename(item)\n if os.path.isdir(item): ret += folderSearch(item, attrList + [itemName])\n elif itemName.endswith(c.SC2_MAP_EXT): ret.append( MapRecord(itemName, item, attrList) )\n return ret\n ############################################################################\n cache.structure = folderSearch(folderPath)\n return cache.structure", "response": "parse the Maps subfolder directory divining criteria for valid maps"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _stringify_number(v):\n '''\n Stringify a number, preventing unwanted scientific notations.\n '''\n if isinstance(v, (float, Decimal)):\n if math.isinf(v) and v > 0:\n v = 'Infinity'\n elif math.isinf(v) and v < 0:\n v = '-Infinity'\n else:\n v = '{:f}'.format(v)\n elif isinstance(v, BinarySize):\n v = '{:d}'.format(int(v))\n elif isinstance(v, int):\n v = '{:d}'.format(v)\n else:\n v = str(v)\n return v", "response": "Stringify a number preventing unwanted scientific notations."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\nasync def resolve_alias(cls, alias_key: str, etcd: etcd.AsyncEtcd):\n '''\n Resolve the tag using etcd so that the current instance indicates\n a concrete, latest image.\n\n Note that alias resolving does not take the registry component into\n account.\n '''\n alias_target = None\n repeats = 0\n while repeats < 8:\n prev_alias_key = alias_key\n alias_key = await etcd.get(f'images/_aliases/{alias_key}')\n if alias_key is None:\n alias_target = prev_alias_key\n break\n repeats += 1\n else:\n raise AliasResolutionFailed('Could not resolve the given image name!')\n known_registries = await get_known_registries(etcd)\n return cls(alias_target, known_registries)", "response": "Resolve the tag using etcd so that the current instance indicates that the given image name is a concrete latest image."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets the list of categories for a given curie.", "response": "def getOntoCategory(curie, alwaysFetch=False):\n \"\"\"\n Accessing web-based ontology service is too long, so we cache the \n information in a pickle file and query the services only if the info\n has not already been cached. \n \"\"\"\n \n fileName = os.path.join(os.path.dirname(__file__), \"ontoCategories.bin\") \n if not alwaysFetch:\n try:\n with open(fileName, \"rb\") as catFile:\n ontoCat = pickle.load(catFile)\n \n if curie in ontoCat:\n return ontoCat[curie] \n except:\n ontoCat = {}\n \n base = bases[\"KS\"] \n query = base + \"/vocabulary/id/\" + curie\n response = requests.get(query)\n if not response.ok:\n ontoCat[curie] = []\n else:\n\n try:\n concepts = response.json()\n except ValueError:\n print(query)\n print(response)\n raise\n \n if len(concepts[\"categories\"]):\n ontoCat[curie] = concepts[\"categories\"]\n else:\n ontoCat[curie] = []\n\n\n try:\n with open(fileName, \"wb\") as catFile:\n pickle.dump(ontoCat, catFile)\n except:\n pass \n\n return ontoCat[curie]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _init_logging(anteater_log):\n\n LOG.setLevel(logging.DEBUG)\n ch = logging.StreamHandler()\n formatter = logging.Formatter('%(asctime)s - %(name)s - '\n '%(levelname)s - %(message)s')\n ch.setFormatter(formatter)\n ch.setLevel(logging.DEBUG)\n\n # create the directory if it does not exist\n path = os.path.dirname(anteater_log)\n try:\n os.makedirs(path)\n except OSError as e:\n if e.errno != errno.EEXIST:\n raise\n\n handler = logging.FileHandler(anteater_log)\n handler.setFormatter(formatter)\n handler.setLevel(logging.DEBUG)\n del logging.root.handlers[:]\n logging.root.addHandler(ch)\n logging.root.addHandler(handler)", "response": "Setup root logger for the AETTLE package"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate a directory for scan reports", "response": "def check_dir():\n \"\"\" Creates a directory for scan reports \"\"\"\n try:\n os.makedirs(reports_dir)\n logger.info('Creating reports directory: %s', reports_dir)\n except OSError as e:\n if e.errno != errno.EEXIST:\n raise"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fit(model_code, *args, **kwargs):\n kwargs = dict(kwargs)\n kwargs['model_code'] = model_code\n if 'n_jobs' not in kwargs:\n kwargs['n_jobs'] = -1\n if model_code in FIT_CACHE:\n print(\"Reusing model.\")\n kwargs['fit'] = FIT_CACHE[model_code]\n else:\n print(\"NOT reusing model.\")\n start = time.time()\n FIT_CACHE[model_code] = pystan.stan(*args, **kwargs)\n print(\"Ran in %0.3f sec.\" % (time.time() - start))\n return FIT_CACHE[model_code]", "response": "Fit a Stan model."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\niterates over all pairs of pairs of KDNode objects with distance rmax", "response": "def enumiter(self, other, rmax, bunch=100000):\n \"\"\" cross correlate with other, for all pairs\n closer than rmax, iterate.\n\n for r, i, j in A.enumiter(...):\n ...\n where r is the distance, i and j are the original\n input array index of the data.\n\n This uses a thread to convert from KDNode.enum.\n \"\"\"\n\n def feeder(process):\n self.enum(other, rmax, process, bunch)\n for r, i, j in makeiter(feeder):\n yield r, i, j"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef enum(self, other, rmax, process=None, bunch=100000, **kwargs):\n rall = None\n if process is None:\n rall = [numpy.empty(0, 'f8')]\n iall = [numpy.empty(0, 'intp')]\n jall = [numpy.empty(0, 'intp')]\n def process(r1, i1, j1, **kwargs):\n rall[0] = numpy.append(rall[0], r1)\n iall[0] = numpy.append(iall[0], i1)\n jall[0] = numpy.append(jall[0], j1)\n\n _core.KDNode.enum(self, other, rmax, process, bunch, **kwargs)\n\n if rall is not None:\n return rall[0], iall[0], jall[0]\n else:\n return None", "response": "cross correlate with other for all pairs\n closer than rmax and iterate."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef count(self, other, r, attrs=None, info={}):\n\n r = numpy.array(r, dtype='f8')\n\n return _core.KDNode.count(self, other, r, attrs, info=info)", "response": "Gray & Moore based fast dual tree counting."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef integrate(self, min, max, attr=None, info={}):\n if numpy.isscalar(min):\n min = [min for i in range(self.ndims)]\n if numpy.isscalar(max):\n max = [max for i in range(self.ndims)]\n\n min = numpy.array(min, dtype='f8', order='C')\n max = numpy.array(max, dtype='f8', order='C')\n\n if (min).shape[-1] != self.ndims:\n raise ValueError(\"dimension of min does not match Node\")\n if (max).shape[-1] != self.ndims:\n raise ValueError(\"dimension of max does not match Node\")\n min, max = broadcast_arrays(min, max)\n return _core.KDNode.integrate(self, min, max, attr, info)", "response": "Integrate the number of points between min and max."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndivide a tree branch to a forest each subtree of size at most chunksize", "response": "def make_forest(self, chunksize):\n \"\"\" Divide a tree branch to a forest, \n each subtree of size at most chunksize \"\"\"\n heap = []\n heappush(heap, (-self.size, self))\n while True:\n w, x = heappop(heap)\n if w == 0: \n heappush(heap, (0, x))\n break\n if x.less is None \\\n or (x.size < chunksize):\n heappush(heap, (0, x))\n continue\n heappush(heap, (x.less.size, x.less))\n heappush(heap, (x.greater.size, x.greater))\n for w, x in heap:\n yield x"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ninstall a third party odoo add - on.", "response": "def _install_one(\n repo_url, branch, destination, commit='', patches=None,\n exclude_modules=None, include_modules=None, base=False, work_directory=''\n):\n \"\"\" Install a third party odoo add-on\n\n :param string repo_url: url of the repo that contains the patch.\n :param string branch: name of the branch to checkout.\n :param string destination: the folder where the add-on should end up at.\n :param string commit: Optional commit rev to checkout to. If mentioned, that take over the branch\n :param string work_directory: the path to the directory of the yaml file.\n :param list patches: Optional list of patches to apply.\n \"\"\"\n patches = patches or []\n patches = [\n core.FilePatch(file=patch['file'], work_directory=work_directory)\n if 'file' in patch else core.Patch(**patch)\n for patch in patches\n ]\n addon_cls = core.Base if base else core.Addon\n addon = addon_cls(\n repo_url, branch, commit=commit, patches=patches,\n exclude_modules=exclude_modules, include_modules=include_modules)\n addon.install(destination)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ninstalling all the third party add -ons.", "response": "def _install_all(destination='', conf_file=''):\n \"\"\"Use the conf file to list all the third party Odoo add-ons that will be installed\n and the patches that should be applied.\n\n :param string destination: the folder where add-ons should end up at.\n Default: pwd/3rd\n :param string conf_file: path to a conf file that describe the add-ons to install.\n Default: pwd/third_party_addons.yaml\n \"\"\"\n dir_path = os.path.dirname(os.path.realpath(__file__))\n destination = destination or os.path.join(dir_path, '..', '3rd')\n conf_file = conf_file or os.path.join(dir_path, '..', \"third_party_addons.yaml\")\n work_directory = os.path.dirname(os.path.realpath(conf_file))\n\n with open(conf_file, \"r\") as conf_data:\n data = yaml.load(conf_data)\n for addons in data:\n _install_one(\n addons['url'],\n addons['branch'],\n os.path.abspath(destination),\n commit=addons.get('commit'),\n patches=addons.get('patches'),\n exclude_modules=addons.get('excludes'),\n include_modules=addons.get('includes'),\n base=addons.get('base'),\n work_directory=work_directory,\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef find_lt(a, x):\n i = bisect.bisect_left(a, x)\n if i:\n return a[i-1]\n raise ValueError", "response": "Find rightmost value less than x"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef parse(isatab_ref):\n if os.path.isdir(isatab_ref):\n fnames = glob.glob(os.path.join(isatab_ref, \"i_*.txt\")) + \\\n glob.glob(os.path.join(isatab_ref, \"*.idf.txt\"))\n assert len(fnames) == 1\n isatab_ref = fnames[0]\n assert os.path.exists(isatab_ref), \"Did not find investigation file: %s\" % isatab_ref\n i_parser = InvestigationParser()\n with open(isatab_ref, \"rU\") as in_handle:\n rec = i_parser.parse(in_handle)\n s_parser = StudyAssayParser(isatab_ref)\n rec = s_parser.parse(rec)\n return rec", "response": "Entry point to parse an ISA - Tab directory."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _parse_region(self, rec, line_iter):\n had_info = False\n keyvals, section = self._parse_keyvals(line_iter)\n\n if keyvals:\n rec.metadata = keyvals[0]\n while section and section[0] != \"STUDY\":\n had_info = True\n keyvals, next_section = self._parse_keyvals(line_iter)\n attr_name = self._sections[section[0]]\n if attr_name in self._nolist:\n try:\n keyvals = keyvals[0]\n except IndexError:\n keyvals = {}\n\n setattr(rec, attr_name, keyvals)\n section = next_section\n return rec, had_info", "response": "Parse a section of an ISA - Tab and assign information to a supplied record."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\niterates over the line of ISA - Tab delimited file.", "response": "def _line_iter(self, in_handle):\n \"\"\"Read tab delimited file, handling ISA-Tab special case headers.\n \"\"\"\n reader = csv.reader(in_handle, dialect=\"excel-tab\")\n for line in reader:\n if len(line) > 0 and line[0]:\n # check for section headers; all uppercase and a single value\n if line[0].upper() == line[0] and \"\".join(line[1:]) == \"\":\n line = [line[0]]\n yield line"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _parse_keyvals(self, line_iter):\n out = None\n line = None\n for line in line_iter:\n if len(line) == 1 and line[0].upper() == line[0]:\n break\n else:\n # setup output dictionaries, trimming off blank columns\n if out is None:\n while not line[-1]:\n line = line[:-1]\n out = [{} for _ in line[1:]]\n # add blank values if the line is stripped\n while len(line) < len(out) + 1:\n line.append(\"\")\n for i in range(len(out)):\n out[i][line[0]] = line[i+1].strip()\n line = None\n return out, line", "response": "Generate dictionary from key - value pairs."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef parse(self, rec):\n final_studies = []\n for study in rec.studies:\n source_data = self._parse_study(study.metadata[\"Study File Name\"],\n [\"Source Name\", \"Sample Name\", \"Comment[ENA_SAMPLE]\"])\n if source_data:\n study.nodes = source_data\n final_assays = []\n for assay in study.assays:\n cur_assay = ISATabAssayRecord(assay)\n assay_data = self._parse_study(assay[\"Study Assay File Name\"],\n [\"Sample Name\",\"Extract Name\",\"Raw Data File\",\"Derived Data File\", \"Image File\", \"Acquisition Parameter Data File\", \"Free Induction Decay Data File\"])\n cur_assay.nodes = assay_data\n self._get_process_nodes(assay[\"Study Assay File Name\"], cur_assay)\n final_assays.append(cur_assay)\n study.assays = final_assays\n\n #get process nodes\n self._get_process_nodes(study.metadata[\"Study File Name\"], study)\n final_studies.append(study)\n rec.studies = final_studies\n return rec", "response": "Retrieve row data from files associated with the ISATabRecord."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nparse study or assay row oriented file around the supplied base node.", "response": "def _parse_study(self, fname, node_types):\n \"\"\"Parse study or assay row oriented file around the supplied base node.\n \"\"\"\n if not os.path.exists(os.path.join(self._dir, fname)):\n return None\n nodes = {}\n with open(os.path.join(self._dir, fname), \"rU\") as in_handle:\n reader = csv.reader(in_handle, dialect=\"excel-tab\")\n header = self._swap_synonyms(next(reader))\n hgroups = self._collapse_header(header)\n htypes = self._characterize_header(header, hgroups)\n\n for node_type in node_types:\n try:\n name_index = header.index(node_type)\n except ValueError:\n name_index = None\n\n if name_index is None:\n #print \"Could not find standard header name: %s in %s\" \\\n # % (node_type, header)\n continue\n\n in_handle.seek(0, 0)\n for line in reader:\n name = line[name_index]\n #to deal with same name used for different node types (e.g. Source Name and Sample Name using the same string)\n node_index = self._build_node_index(node_type,name)\n #skip the header line and empty lines\n if name in header:\n continue\n if (not name):\n continue\n try:\n node = nodes[node_index]\n except KeyError:\n #print(\"creating node \", name, \" index\", node_index)\n node = NodeRecord(name, node_type)\n node.metadata = collections.defaultdict(set)\n nodes[node_index] = node\n attrs = self._line_keyvals(line, header, hgroups, htypes, node.metadata)\n nodes[node_index].metadata = attrs\n\n return dict([(k, self._finalize_metadata(v)) for k, v in nodes.items()])"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting node metadata back into a standard dictionary and list.", "response": "def _finalize_metadata(self, node):\n \"\"\"Convert node metadata back into a standard dictionary and list.\n \"\"\"\n final = {}\n for key, val in iter(node.metadata.items()):\n #val = list(val)\n #if isinstance(val[0], tuple):\n # val = [dict(v) for v in val]\n final[key] = list(val)\n node.metadata = final\n return node"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _line_by_type(self, line, header, hgroups, htypes, out, want_type,\n collapse_quals_fn = None):\n \"\"\"Parse out key value pairs for line information based on a group of values.\n \"\"\"\n for index, htype in ((i, t) for i, t in enumerate(htypes) if t == want_type):\n col = hgroups[index][0]\n key = header[col]#self._clean_header(header[col])\n if collapse_quals_fn:\n val = collapse_quals_fn(line, header, hgroups[index])\n else:\n val = line[col]\n out[key].add(val)\n return out", "response": "Parse out key value pairs based on a line of data."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncombining attributes in multiple columns into single named tuple.", "response": "def _collapse_attributes(self, line, header, indexes):\n \"\"\"Combine attributes in multiple columns into single named tuple.\n \"\"\"\n names = []\n vals = []\n pat = re.compile(\"[\\W]+\")\n for i in indexes:\n names.append(pat.sub(\"_\", self._clean_header(header[i])))\n vals.append(line[i])\n Attrs = collections.namedtuple('Attrs', names)\n return Attrs(*vals)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nremoves ISA - Tab specific information from Header [ real name ] headers.", "response": "def _clean_header(self, header):\n \"\"\"Remove ISA-Tab specific information from Header[real name] headers.\n \"\"\"\n if header.find(\"[\") >= 0:\n header = header.replace(\"]\", \"\").split(\"[\")[-1]\n # ISATab can start with numbers but this is not supported in\n # the python datastructure, so prefix with isa_ to make legal\n try:\n int(header[0])\n header = \"isa_\" + header\n except ValueError:\n pass\n return header"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncharacterize header groups into different data types.", "response": "def _characterize_header(self, header, hgroups):\n \"\"\"Characterize header groups into different data types.\n \"\"\"\n out = []\n for h in [header[g[0]] for g in hgroups]:\n this_ctype = None\n for ctype, names in self._col_types.items():\n if h.startswith(names):\n this_ctype = ctype\n break\n out.append(this_ctype)\n return out"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _collapse_header(self, header):\n out = []\n for i, h in enumerate(header):\n if h.startswith(self._col_quals):\n out[-1].append(i)\n else:\n out.append([i])\n return out", "response": "Combine header columns into related groups."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nnaming mapInfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo - getmapinfo", "response": "def main(): # mini/unit test\n \"\"\"\n PURPOSE: command-line interface for map information\n \"\"\"\n options = optionsParser().parse_args()\n params = getSelectionParams(options)\n if options.list or options.details:\n specifiedMaps = filterMapNames(\n options.mapname,\n records = filterMapAttrs(**params),\n excludeRegex=options.exclude,\n closestMatch=options.best\n )\n if specifiedMaps:\n for v in specifiedMaps:\n if options.details: v.display()\n else: print(v)\n print(\"Found %d maps that match given criteria.\"%(len(specifiedMaps)))\n else:\n print(\"No matching maps found.\")\n else:\n try:\n specifiedMaps = selectMap(\n options.mapname,\n excludeName =options.exclude,\n closestMatch=options.best,\n **params)\n except Exception as e:\n specifiedMaps = []\n print(\"No matching maps found: %s\"%e)\n if not isinstance(specifiedMaps, list):\n specifiedMaps = [specifiedMaps]\n for m in specifiedMaps:\n if options.path: print(m.path)\n else: print(m.name)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef is_pickle_file(abspath):\n abspath = abspath.lower()\n fname, ext = os.path.splitext(abspath)\n if ext in [\".pickle\", \".pk\", \".p\"]:\n is_pickle = True\n elif ext == \".gz\":\n is_pickle = False\n elif ext == \".tmp\":\n return is_pickle_file(fname)\n else:\n raise PickleExtError(\n \"'%s' is not a valid pickle file. \"\n \"extension has to be '.pickle' for uncompressed, '.gz' \"\n \"for compressed.\" % abspath)\n return is_pickle", "response": "Parse file extension.\n \n - *.pickle: uncompressed, utf-8 encode pickle file\n - *.gz: compressed, utf-8 encode pickle file"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef load(abspath, default=None, enable_verbose=True):\n if default is None:\n default = dict()\n\n prt(\"\\nLoad from '%s' ...\" % abspath, enable_verbose)\n\n abspath = lower_ext(str(abspath))\n is_pickle = is_pickle_file(abspath)\n\n if not os.path.exists(abspath):\n prt(\" File not found, use default value: %r\" % default,\n enable_verbose)\n return default\n\n st = time.clock()\n if is_pickle:\n data = pickle.loads(textfile.readbytes(abspath))\n else:\n data = pickle.loads(compress.read_gzip(abspath))\n\n prt(\" Complete! Elapse %.6f sec.\" % (time.clock() - st), enable_verbose)\n return data", "response": "Load a pickle from file."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndumping picklable object to file.", "response": "def dump(data, abspath, pk_protocol=py23.pk_protocol,\n overwrite=False, enable_verbose=True):\n \"\"\"Dump picklable object to file.\n Provides multiple choice to customize the behavior.\n\n :param data: picklable python object.\n :type data: dict or list\n\n :param abspath: ``save as`` path, file extension has to be ``.pickle`` or ``.gz``\n (for compressed Pickle)\n :type abspath: string\n\n :param pk_protocol: default = your python version, use 2, to make a\n py2.x/3.x compatible pickle file. But 3 is faster.\n :type pk_protocol: int\n\n :param overwrite: default ``False``, If ``True``, when you dump to existing\n file, it silently overwrite it. If ``False``, an alert message is shown.\n Default setting ``False`` is to prevent overwrite file by mistake.\n :type overwrite: boolean\n\n :param enable_verbose: default True, help-message-display trigger.\n :type enable_verbose: boolean\n\n Usage::\n\n >>> from dataIO import pk\n >>> data = {\"a\": 1, \"b\": 2}\n >>> dump(data, \"test.pickle\", overwrite=True)\n Dump to `test.pickle` ...\n Complete! Elapse 0.002432 sec\n\n **\u4e2d\u6587\u6587\u6863**\n\n \u5c06Python\u4e2d\u53ef\u88ab\u5e8f\u5217\u5316\u7684\"\u5b57\u5178\", \"\u5217\u8868\"\u4ee5\u53ca\u4ed6\u4eec\u7684\u7ec4\u5408, \u6309\u7167Json\u7684\u7f16\u7801\u65b9\u5f0f\u5199\u5165\u6587\u4ef6\n \u6587\u4ef6\n\n \u53c2\u6570\u5217\u8868\n\n :param data: \u53efPickle\u5316\u7684Python\u5bf9\u8c61\n :type data: ``\u5b57\u5178`` \u6216 ``\u5217\u8868``\n\n :param abspath: Pickle\u6587\u4ef6\u7edd\u5bf9\u8def\u5f84, \u6269\u5c55\u540d\u9700\u4e3a ``.pickle`` \u6216 ``.gz``, \u5176\u4e2d ``.gz``\n \u662f\u88ab\u538b\u7f29\u540e\u7684Pickle\u6587\u4ef6\n :type abspath: ``\u5b57\u7b26\u4e32``\n\n :param pk_protocol: \u9ed8\u8ba4\u503c\u4e3a\u4f60\u7684Python\u5927\u7248\u672c\u53f7, \u4f7f\u75282\u53ef\u4ee5\u4f7f\u5f97Python2/3\u90fd\u80fd\n \u517c\u5bb9\u4f60\u7684Pickle\u6587\u4ef6\u3002\u4e0d\u8fc7Python3\u7684\u901f\u5ea6\u66f4\u5feb\u3002\n :type pk_protocol: int\n\n :param overwrite: \u9ed8\u8ba4 ``False``, \u5f53\u4e3a``True``\u65f6, \u5982\u679c\u5199\u5165\u8def\u5f84\u5df2\u7ecf\u5b58\u5728, \u5219\u4f1a\n \u81ea\u52a8\u8986\u76d6\u539f\u6587\u4ef6\u3002\u800c\u4e3a``False``\u65f6, \u5219\u4f1a\u6253\u5370\u8b66\u544a\u6587\u4ef6, \u9632\u6b62\u8bef\u64cd\u4f5c\u8986\u76d6\u6e90\u6587\u4ef6\u3002\n :type overwrite: \"\u5e03\u5c14\u503c\"\n\n :param enable_verbose: \u9ed8\u8ba4 ``True``, \u4fe1\u606f\u63d0\u793a\u7684\u5f00\u5173, \u6279\u5904\u7406\u65f6\u5efa\u8bae\u5173\u95ed\n :type enable_verbose: ``\u5e03\u5c14\u503c``\n \"\"\"\n prt(\"\\nDump to '%s' ...\" % abspath, enable_verbose)\n\n abspath = lower_ext(str(abspath))\n is_pickle = is_pickle_file(abspath)\n\n if os.path.exists(abspath):\n if not overwrite: # \u5b58\u5728, \u5e76\u4e14overwrite=False\n prt(\" Stop! File exists and overwrite is not allowed\",\n enable_verbose)\n return\n\n st = time.clock()\n content = pickle.dumps(data, pk_protocol)\n if is_pickle:\n textfile.writebytes(content, abspath)\n else:\n compress.write_gzip(content, abspath)\n\n prt(\" Complete! Elapse %.6f sec.\" % (time.clock() - st), enable_verbose)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef safe_dump(data, abspath, pk_protocol=py23.pk_protocol, enable_verbose=True):\n abspath = lower_ext(str(abspath))\n abspath_temp = \"%s.tmp\" % abspath\n dump(data, abspath_temp,\n pk_protocol=pk_protocol, enable_verbose=enable_verbose)\n shutil.move(abspath_temp, abspath)", "response": "A stable version of pickle."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef obj2bytes(obj, pk_protocol=py23.pk_protocol):\n return pickle.dumps(obj, protocol=pk_protocol)", "response": "Convert arbitrary pickable Python Object to bytes."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef obj2str(obj, pk_protocol=py23.pk_protocol):\n return base64.urlsafe_b64encode(pickle.dumps(\n obj, protocol=pk_protocol)).decode(\"utf-8\")", "response": "Convert arbitrary object to base64 encoded string."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreset the users network to make changes take effect", "response": "def reset_network(message):\n \"\"\"Resets the users network to make changes take effect\"\"\"\n for command in settings.RESTART_NETWORK:\n try:\n subprocess.check_call(command)\n except:\n pass\n print(message)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef improve():\n with open(settings.HOSTS_FILE, \"r+\") as hosts_file:\n contents = hosts_file.read()\n if not settings.START_TOKEN in contents and not settings.END_TOKEN in contents:\n hosts_file.write(settings.START_TOKEN + \"\\n\")\n for site in set(settings.DISTRACTORS):\n hosts_file.write(\"{0}\\t{1}\\n\".format(settings.REDIRECT_TO, site))\n for sub_domain in settings.SUB_DOMAINS:\n hosts_file.write(\"{0}\\t{1}.{2}\\n\".format(settings.REDIRECT_TO, sub_domain, site))\n hosts_file.write(settings.END_TOKEN + \"\\n\")\n\n reset_network(\"Concentration is now improved :D!\")", "response": "Disables access to websites that are defined as distractors"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef lose():\n changed = False\n with open(settings.HOSTS_FILE, \"r\") as hosts_file:\n new_file = []\n in_block = False\n for line in hosts_file:\n if in_block:\n if line.strip() == settings.END_TOKEN:\n in_block = False\n changed = True\n elif line.strip() == settings.START_TOKEN:\n in_block = True\n else:\n new_file.append(line)\n if changed:\n with open(settings.HOSTS_FILE, \"w\") as hosts_file:\n hosts_file.write(\"\".join(new_file))\n\n reset_network(\"Concentration is now lost :(.\")", "response": "Enables access to websites that are defined as distractors"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef take_break(minutes: hug.types.number=5):\n print(\"\")\n print(\"######################################### ARE YOU SURE? #####################################\")\n try:\n for remaining in range(60, -1, -1):\n sys.stdout.write(\"\\r\")\n sys.stdout.write(\"{:2d} seconds to change your mind. Won't you prefer programming? Or a book?\".format(remaining))\n sys.stdout.flush()\n time.sleep(1)\n except KeyboardInterrupt:\n print(\"\")\n print(\"\")\n print(\":D :D :D\\nGood on you! <3\")\n return\n\n # The user insisted on breaking concentration.\n lose()\n print(\"\")\n print(\"######################################### TAKING A BREAK ####################################\")\n try:\n for remaining in range(minutes * 60, -1, -1):\n sys.stdout.write(\"\\r\")\n sys.stdout.write(\"{:2d} seconds remaining without concentration.\".format(remaining))\n sys.stdout.flush()\n time.sleep(1)\n except KeyboardInterrupt:\n pass\n finally:\n sys.stdout.write(\"\\rEnough distraction! \\n\")\n print(\"######################################### BREAK OVER :) #####################################\")\n print(\"\")\n improve()", "response": "Enables temporarily breaking concentration and performs some additional checks."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef local_port_range(self):\n if self._local_port_range is None:\n with open('/proc/sys/net/ipv4/ip_local_port_range', 'r') as f:\n self._local_port_range = tuple(map(int, f.read().split('\\t')))\n return self._local_port_range", "response": "Tuple of low_port high_port reflecting the local port range\n assigned to inbound connections."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _si(number):\n prefixes = ['', 'K', 'M', 'G', 'T', 'P', 'E', 'Z', 'Y']\n while number > 1024:\n number /= 1024.0\n prefixes.pop(0)\n return '%0.2f%s' % (number, prefixes.pop(0))", "response": "Format a number using base - 2 SI prefixes"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _extract_file(zip_fp, info, path):\n zip_fp.extract(info.filename, path=path)\n out_path = os.path.join(path, info.filename)\n\n perm = info.external_attr >> 16\n perm |= stat.S_IREAD # make sure we're not accidentally setting this to 0\n os.chmod(out_path, perm)", "response": "Extract files while explicitly setting the proper permissions"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nbuilding a string that can be used to build a Taskcluster entry.", "response": "def build_string(self):\n \"\"\"\n Taskcluster denotes builds in one of two formats - i.e. linux64-asan or linux64-asan-opt\n The latter is generated. If it fails, the caller should try the former.\n \"\"\"\n return (('-ccov' if self.coverage else '') +\n ('-fuzzing' if self.fuzzing else '') +\n ('-asan' if self.asan else '') +\n ('-valgrind' if self.valgrind else '') +\n ('-debug' if self.debug else '-opt'))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ngenerate platform prefix for cross - platform downloads.", "response": "def auto_name_prefix(self):\n \"\"\"\n Generate platform prefix for cross-platform downloads.\n \"\"\"\n # if the platform is not native, auto_name would clobber native downloads.\n # make a prefix to avoid this\n native_system = std_platform.system()\n native_machine = self.CPU_ALIASES.get(std_platform.machine(), std_platform.machine())\n if native_system == self.system and native_machine == self.machine:\n return ''\n platform = {\n 'linux': 'linux32',\n 'android-api-16': 'android-arm',\n 'android-aarch64': 'android-arm64',\n }.get(self.gecko_platform, self.gecko_platform)\n return platform + '-'"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nyields all URLs that are available for a given push date.", "response": "def _pushdate_urls(cls, pushdate, branch, target_platform):\n \"\"\"Multiple entries exist per push date. Iterate over all until a working entry is found\"\"\"\n url_base = cls.URL_BASE + '/namespaces/gecko.v2.mozilla-' + branch + '.pushdate.' + pushdate\n\n try:\n base = HTTP_SESSION.post(url_base, json={})\n base.raise_for_status()\n except requests.exceptions.RequestException as exc:\n raise FetcherException(exc)\n\n product = 'mobile' if 'android' in target_platform else 'firefox'\n json = base.json()\n for namespace in sorted(json['namespaces'], key=lambda x: x['name']):\n yield cls.URL_BASE + '/task/' + namespace['namespace'] + '.' + product + '.' + target_platform"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _revision_url(cls, rev, branch, target_platform):\n namespace = 'gecko.v2.mozilla-' + branch + '.revision.' + rev\n product = 'mobile' if 'android' in target_platform else 'firefox'\n return cls.URL_BASE + '/task/' + namespace + '.' + product + '.' + target_platform", "response": "Retrieve the URL for revision based builds"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef iterall(cls, target, branch, build, flags, platform=None):\n flags = BuildFlags(*flags)\n for task in BuildTask.iterall(build, branch, flags, platform):\n yield cls(target, branch, task, flags, platform)", "response": "Return an iterable for all available builds matching a particular build type"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nretrieving the artifacts json object", "response": "def _artifacts(self):\n \"\"\"Retrieve the artifacts json object\"\"\"\n if '_artifacts' not in self._memo:\n json = _get_url(self._artifacts_url).json()\n self._memo['_artifacts'] = json['artifacts']\n return self._memo['_artifacts']"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _artifact_base(self):\n if '_artifact_base' not in self._memo:\n for artifact in self._artifacts:\n if self.re_target.search(artifact['name']) is not None:\n artifact_base = os.path.splitext(artifact['name'])[0]\n break\n else:\n raise FetcherException('Could not find build info in artifacts')\n self._memo['_artifact_base'] = artifact_base\n return self._memo['_artifact_base']", "response": "Return the base build name for the artifact"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef build_info(self):\n if 'build_info' not in self._memo:\n self._memo['build_info'] = _get_url(self.artifact_url('json')).json()\n return self._memo['build_info']", "response": "Return the build s info"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef moz_info(self):\n if 'moz_info' not in self._memo:\n self._memo['moz_info'] = _get_url(self.artifact_url('mozinfo.json')).json()\n return self._memo['moz_info']", "response": "Return the build s mozinfo json"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndownload and extract the build and requested extra artifacts.", "response": "def extract_build(self, path='.', tests=None, full_symbols=False):\n \"\"\"\n Download and extract the build and requested extra artifacts\n\n @type path:\n @param path:\n\n @type tests:\n @param tests:\n\n @type full_symbols:\n @param full_symbols:\n \"\"\"\n if self._target == 'js':\n self.extract_zip('jsshell.zip', path=os.path.join(path))\n else:\n if self._platform.system == 'Linux':\n self.extract_tar('tar.bz2', path)\n elif self._platform.system == 'Darwin':\n self.extract_dmg(path)\n elif self._platform.system == 'Windows':\n self.extract_zip('zip', path)\n # windows builds are extracted under 'firefox/'\n # move everything under firefox/ up a level to the destination path\n firefox = os.path.join(path, 'firefox')\n for root, dirs, files in os.walk(firefox):\n newroot = root.replace(firefox, path)\n for dirname in dirs:\n os.mkdir(os.path.join(newroot, dirname))\n for filename in files:\n os.rename(os.path.join(root, filename), os.path.join(newroot, filename))\n shutil.rmtree(firefox, onerror=onerror)\n elif self._platform.system == 'Android':\n self.download_apk(path)\n else:\n raise FetcherException(\"'%s' is not a supported platform\" % self._platform.system)\n\n if tests:\n # validate tests\n tests = set(tests or [])\n if not tests.issubset(self.TEST_CHOICES):\n invalid_test = tuple(tests - self.TEST_CHOICES)[0]\n raise FetcherException(\"'%s' is not a supported test type\" % invalid_test)\n\n os.mkdir(os.path.join(path, 'tests'))\n if 'common' in tests:\n try:\n self.extract_tar('common.tests.tar.gz', path=os.path.join(path, 'tests'))\n except FetcherException:\n self.extract_zip('common.tests.zip', path=os.path.join(path, 'tests'))\n if 'reftests' in tests:\n try:\n self.extract_tar('reftest.tests.tar.gz', path=os.path.join(path, 'tests'))\n except FetcherException:\n self.extract_zip('reftest.tests.zip', path=os.path.join(path, 'tests'))\n if 'gtest' in tests:\n try:\n self.extract_tar('gtest.tests.tar.gz', path=path)\n except FetcherException:\n self.extract_zip('gtest.tests.zip', path=path)\n if self._platform.system == 'Windows':\n libxul = 'xul.dll'\n elif self._platform.system == 'Linux':\n libxul = 'libxul.so'\n elif self._platform.system == 'Darwin':\n libxul = 'XUL'\n else:\n raise FetcherException(\"'%s' is not a supported platform for gtest\" % self._platform.system)\n os.rename(os.path.join(path, 'gtest', 'gtest_bin', 'gtest', libxul),\n os.path.join(path, 'gtest', libxul))\n shutil.copy(os.path.join(path, 'gtest', 'dependentlibs.list.gtest'),\n os.path.join(path, 'dependentlibs.list.gtest'))\n if self._flags.coverage:\n self.extract_zip('code-coverage-gcno.zip', path=path)\n\n if not self._flags.asan and not self._flags.valgrind:\n if full_symbols:\n symbols = 'crashreporter-symbols-full.zip'\n else:\n symbols = 'crashreporter-symbols.zip'\n os.mkdir(os.path.join(path, 'symbols'))\n self.extract_zip(symbols, path=os.path.join(path, 'symbols'))\n\n self._layout_for_domfuzz(path)\n self._write_fuzzmanagerconf(path)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nupdating directory to work with DOMFuzz @type path: str @param path: A string representation of the fuzzmanager config path", "response": "def _layout_for_domfuzz(self, path):\n \"\"\"\n Update directory to work with DOMFuzz\n\n @type path: str\n @param path: A string representation of the fuzzmanager config path\n \"\"\"\n old_dir = os.getcwd()\n os.chdir(os.path.join(path))\n try:\n os.mkdir('dist')\n link_name = os.path.join('dist', 'bin')\n if self._platform.system == 'Darwin' and self._target == 'firefox':\n ff_loc = glob.glob('*.app/Contents/MacOS/firefox')\n assert len(ff_loc) == 1\n os.symlink(os.path.join(os.pardir, os.path.dirname(ff_loc[0])), # pylint: disable=no-member\n link_name)\n os.symlink(os.path.join(os.pardir, os.pardir, os.pardir, 'symbols'), # pylint: disable=no-member\n os.path.join(os.path.dirname(ff_loc[0]), 'symbols'))\n elif self._platform.system == 'Linux':\n os.symlink(os.pardir, link_name) # pylint: disable=no-member\n elif self._platform.system == 'Windows':\n # create a junction point at dist\\bin pointing to the firefox.exe path\n junction_path.symlink(os.curdir, link_name)\n finally:\n os.chdir(old_dir)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nwriting fuzzmanager config file for selected build_id.", "response": "def _write_fuzzmanagerconf(self, path):\n \"\"\"\n Write fuzzmanager config file for selected build\n\n @type path: basestring\n @param path: A string representation of the fuzzmanager config path\n \"\"\"\n output = configparser.RawConfigParser()\n output.add_section('Main')\n output.set('Main', 'platform', self.moz_info['processor'].replace('_', '-'))\n output.set('Main', 'product', 'mozilla-' + self._branch)\n output.set('Main', 'product_version', '%.8s-%.12s' % (self.build_id, self.changeset))\n # make sure 'os' match what FM expects\n os_name = self.moz_info['os'].lower()\n if os_name.startswith('android'):\n output.set('Main', 'os', 'android')\n elif os_name.startswith('lin'):\n output.set('Main', 'os', 'linux')\n elif os_name.startswith('mac'):\n output.set('Main', 'os', 'macosx')\n elif os_name.startswith('win'):\n output.set('Main', 'os', 'windows')\n else:\n output.set('Main', 'os', self.moz_info['os'])\n output.add_section('Metadata')\n output.set('Metadata', 'pathPrefix', self.moz_info['topsrcdir'])\n output.set('Metadata', 'buildFlags', self._flags.build_string().lstrip('-'))\n\n if self._platform.system == \"Windows\":\n fm_name = self._target + '.exe.fuzzmanagerconf'\n conf_path = os.path.join(path, 'dist', 'bin', fm_name)\n elif self._platform.system == \"Android\":\n conf_path = os.path.join(path, 'target.apk.fuzzmanagerconf')\n else:\n fm_name = self._target + '.fuzzmanagerconf'\n conf_path = os.path.join(path, 'dist', 'bin', fm_name)\n with open(conf_path, 'w') as conf_fp:\n output.write(conf_fp)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndownloading and extract a zip artifact .", "response": "def extract_zip(self, suffix, path='.'):\n \"\"\"\n Download and extract a zip artifact\n\n @type suffix:\n @param suffix:\n\n @type path:\n @param path:\n \"\"\"\n zip_fd, zip_fn = tempfile.mkstemp(prefix='fuzzfetch-', suffix='.zip')\n os.close(zip_fd)\n try:\n _download_url(self.artifact_url(suffix), zip_fn)\n LOG.info('.. extracting')\n with zipfile.ZipFile(zip_fn) as zip_fp:\n for info in zip_fp.infolist():\n _extract_file(zip_fp, info, path)\n finally:\n os.unlink(zip_fn)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef extract_tar(self, suffix, path='.'):\n mode = suffix.split('.')[-1]\n tar_fd, tar_fn = tempfile.mkstemp(prefix='fuzzfetch-', suffix='.tar.%s' % mode)\n os.close(tar_fd)\n try:\n _download_url(self.artifact_url(suffix), tar_fn)\n LOG.info('.. extracting')\n with tarfile.open(tar_fn, mode='r:%s' % mode) as tar:\n members = []\n for member in tar.getmembers():\n if member.path.startswith(\"firefox/\"):\n member.path = member.path[8:]\n members.append(member)\n elif member.path != \"firefox\":\n # Ignore top-level build directory\n members.append(member)\n tar.extractall(members=members, path=path)\n finally:\n os.unlink(tar_fn)", "response": "Extract builds with. tar.(*) extension\n When unpacking a build archive only extract the firefox directory\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef download_apk(self, path='.'):\n apk_fd, apk_fn = tempfile.mkstemp(prefix='fuzzfetch-', suffix='.apk')\n os.close(apk_fd)\n try:\n _download_url(self.artifact_url('apk'), apk_fn)\n shutil.copy(apk_fn, os.path.join(path, 'target.apk'))\n finally:\n os.unlink(apk_fn)", "response": "Download Android. apk file and copy it to path."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef extract_dmg(self, path='.'):\n dmg_fd, dmg_fn = tempfile.mkstemp(prefix='fuzzfetch-', suffix='.dmg')\n os.close(dmg_fd)\n out_tmp = tempfile.mkdtemp(prefix='fuzzfetch-', suffix='.tmp')\n try:\n _download_url(self.artifact_url('dmg'), dmg_fn)\n if std_platform.system() == 'Darwin':\n LOG.info('.. extracting')\n subprocess.check_call(['hdiutil', 'attach', '-quiet', '-mountpoint', out_tmp, dmg_fn])\n try:\n apps = [mt for mt in os.listdir(out_tmp) if mt.endswith('app')]\n assert len(apps) == 1\n shutil.copytree(os.path.join(out_tmp, apps[0]), os.path.join(path, apps[0]), symlinks=True)\n finally:\n subprocess.check_call(['hdiutil', 'detach', '-quiet', out_tmp])\n else:\n LOG.warning('.. can\\'t extract target.dmg on %s', std_platform.system())\n shutil.copy(dmg_fn, os.path.join(path, 'target.dmg'))\n finally:\n shutil.rmtree(out_tmp, onerror=onerror)\n os.unlink(dmg_fn)", "response": "Extract builds with. dmg extension."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconstruct a Fetcher object from given command line arguments.", "response": "def from_args(cls, args=None, skip_dir_check=False):\n \"\"\"\n Construct a Fetcher from given command line arguments.\n\n @type args: list(str)\n @param args: Command line arguments (optional). Default is to use args from sys.argv\n\n @type skip_dir_check: bool\n @param skip_dir_check: Boolean identifying whether to check for existing build directory\n\n @rtype: tuple(Fetcher, output path)\n @return: Returns a Fetcher object and keyword arguments for extract_build.\n \"\"\"\n parser = argparse.ArgumentParser()\n parser.set_defaults(target='firefox', build='latest', tests=None) # branch default is set after parsing\n\n target_group = parser.add_argument_group('Target')\n target_group.add_argument('--target', choices=sorted(cls.TARGET_CHOICES),\n help=('Specify the build target. (default: %(default)s)'))\n target_group.add_argument('--os', choices=sorted(Platform.SUPPORTED),\n help=('Specify the target system. (default: ' + std_platform.system() + ')'))\n cpu_choices = sorted(set(itertools.chain(itertools.chain.from_iterable(Platform.SUPPORTED.values()),\n Platform.CPU_ALIASES)))\n target_group.add_argument('--cpu', choices=cpu_choices,\n help=('Specify the target CPU. (default: ' + std_platform.machine() + ')'))\n\n type_group = parser.add_argument_group('Build')\n type_group.add_argument('--build', metavar='DATE|REV|NS',\n help='Specify the build to download, (default: %(default)s)'\n ' Accepts values in format YYYY-MM-DD (2017-01-01)'\n ' revision (57b37213d81150642f5139764e7044b07b9dccc3)'\n ' or TaskCluster namespace (gecko.v2....)')\n\n branch_group = parser.add_argument_group('Branch')\n branch_args = branch_group.add_mutually_exclusive_group()\n branch_args.add_argument('--inbound', action='store_const', const='inbound', dest='branch',\n help='Download from mozilla-inbound')\n branch_args.add_argument('--central', action='store_const', const='central', dest='branch',\n help='Download from mozilla-central (default)')\n branch_args.add_argument('--release', action='store_const', const='release', dest='branch',\n help='Download from mozilla-release')\n branch_args.add_argument('--beta', action='store_const', const='beta', dest='branch',\n help='Download from mozilla-beta')\n branch_args.add_argument('--esr52', action='store_const', const='esr52', dest='branch',\n help='Download from mozilla-esr52')\n branch_args.add_argument('--esr', action='store_const', const='esr60', dest='branch',\n help='Download from mozilla-esr60')\n\n build_group = parser.add_argument_group('Build Arguments')\n build_group.add_argument('-d', '--debug', action='store_true',\n help='Get debug builds w/ symbols (default=optimized).')\n build_group.add_argument('-a', '--asan', action='store_true',\n help='Download AddressSanitizer builds.')\n build_group.add_argument('--fuzzing', action='store_true',\n help='Download --enable-fuzzing builds.')\n build_group.add_argument('--coverage', action='store_true',\n help='Download --coverage builds. This also pulls down the *.gcno files')\n build_group.add_argument('--valgrind', action='store_true',\n help='Download Valgrind builds.')\n\n test_group = parser.add_argument_group('Test Arguments')\n test_group.add_argument('--tests', nargs='+', metavar='', choices=cls.TEST_CHOICES,\n help=('Download tests associated with this build. Acceptable values are: ' +\n ', '.join(cls.TEST_CHOICES)))\n test_group.add_argument('--full-symbols', action='store_true',\n help='Download the full crashreport-symbols.zip archive.')\n\n misc_group = parser.add_argument_group('Misc. Arguments')\n misc_group.add_argument('-n', '--name',\n help='Specify a name (default=auto)')\n misc_group.add_argument('-o', '--out', default=os.getcwd(),\n help='Specify output directory (default=.)')\n misc_group.add_argument('--dry-run', action='store_true',\n help=\"Search for build and output metadata only, don't download anything.\")\n\n args = parser.parse_args(args=args)\n\n if re.match(r'(\\d{4}-\\d{2}-\\d{2}|[0-9A-Fa-f]{40}|latest)$', args.build) is None:\n # this is a custom build\n # ensure conflicting options are not set\n if args.branch is not None:\n parser.error('Cannot specify --build namespace and branch argument: %s' % args.branch)\n if args.debug:\n parser.error('Cannot specify --build namespace and --debug')\n if args.asan:\n parser.error('Cannot specify --build namespace and --asan')\n if args.fuzzing:\n parser.error('Cannot specify --build namespace and --fuzzing')\n if args.coverage:\n parser.error('Cannot specify --build namespace and --coverage')\n if args.valgrind:\n parser.error('Cannot specify --build namespace and --valgrind')\n\n # do this default manually so we can error if combined with --build namespace\n # parser.set_defaults(branch='central')\n elif args.branch is None:\n args.branch = 'central'\n\n flags = BuildFlags(args.asan, args.debug, args.fuzzing, args.coverage, args.valgrind)\n obj = cls(args.target, args.branch, args.build, flags, Platform(args.os, args.cpu))\n\n if args.name is None:\n args.name = obj.get_auto_name()\n\n final_dir = os.path.realpath(os.path.join(args.out, args.name))\n if not skip_dir_check and os.path.exists(final_dir):\n parser.error('Folder exists: %s .. exiting' % final_dir)\n\n extract_options = {\n 'dry_run': args.dry_run,\n 'out': final_dir,\n 'full_symbols': args.full_symbols,\n 'tests': args.tests\n }\n\n return obj, extract_options"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting the original response for comparing confirm is_cookie_necessary", "response": "def init_original_response(self):\n \"\"\"Get the original response for comparing, confirm ``is_cookie_necessary``\"\"\"\n if 'json' in self.request:\n self.request['data'] = json.dumps(self.request.pop('json')).encode(\n self.encoding)\n r1 = self.req.request(\n retry=self.retry, timeout=self.timeout, **self.request)\n resp = r1.x\n assert resp, ValueError(\n 'original_response should not be failed. %s' % self.request)\n self.encoding = self.encoding or resp.encoding\n self.original_response = self.ensure_response(r1)\n return self.original_response"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the original response for comparing confirm is_cookie_necessary", "response": "def init_original_response(self):\n \"\"\"Get the original response for comparing, confirm is_cookie_necessary\"\"\"\n no_cookie_resp = None\n self.is_cookie_necessary = True\n if 'json' in self.request:\n self.request['data'] = json.dumps(self.request.pop('json')).encode(\n self.encoding)\n r1 = self.req.request(\n retry=self.retry, timeout=self.timeout, **self.request)\n if 'headers' in self.request:\n # test is_cookie_necessary\n cookie = self.request['headers'].get('Cookie', None)\n if cookie:\n new_request = deepcopy(self.request)\n new_request['headers']['Cookie'] = ''\n r2 = self.req.request(\n retry=self.retry, timeout=self.timeout, **new_request)\n no_cookie_resp = self.ensure_response(r2)\n resp = r1.x\n assert resp, ValueError(\n 'original_response should not be failed. %s' % self.request)\n self.original_response = self.ensure_response(r1)\n self.encoding = self.encoding or resp.encoding\n if no_cookie_resp == self.original_response:\n self.ignore['headers'].append('Cookie')\n self.is_cookie_necessary = False\n return self.original_response"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndoes nothing but sort the params of url.", "response": "def sort_url_qsl(cls, raw_url, **kwargs):\n \"\"\"Do nothing but sort the params of url.\n\n raw_url: the raw url to be sorted; \n kwargs: (optional) same kwargs for ``sorted``.\n \"\"\"\n parsed_url = urlparse(raw_url)\n qsl = parse_qsl(parsed_url.query)\n return cls._join_url(parsed_url, sorted(qsl, **kwargs))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef clean_cookie(self):\n if not self.is_cookie_necessary:\n return self\n headers = self.request.get('headers', {})\n cookies = SimpleCookie(headers['Cookie'])\n for k, v in cookies.items():\n new_cookie = '; '.join(\n [i.OutputString() for i in cookies.values() if i != v])\n new_request = deepcopy(self.request)\n new_request['headers']['Cookie'] = new_cookie\n self._add_task('Cookie', k, new_request)\n return self", "response": "Only clean the cookie from headers and return self."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef clean_headers(self):\n if not isinstance(self.request.get('headers'), dict):\n return self\n headers = self.request['headers']\n if 'Cookie' in headers:\n self.clean_cookie()\n for key in headers:\n # cookie will be checked in other methods.\n if key == 'Cookie':\n continue\n new_request = deepcopy(self.request)\n new_headers = deepcopy(headers)\n new_headers.pop(key)\n new_request['headers'] = new_headers\n self._add_task('headers', key, new_request)\n return self", "response": "Only clean the headers ( cookie include and return self."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef reset_new_request(self):\n raw_url = self.new_request['url']\n parsed_url = urlparse(raw_url)\n qsl = parse_qsl(parsed_url.query)\n new_url = self._join_url(\n parsed_url, [i for i in qsl if i not in self.ignore['qsl']])\n self.new_request['url'] = new_url\n self.logger_function('ignore: %s' % self.ignore)\n for key in self.ignore['headers']:\n self.new_request['headers'].pop(key)\n\n if not self.new_request.get('headers'):\n self.new_request.pop('headers', None)\n if self.ignore['Cookie'] and 'Cookie' not in self.ignore['headers']:\n headers = self.new_request['headers']\n headers = {key.title(): headers[key] for key in headers}\n if 'Cookie' in headers:\n cookies = SimpleCookie(headers['Cookie'])\n new_cookie = '; '.join([\n i[1].OutputString()\n for i in cookies.items()\n if i[0] not in self.ignore['Cookie']\n ])\n self.new_request['headers']['Cookie'] = new_cookie\n\n if self.new_request['method'] == 'post':\n data = self.new_request.get('data')\n if data:\n if isinstance(data, dict):\n for key in self.ignore['form_data']:\n data.pop(key)\n if (not data) or self.ignore['total_data']:\n # not need data any more\n self.new_request.pop('data', None)\n if self.has_json_data and 'data' in self.new_request:\n json_data = json.loads(data.decode(self.encoding))\n for key in self.ignore['json_data']:\n json_data.pop(key)\n self.new_request['data'] = json.dumps(json_data).encode(\n self.encoding)\n return self.new_request", "response": "Remove the non - sense args from the self. ignore and return self. new_request"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef result(self):\n if not self.tasks:\n self.clean_all()\n tasks_length = len(self.tasks)\n self.logger_function(\n '%s tasks of request, will cost at least %s seconds.' %\n (tasks_length,\n round(self.req.interval / self.req.n * tasks_length, 2)))\n self.req.x\n for task in self.tasks:\n key, value, fut = task\n if fut.x and fut.cx:\n # fut.x == req success & fut.cx == response not changed.\n self.ignore[key].append(value)\n return self.reset_new_request()", "response": "Whole task clean_all + reset_new_request return self. new_request."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget value stored in cache file or store there default value.", "response": "def get_or_save_default(self, key, default_value):\n \"\"\"\n Get value stored in cache file or store there default value.\n :param key:\n :param default_value:\n :return:\n \"\"\"\n val = self.get_or_none(key)\n if val is None:\n self.update(key, default_value)\n val = default_value\n return val"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef saveOverlayToDicomCopy(input_dcmfilelist, output_dicom_dir, overlays,\n crinfo, orig_shape):\n \"\"\" Save overlay to dicom. \"\"\"\n from . import datawriter as dwriter\n\n # import qmisc\n if not os.path.exists(output_dicom_dir):\n os.makedirs(output_dicom_dir)\n\n import imtools.image_manipulation\n # uncrop all overlays\n for key in overlays:\n overlays[key] = imtools.image_manipulation.uncrop(overlays[key], crinfo, orig_shape)\n\n dw = dwriter.DataWriter()\n dw.DataCopyWithOverlay(input_dcmfilelist, output_dicom_dir, overlays)", "response": "Save overlay to dicom."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate path with _segmentation suffix and keep extension.", "response": "def __get_segmentation_path(self, path):\n \"\"\" Create path with \"_segmentation\" suffix and keep extension.\n\n :param path:\n :return:\n \"\"\"\n startpath, ext = os.path.splitext(path)\n segmentation_path = startpath + \"_segmentation\" + ext\n return segmentation_path"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef Write3DData(self, data3d, path, filetype='auto', metadata=None, progress_callback=None, sfin=True):\n\n self.orig_path = path\n path = os.path.expanduser(path)\n\n try:\n d3d = data3d.pop('data3d')\n metadata = data3d\n data3d = d3d\n except:\n pass\n\n if progress_callback is not None:\n self.progress_callback = progress_callback\n\n if filetype == 'auto':\n startpath, ext = os.path.splitext(path)\n filetype = ext[1:].lower()\n\n segmentation = None\n if metadata is not None and \"segmentation\" in metadata.keys():\n segmentation_path = self.__get_segmentation_path(path)\n segmentation = metadata[\"segmentation\"]\n\n\n\n mtd = {'voxelsize_mm': [1, 1, 1]}\n if metadata is not None:\n mtd.update(metadata)\n metadata=mtd\n\n if path.find('{') >= 0:\n filetype = 'image_stack'\n # one_file_per_slice = True\n\n # if one_file_per_slice:\n # self._one_file_per_slice(self, data3d, path, filetype, metadata)\n # else:\n # self._all_in_one_file(self, data3d, path, filetype, metadata)\n #\n # def _all_in_one_file(self, data3d, path, filetype, metadata):\n\n if filetype in ['vtk', 'tiff', 'tif', \"mhd\", \"nii\", \"raw\"]:\n self._write_with_sitk(path, data3d, metadata)\n if sfin and segmentation is not None:\n self._write_with_sitk(segmentation_path, segmentation, metadata)\n elif filetype in ['dcm', 'DCM', 'dicom']:\n self._write_with_sitk(path, data3d, metadata)\n self._fix_sitk_bug(path, metadata)\n if sfin and segmentation is not None:\n self._write_with_sitk(segmentation_path, segmentation, metadata)\n self._fix_sitk_bug(segmentation_path, metadata)\n elif filetype in ['rawiv']:\n rawN.write(path, data3d, metadata)\n elif filetype in ['image_stack']:\n self.save_image_stack(data3d, path, metadata)\n elif filetype in ['hdf5', 'hdf', 'h5', 'he5']:\n self.save_hdf5(data3d, path, metadata)\n elif filetype in ['pkl', 'pklz']:\n from . import misc\n metadata['data3d'] = data3d\n datap = metadata\n\n misc.obj_to_file(datap, path)\n\n else:\n logger.error('Unknown filetype: \"' + filetype + '\"')\n raise ValueError(\"Unknown filetype: '\" + filetype + \"'\")", "response": "Write 3D data to file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _fix_sitk_bug(self, path, metadata):\n ds = dicom.read_file(path)\n ds.SpacingBetweenSlices = str(metadata[\"voxelsize_mm\"][0])[:16]\n dicom.write_file(path, ds)", "response": "This function fixes the bug in simple ITK for Z axis in 3D images."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef DataCopyWithOverlay(self, dcmfilelist, out_dir, overlays):\n dcmlist = dcmfilelist\n # data3d = []\n\n for i in range(len(dcmlist)):\n onefile = dcmlist[i]\n\n logger.info(onefile)\n data = dicom.read_file(onefile)\n\n for i_overlay in overlays.keys():\n overlay3d = overlays[i_overlay]\n data = self.encode_overlay_slice(data,\n overlay3d[-1 - i, :, :],\n i_overlay)\n\n # construct output path\n head, tail = os.path.split(os.path.normpath(onefile))\n filename_out = os.path.join(out_dir, tail)\n\n# save\n data.save_as(filename_out)", "response": "Function make 3D data from dicom files and overlays"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfunctions adds overlay to existing file.", "response": "def add_overlay_to_slice_file(\n self,\n filename,\n overlay,\n i_overlay,\n filename_out=None\n ):\n \"\"\" Function adds overlay to existing file.\n \"\"\"\n if filename_out is None:\n filename_out = filename\n filename = op.expanduser(filename)\n data = dicom.read_file(filename)\n data = self.encode_overlay_slice(data, overlay, i_overlay)\n data.save_as(filename_out)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef deprecated(instructions):\n def decorator(func):\n \"\"\"This is a decorator which can be used to mark functions as deprecated.\n\n It will result in a warning being emitted when the function is used.\n \"\"\"\n @functools.wraps(func)\n def wrapper(*args, **kwargs):\n message = 'Call to deprecated function {}. {}'.format(func.__name__,\n instructions)\n frame = inspect.currentframe().f_back\n warnings.warn_explicit(message,\n category=DeprecatedWarning,\n filename=inspect.getfile(frame.f_code),\n lineno=frame.f_lineno)\n return func(*args, **kwargs)\n return wrapper\n return decorator", "response": "A decorator that marks a method as deprecated."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef log_verbose(self, message):\n if self.get_verbosity() >= Output.VERBOSITY_VERBOSE:\n self.writeln(message)", "response": "Logs a message only when logging level is verbose."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nlogs a message only when logging level is very verbose.", "response": "def log_very_verbose(self, message):\n \"\"\"\n Logs a message only when logging level is very verbose.\n\n :param str|list[str] message: The message.\n \"\"\"\n if self.get_verbosity() >= Output.VERBOSITY_VERY_VERBOSE:\n self.writeln(message)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets a feed from the given URL.", "response": "async def get_feed(config, url):\n \"\"\" Get a feed\n\n Arguments:\n\n config -- the configuration\n url -- The URL of the feed\n\n retval -- a tuple of feed,previous_version,changed\n \"\"\"\n\n LOGGER.debug(\"++WAIT: cache get feed %s\", url)\n previous = config.cache.get(\n 'feed', url, schema_version=SCHEMA_VERSION) if config.cache else None\n LOGGER.debug(\"++DONE: cache get feed %s\", url)\n\n headers = previous.caching if previous else None\n\n LOGGER.debug(\"++WAIT: request get %s\", url)\n request = await utils.retry_get(config, url, headers=headers)\n LOGGER.debug(\"++DONE: request get %s\", url)\n if not request or not request.success:\n LOGGER.error(\"Could not get feed %s: %d\",\n url,\n request.status if request else -1)\n return None, previous, False\n\n if request.cached:\n LOGGER.debug(\"%s: Reusing cached version\", url)\n return previous, previous, False\n\n current = Feed(request)\n\n if config.cache:\n LOGGER.debug(\"%s: Saving to cache\", url)\n LOGGER.debug(\"++WAIT: cache set feed %s\", url)\n config.cache.set('feed', url, current)\n LOGGER.debug(\"++DONE: cache set feed %s\", url)\n\n LOGGER.debug(\"%s: Returning new content\", url)\n return current, previous, (not previous\n or current.digest != previous.digest\n or current.status != previous.status)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the known namespace of the RFC5005 extension.", "response": "def archive_namespace(self):\n \"\"\" Returns the known namespace of the RFC5005 extension, if any \"\"\"\n try:\n for ns_prefix, url in self.feed.namespaces.items():\n if url == 'http://purl.org/syndication/history/1.0':\n return ns_prefix\n except AttributeError:\n pass\n return None"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef entry_links(self):\n return {entry['link'] for entry in self.feed.entries if entry and entry.get('link')}", "response": "Given a parsed feed return the links to its entries including ones\n which disappeared"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef is_archive(self):\n\n ns_prefix = self.archive_namespace\n if ns_prefix:\n if ns_prefix + '_archive' in self.feed.feed:\n # This is declared to be an archive view\n return True\n if ns_prefix + '_current' in self.feed.feed:\n # This is declared to be the current view\n return False\n\n # Either we don't have the namespace, or the view wasn't declared.\n rels = collections.defaultdict(list)\n for link in self.feed.feed.links:\n rels[link.rel].append(link.href)\n\n return ('current' in rels and\n ('self' not in rels or\n rels['self'] != rels['current']))", "response": "Given a parsed feed returns True if this is an archive feed False otherwise"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\nasync def update_websub(self, config, hub):\n try:\n LOGGER.debug(\"WebSub: Notifying %s of %s\", hub, self.url)\n request = await utils.retry_post(\n config,\n hub,\n data={\n 'hub.mode': 'publish',\n 'hub.url': self.url\n })\n\n if request.success:\n LOGGER.info(\"%s: WebSub notification sent to %s\",\n self.url, hub)\n else:\n LOGGER.warning(\"%s: Hub %s returned status code %s: %s\", self.url, hub,\n request.status, request.text)\n except Exception as err: # pylint:disable=broad-except\n LOGGER.warning(\"WebSub %s: got %s: %s\",\n hub, err.__class__.__name__, err)", "response": "Update the WebSub hub to know about this feed"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of tags.", "response": "def get_tags(self, name):\n \"\"\"\n Returns a list of tags.\n\n @param str name: The name of the tag.\n\n :rtype: list[str]\n \"\"\"\n tags = list()\n for tag in self._tags:\n if tag[0] == name:\n tags.append(tag[1])\n\n return tags"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef __remove_leading_empty_lines(lines):\n tmp = list()\n empty = True\n for i in range(0, len(lines)):\n empty = empty and lines[i] == ''\n if not empty:\n tmp.append(lines[i])\n\n return tmp", "response": "Removes leading empty lines from a list of lines."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nremoves leading empty lines from a list of lines.", "response": "def __remove_trailing_empty_lines(lines):\n \"\"\"\n Removes leading empty lines from a list of lines.\n\n :param list[str] lines: The lines.\n \"\"\"\n lines.reverse()\n tmp = DocBlockReflection.__remove_leading_empty_lines(lines)\n lines.reverse()\n tmp.reverse()\n\n return tmp"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncleans the DocBlock from leading and trailing white space and comment tokens.", "response": "def __clean_doc_block(self):\n \"\"\"\n Cleans the DocBlock from leading and trailing white space and comment tokens.\n \"\"\"\n # Return immediately if the DockBlock is empty.\n if not self._comment:\n return\n\n for i in range(1, len(self._comment) - 1):\n self._comment[i] = re.sub(r'^\\s*\\*', '', self._comment[i])\n\n self._comment[0] = re.sub(r'^\\s*/\\*\\*', '', self._comment[0])\n\n self._comment[-1] = re.sub(r'\\*/\\s*$', '', self._comment[-1])\n\n for i, line in enumerate(self._comment):\n self._comment[i] = line.strip()\n\n self._comment = self.__remove_leading_empty_lines(self._comment)\n self._comment = self.__remove_trailing_empty_lines(self._comment)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nextracts the description from the DocBlock.", "response": "def __extract_description(self):\n \"\"\"\n Extracts the description from the DocBlock. The description start at the first line and stops at the first tag\n or the end of the DocBlock.\n \"\"\"\n tmp = list()\n for line in self._comment:\n if len(line) >= 1 and line[0] == '@':\n break\n\n tmp.append(line)\n\n tmp = self.__remove_trailing_empty_lines(tmp)\n\n self._description = os.linesep.join(tmp)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef __extract_tags(self):\n tags = list()\n current = None\n for line in self._comment:\n parts = re.match(r'^@(\\w+)', line)\n if parts:\n current = (parts.group(1), list())\n tags.append(current)\n\n if current:\n if line == '':\n current = None\n else:\n current[1].append(line)\n\n for tag in tags:\n self._tags.append((tag[0], os.linesep.join(tag[1])))", "response": "Extract tags from the DocBlock."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef handle(self):\n self.output = PyStratumStyle(self.input, self.output)\n\n config_file = self.input.get_argument('config_file')\n self.run_command(config_file)", "response": "Executes constants command when PyStratumCommand is activated."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nrunning the neccesary command.", "response": "def run_command(self, config_file):\n \"\"\"\n :param str config_file: The name of config file.\n \"\"\"\n config = configparser.ConfigParser()\n\n config.read(config_file)\n\n rdbms = config.get('database', 'rdbms').lower()\n label_regex = config.get('constants', 'label_regex')\n\n constants = self.create_constants(rdbms)\n constants.main(config_file, label_regex)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a factory function for creating a Constants objects based on column widths and auto - increment columns and labels.", "response": "def create_constants(self, rdbms):\n \"\"\"\n Factory for creating a Constants objects (i.e. objects for creating constants based on column widths, and auto\n increment columns and labels).\n\n :param str rdbms: The target RDBMS (i.e. mysql, mssql or pgsql).\n\n :rtype: pystratum.Constants.Constants\n \"\"\"\n # Note: We load modules and classes dynamically such that on the end user's system only the required modules\n # and other dependencies for the targeted RDBMS must be installed (and required modules and other\n # dependencies for the other RDBMSs are not required).\n\n if rdbms == 'mysql':\n module = locate('pystratum_mysql.MySqlConstants')\n return module.MySqlConstants(self.output)\n\n if rdbms == 'mssql':\n module = locate('pystratum_mssql.MsSqlConstants')\n return module.MsSqlConstants(self.output)\n\n if rdbms == 'pgsql':\n module = locate('pystratum_pgsql.PgSqlConstants')\n return module.PgSqlConstants(self.output)\n\n raise Exception(\"Unknown RDBMS '{0!s}'.\".format(rdbms))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning formatted redis key name.", "response": "def _format_key_name(self) -> str:\n \"\"\"Return formatted redis key name.\"\"\"\n key_name = 'ld:{0}:{1}:features'.format(\n self.project_key,\n self.environment_key\n )\n return key_name"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses Connection string to extract host and port.", "response": "def connection_string_parser(uri: str) -> list:\n \"\"\"\n Parse Connection string to extract host and port.\n\n :param uri: full URI for redis connection in the form of host:port\n\n :returns: list of RedisConnection objects\n \"\"\"\n redis_connections = []\n raw_connections = uri.split(',')\n connections = [\n connection for connection in raw_connections if len(connection) > 0\n ]\n\n for connection in connections:\n raw_connection = connection.split(':')\n if len(raw_connection) == 1:\n host = raw_connection[0].strip()\n port = _DEFAULT_REDIS_PORT\n elif len(raw_connection) == 2:\n host = raw_connection[0].strip()\n port = int(raw_connection[1])\n else:\n raise RuntimeError(\n \"Unable to parse redis connection string: {0}\".format(\n raw_connection\n )\n )\n\n redis_connection = _RedisConnection(host, port)\n redis_connections.append(redis_connection)\n\n return redis_connections"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget feature flag record from redis.", "response": "def get_flag_record(self, feature_key: str) -> str:\n \"\"\"Get feature flag record from redis.\n\n :param feature_key: key for feature flag\n\n :return: value of feature flag key in redis.\n\n :raises: KeyError if key is not found.\n \"\"\"\n key_name = self._format_key_name()\n flag = self.redis.hget(key_name, feature_key)\n\n if flag is None:\n raise KeyError('Redis key: {0} not found.'.format(key_name))\n\n return flag"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nupdating redis record with new state.", "response": "def update_flag_record(self, state: str, feature_key: str) -> None:\n \"\"\"Update redis record with new state.\n\n :param state: state for feature flag.\n :param feature_key: key for feature flag.\n \"\"\"\n key_name = self._format_key_name()\n try:\n parsed_flag = json.loads(self.get_flag_record(feature_key).decode('utf-8'))\n parsed_flag['on'] = state\n parsed_flag['version'] += 1\n updated_flag = json.dumps(parsed_flag).encode('utf-8')\n except KeyError as ex:\n LOG.error(ex)\n sys.exit(1)\n\n LOG.info('updating %s to %s', feature_key, state)\n\n self.redis.hset(key_name, feature_key, updated_flag)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef Asyncme(func, n=None, interval=0, default_callback=None, loop=None):\n return coros(n, interval, default_callback, loop)(func)", "response": "Wrap coro_function into the function return NewTask."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef coros(n=None, interval=0, default_callback=None, loop=None):\n submitter = Loop(\n n=n, interval=interval, default_callback=default_callback, loop=loop\n ).submitter\n\n return submitter", "response": "Decorator for wrap coro_function into the function return NewTask."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nwrapping a function to set the result as self. _callback_result.", "response": "def wrap_callback(function):\n \"\"\"Set the callback's result as self._callback_result.\"\"\"\n\n @wraps(function)\n def wrapped(task):\n task._callback_result = function(task)\n return task._callback_result\n\n return wrapped"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _schedule_callbacks(self, clear_cb=False):\n self.task_end_time = time.time()\n self.task_cost_time = self.task_end_time - self.task_start_time\n callbacks = self._callbacks[:]\n if not callbacks:\n return\n if clear_cb:\n self._callbacks[:] = []\n for callback in callbacks:\n self._loop.call_soon(callback, self, *self.extra_args)", "response": "Schedule the callbacks to be called when the task is finished."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef callback_result(self):\n if self._state == self._PENDING:\n self._loop.run_until_complete(self)\n if self._callbacks:\n result = self._callback_result\n else:\n result = self.result()\n return result", "response": "Blocking until the task finish and return the callback_result. until"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef x(self):\n if self._state == self._PENDING:\n self._loop.run_until_complete(self)\n return self.result()", "response": "Blocking until the task finish and return the self. result"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _wrap_coro_function_with_sem(self, coro_func):\n sem = self.frequency.sem\n interval = self.frequency.interval\n\n @wraps(coro_func)\n async def new_coro_func(*args, **kwargs):\n if sem:\n async with sem:\n result = await coro_func(*args, **kwargs)\n if interval:\n await asyncio.sleep(interval)\n return result\n else:\n result = await coro_func(*args, **kwargs)\n if interval:\n await asyncio.sleep(interval)\n return result\n\n return new_coro_func", "response": "Decorator set the coro_func has sem control."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef run_in_executor(self, executor=None, func=None, *args):\n return self.loop.run_in_executor(executor, func, *args)", "response": "Run a function in an executor."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef run_in_thread_pool(self, pool_size=None, func=None, *args):\n executor = Pool(pool_size)\n return self.loop.run_in_executor(executor, func, *args)", "response": "Run a function in a thread pool."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef run_in_process_pool(self, pool_size=None, func=None, *args):\n executor = ProcessPool(pool_size)\n return self.loop.run_in_executor(executor, func, *args)", "response": "Run a function in a process pool."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nis used when loop running in a single non - main thread.", "response": "def run_coroutine_threadsafe(self, coro, loop=None, callback=None):\n \"\"\"Be used when loop running in a single non-main thread.\"\"\"\n if not asyncio.iscoroutine(coro):\n raise TypeError(\"A await in coroutines. object is required\")\n loop = loop or self.loop\n future = NewFuture(callback=callback)\n\n def callback_func():\n try:\n asyncio.futures._chain_future(NewTask(coro, loop=loop), future)\n except Exception as exc:\n if future.set_running_or_notify_cancel():\n future.set_exception(exc)\n raise\n\n loop.call_soon_threadsafe(callback_func)\n return future"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsubmit a coroutine function to self. loop with frequency control.", "response": "def apply(self, coro_function, args=None, kwargs=None, callback=None):\n \"\"\"Submit a coro_function(*args, **kwargs) as NewTask to self.loop with loop.frequncy control.\n\n ::\n\n from torequests.dummy import Loop\n import asyncio\n loop = Loop()\n\n async def test(i):\n result = await asyncio.sleep(1)\n return (loop.frequency, i)\n\n task = loop.apply(test, [1])\n print(task)\n # loop.x can be ignore\n loop.x\n print(task.x)\n\n # >\n # (Frequency(sem=<0/0>, interval=0, name=loop_sem), 1)\n \"\"\"\n args = args or ()\n kwargs = kwargs or {}\n coro = self._wrap_coro_function_with_sem(coro_function)(*args, **kwargs)\n return self.submit(coro, callback=callback)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef submit(self, coro, callback=None):\n callback = callback or self.default_callback\n if self.async_running:\n return self.run_coroutine_threadsafe(coro, callback=callback)\n else:\n return NewTask(coro, loop=self.loop, callback=callback)", "response": "Submit a coroutine as NewTask to self. loop without loop. frequencyuncy control."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef todo_tasks(self):\n tasks = [task for task in self.all_tasks if task._state == NewTask._PENDING]\n return tasks", "response": "Return tasks in loop which its state is pending."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn tasks in loop which its state is not pending.", "response": "def done_tasks(self):\n \"\"\"Return tasks in loop which its state is not pending.\"\"\"\n tasks = [task for task in self.all_tasks if task._state != NewTask._PENDING]\n return tasks"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nblocking run loop until all tasks completed.", "response": "def run(self, tasks=None, timeout=None):\n \"\"\"Block, run loop until all tasks completed.\"\"\"\n timeout = self._timeout if timeout is None else timeout\n if self.async_running or self.loop.is_running():\n return self.wait_all_tasks_done(timeout)\n else:\n tasks = tasks or self.todo_tasks\n return self.loop.run_until_complete(asyncio.gather(*tasks, loop=self.loop))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nblock until all tasks are done.", "response": "def wait_all_tasks_done(self, timeout=None, delay=0.5, interval=0.1):\n \"\"\"Block, only be used while loop running in a single non-main thread.\"\"\"\n timeout = self._timeout if timeout is None else timeout\n timeout = timeout or float(\"inf\")\n start_time = time.time()\n time.sleep(delay)\n while 1:\n if not self.todo_tasks:\n return self.all_tasks\n if time.time() - start_time > timeout:\n return self.done_tasks\n time.sleep(interval)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\nasync def pendings(self, tasks=None):\n tasks = tasks or self.todo_tasks\n await asyncio.gather(*tasks, loop=self.loop)", "response": "Used for await in coroutines.\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef ensure_frequencies(self, frequencies):\n if not frequencies:\n return {}\n if not isinstance(frequencies, dict):\n raise ValueError(\"frequencies should be dict\")\n frequencies = {\n host: Frequency.ensure_frequency(frequencies[host]) for host in frequencies\n }\n return frequencies", "response": "Ensure frequencies is dict of host - frequencies."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef set_frequency(self, host, sem=None, interval=None):\n # single sem or global sem\n sem = sem or self.sem\n interval = self.interval if interval is None else interval\n frequency = Frequency(sem, interval, host)\n frequencies = {host: frequency}\n self.update_frequency(frequencies)\n return frequency", "response": "Set frequency for a given host with sem and interval."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef initialize_openstack(func):\n '''\n Initialize and refresh openstack connection\n\n '''\n async def wrap(self, *args, **kwargs):\n if not hasattr(self, 'auth') or not self.auth.is_token_valid():\n self.auth = AuthPassword(auth_url=self.config['auth_url'],\n username=self.config['username'],\n password=self.config['password'],\n project_name=self.config['project_name'],\n user_domain_name=self.config['user_domain_name'],\n project_domain_name=self.config['project_domain_name'])\n self.nova = NovaClient(session=self.auth)\n self.glance = GlanceClient(session=self.auth)\n await self.nova.init_api(timeout=self.config.get('http_timeout', 10))\n await self.glance.init_api(timeout=self.config.get('http_timeout', 10))\n\n if not hasattr(self, 'last_init') or self.last_init < (time.time() - 60):\n await self.initialize()\n self.last_init = time.time()\n return await func(self, *args, **kwargs)\n return wrap", "response": "A decorator that initializes and refreshes the openstack connection."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\nasync def initialize(self):\n '''\n Initialize static data like images and flavores and set it as object property\n '''\n flavors = await self._list_flavors()\n images = await self._list_images()\n\n self.flavors_map = bidict()\n self.images_map = bidict()\n self.images_details = {}\n\n for flavor in flavors:\n self.flavors_map.put(flavor['id'], flavor['name'], on_dup_key='OVERWRITE', on_dup_val='OVERWRITE')\n\n for image in images:\n # @TODO filetes :\n # @TODO filtering by owner\n # if hasattr(image, 'owner_id') and image.owner_id in self.config['image_owner_ids']:\n # @TODO enable filtering by tag\n # if 'lastest' in image.tags:\n self.images_details[image['id']] = {\n 'name': image['name'],\n 'created_at': image['created_at'],\n 'latest': 'latest' in image['tags']\n }\n self.images_map.put(image['id'], image['name'], on_dup_key='OVERWRITE', on_dup_val='OVERWRITE')", "response": "Initialize static data like images and flavores and set it as object property\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\nasync def create_vm(self, preset_name, image, flavor, security_groups=None,\n userdata=None, key_name=None, availability_zone=None, subnets=None):\n '''\n Create VM\n :arg preset_name: string\n :arg image: string image id\n :arg flavor: string flavor id\n :arg security_groups: list\n :arg userdata: string\n :arg key_name: string\n :arg availability_zone: string\n :arg subnets: list\n :returns list Vm objects\n @TODO\n 1. returns image id\n '''\n image_id = self.images_map.inv.get(image)\n flavor_id = self.flavors_map.inv.get(flavor)\n spec = {\n \"name\": preset_name,\n \"flavorRef\": flavor_id,\n \"imageRef\": image_id,\n \"security_groups\": [{\"name\": group} for group in security_groups],\n \"user_data\": userdata\n }\n if availability_zone is not None:\n spec.update({\"availability_zone\": availability_zone})\n if subnets is not None:\n spec.update({\"networks\": [{'uuid': subnet['net-id']} for subnet in subnets]})\n if userdata is not None:\n userdata = userdata.encode('utf-8')\n userdata = base64.b64encode(userdata).decode('utf-8')\n spec.update({\"user_data\": userdata})\n\n result = await self.nova.servers.create(server=spec)\n return result[\"server\"]", "response": "Create VM with optional user data."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\nasync def list_vms(self, preset_name):\n '''\n List VMs by preset name\n :arg present_name: string\n '''\n\n response = await self.nova.servers.list(name=f'^{preset_name}$')\n result = []\n for server in response['servers']:\n result.append(self._map_vm_structure(server))\n return result", "response": "List VMs by preset name."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nterminates VM :arg vm_id: string", "response": "async def terminate_vm(self, vm_id):\n '''\n Terminate VM\n :arg vm_id: string\n '''\n try:\n await self.nova.servers.force_delete(vm_id)\n except JSONDecodeError as exc:\n logging.info(\"nova sent 'content-type: application/json' but no content appeared, whatever\")\n pass\n except Exception:\n raise"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets VM :arg vm_id: string :returns vm: object", "response": "async def get_vm(self, vm_id):\n '''\n Get VM\n :arg vm_id: string\n :returns vm: object\n '''\n result = await self.nova.servers.get(vm_id)\n return self._map_vm_structure(result[\"server\"])"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _map_vm_structure(self, vm):\n '''\n Vm unification\n :arg vm: object\n :returns object\n '''\n ip = self._extract_ips(vm['addresses'])\n created = datetime.strptime(vm['created'], '%Y-%m-%dT%H:%M:%SZ')\n flavor = self.flavors_map.get(vm['flavor'].get('id'))\n image = self.images_map.get(vm['image'].get('id'))\n timed_shutdown_at = vm.get('metadata', {}).get('iaas_timed_shutdown')\n timed_shutdown_at = int(timed_shutdown_at) if timed_shutdown_at else None\n state = self._map_vm_status(vm['status'], timed_shutdown_at)\n iaasvm = Vm(self, vm['id'], vm['name'], ip, created, state=state, metadata=vm['metadata'], tags=vm.get('tags', []), flavor=flavor,\n image=image, timed_shutdown_at=timed_shutdown_at)\n return iaasvm", "response": "Map a vm structure to a new object."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _map_vm_status(self, openstack_status, timed_shutdown_at=None):\n '''\n Map openstack vm statuses to vmshepherd vm statuses\n openstack vm statuses: ACTIVE, BUILD, DELETED, ERROR, HARD_REBOOT, MIGRATING, PASSWORD, PAUSED, REBOOT,\n REBUILD, RESCUE, RESIZE, REVERT_RESIZE, SHELVED, SHELVED_OFFLOADED, SHUTOFF, SOFT_DELETED, SUSPENDED, UNKNOWN,\n VERIFY_RESIZE\n\n :arg string openstack_status\n :returns string\n '''\n statuses = {\n VmState.TERMINATED: [\n 'ERROR',\n 'DELETED',\n 'SHUTOFF',\n 'SOFT_DELETED',\n 'SUSPENDED'\n ],\n VmState.PENDING: [\n 'BUILD',\n 'REBUILD'\n ],\n VmState.RUNNING: ['ACTIVE']\n }\n\n state = VmState.UNKNOWN\n for vmstate, value in statuses.items():\n if openstack_status in value:\n state = vmstate\n break\n if timed_shutdown_at:\n now = time.time()\n if timed_shutdown_at < now:\n state = VmState.AFTER_TIME_SHUTDOWN\n elif (timed_shutdown_at - now) < self.config.get('shutdown_grace_period', 900):\n state = VmState.NEARBY_SHUTDOWN\n\n return state", "response": "Map openstack status to vmshepherd status"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nextracts ip addressess from openstack structure", "response": "def _extract_ips(self, data):\n '''\n Extract ip addressess from openstack structure\n {\n 'pl-krk-2-int-301-c2-int-1': [\n {\n 'OS-EXT-IPS-MAC:mac_addr': 'fa:16:3e:29:f1:bb',\n 'version': 4,\n 'addr': '10.185.138.36',\n 'OS-EXT-IPS:type': 'fixed'\n }\n ]\n }\n :arg data: dict\n :returns list\n '''\n result = []\n for region in data.items():\n for interface in region[1]:\n result.append(interface['addr'])\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsets the text of the object to the given text.", "response": "def _set_label_text(obj, text, tooltip=None, replace_all=False):\n \"\"\"\n Keep text before first colon and replace the rest with new text.\n\n If there is no colon in the\n :param obj:\n :param text:\n :param tooltip:\n :param replace_all: No colon is searched and whole text is replaced\n :return:\n \"\"\"\n dlab = str(obj.text())\n index_of_colon = dlab.find(': ')\n if index_of_colon == -1:\n index_of_colon = 0\n else:\n index_of_colon += 2\n if replace_all:\n index_of_colon = 0\n obj.setText(dlab[:index_of_colon] + '%s' % text)\n if tooltip is not None:\n obj.setToolTip(tooltip)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndraw a dialog for directory selection.", "response": "def __get_datafile(self, app=False):\n \"\"\"\n Draw a dialog for directory selection.\n \"\"\"\n\n if self.cache is not None:\n cache_loadfiledir = self.cache.get_or_none('loadfiledir')\n self.loadfiledir = str(cache_loadfiledir)\n\n if self.loadfiledir is None:\n self.loadfiledir = ''\n directory = str(self.loadfiledir)\n from PyQt4.QtGui import QFileDialog\n if not app:\n inner_app = QApplication(sys.argv)\n if self._skip_get_path_dialog_for_tests:\n dcmdir = self.datapath\n else:\n dcmdir = QFileDialog.getOpenFileName(\n caption='Select Data File',\n directory=directory\n # ptions=QFileDialog.ShowDirsOnly,\n )\n # dcmdir = QFileDialog.getOpenFileName(\n # caption='Select Data file',\n # # ptions=QFileDialog.ShowDirsOnly,\n # directory=directory\n # )\n # pp.exec_()\n if not app:\n inner_app.exit(0)\n\n dcmdir = get_str(dcmdir)\n\n if len(dcmdir) > 0:\n #\n # dcmdir = \"%s\" % (dcmdir)\n # dcmdir = dcmdir.encode(\"utf8\")\n pass\n else:\n dcmdir = None\n\n\n head, teil = os.path.split(dcmdir)\n if self.cache is not None:\n self.cache.update('loadfiledir', head)\n return dcmdir"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndrawing a dialog for directory selection.", "response": "def __get_datadir(self, app=False):\n \"\"\"\n Draw a dialog for directory selection.\n \"\"\"\n # if 'datadir' in self.oseg.cache.data.keys():\n # if :\n # directory = self.oseg.input_datapath_start\n if self.cache is not None:\n cache_loaddir = self.cache.get_or_none('loaddir')\n self.loaddir = str(cache_loaddir)\n # self.loaddir = str(self.cache.get_or_none('loaddir'))\n\n if self.loaddir is None:\n self.loaddir = ''\n\n directory = self.loaddir\n\n from PyQt4.QtGui import QFileDialog\n if not app:\n app_inner = QApplication(sys.argv)\n if self._skip_get_path_dialog_for_tests:\n dcmdir = self.datapath\n else:\n dcmdir = QFileDialog.getExistingDirectory(\n caption='Select DICOM Folder',\n options=QFileDialog.ShowDirsOnly,\n directory=directory\n )\n # pp.exec_()\n if not app:\n app_inner.exit(0)\n\n dcmdir = get_str(dcmdir)\n\n if len(dcmdir) > 0:\n\n\n # dcmdir = \"%s\" % (dcmdir)\n # dcmdir = dcmdir.encode(\"utf8\")\n pass\n else:\n dcmdir = None\n\n if self.cache is not None:\n self.cache.update('loaddir', dcmdir)\n return str(dcmdir)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef read_data_from_prepared_datapath(self):\n\n reader = datareader.DataReader()\n\n self.datap = reader.Get3DData(self.datapath, dataplus_format=True, gui=True, qt_app=self.qt_app)\n\n _set_label_text(self.text_dcm_dir, _make_text_short(self.datapath), self.datapath)\n _set_label_text(self.text_dcm_data, self.get_data_info(), replace_all=True)\n if self.after_function is not None:\n self.after_function(self)\n self.__show_message('Data read finished')", "response": "Function is called in the end of process\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef __generate_wrapper_class(self):\n routines = self._read_routine_metadata()\n\n self._write_class_header()\n\n if routines:\n for routine_name in sorted(routines):\n if routines[routine_name]['designation'] != 'hidden':\n self._write_routine_function(routines[routine_name])\n else:\n self._io.error('No files with stored routines found')\n\n self._write_class_trailer()\n\n Util.write_two_phases(self._wrapper_filename, self._code, self._io)", "response": "Generates the wrapper class."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _read_configuration_file(self, config_filename):\n config = configparser.ConfigParser()\n config.read(config_filename)\n\n self._parent_class_name = config.get('wrapper', 'parent_class')\n self._parent_class_namespace = config.get('wrapper', 'parent_class_namespace')\n self._wrapper_class_name = config.get('wrapper', 'wrapper_class')\n self._wrapper_filename = config.get('wrapper', 'wrapper_file')\n self._metadata_filename = config.get('wrapper', 'metadata')\n self._lob_as_string_flag = config.get('wrapper', 'lob_as_string')", "response": "Reads the configuration file and sets the attributes of the current object."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _read_routine_metadata(self):\n metadata = {}\n if os.path.isfile(self._metadata_filename):\n with open(self._metadata_filename, 'r') as file:\n metadata = json.load(file)\n\n return metadata", "response": "Reads the metadata of stored routines."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _write_class_header(self):\n self._write_line('from {0!s} import {1!s}'.format(self._parent_class_namespace, self._parent_class_name))\n self._write_line()\n self._write_line()\n self._write_line('# ' + ('-' * 118))\n self._write_line('class {0!s}({1!s}):'.format(self._wrapper_class_name, self._parent_class_name))\n self._write_line(' \"\"\"')\n self._write_line(' The stored routines wrappers.')\n self._write_line(' \"\"\"')", "response": "Generates a class header for the stored routine wrapper."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nwrite a line with Python code to the generate code buffer.", "response": "def _write_line(self, text=''):\n \"\"\"\n Writes a line with Python code to the generate code buffer.\n\n :param str text: The line with Python code.\n \"\"\"\n if text:\n self._code += str(text) + \"\\n\"\n else:\n self._code += \"\\n\""} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns as a generator of tasks order by completed sequence.", "response": "def get_results_generator(future_list, timeout=None, sort_by_completed=False):\n \"\"\"Return as a generator of tasks order by completed sequence.\"\"\"\n try:\n # python2 not support yield from\n if sort_by_completed:\n for future in as_completed(future_list, timeout=timeout):\n yield future.x\n else:\n for future in future_list:\n yield future.x\n except TimeoutError:\n return"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef run_after_async(seconds, func, *args, **kwargs):\n t = Timer(seconds, func, args, kwargs)\n t.daemon = True\n t.start()\n return t", "response": "Run the function after seconds asynchronously."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef async_func(self, function):\n\n @wraps(function)\n def wrapped(*args, **kwargs):\n return self.submit(function, *args, **kwargs)\n\n return wrapped", "response": "Decorator for let a normal function return the NewFuture"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the cpu count.", "response": "def _get_cpu_count(self):\n \"\"\"Get the cpu count.\"\"\"\n try:\n from multiprocessing import cpu_count\n\n return cpu_count()\n except Exception as e:\n Config.main_logger.error(\"_get_cpu_count failed for %s\" % e)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsubmitting a function to the pool.", "response": "def submit(self, func, *args, **kwargs):\n \"\"\"Submit a function to the pool, `self.submit(function,arg1,arg2,arg3=3)`\"\"\"\n\n with self._shutdown_lock:\n if self._shutdown:\n raise RuntimeError(\"cannot schedule new futures after shutdown\")\n callback = kwargs.pop(\"callback\", self.default_callback)\n future = NewFuture(\n self._timeout,\n args,\n kwargs,\n callback=callback,\n catch_exception=self.catch_exception,\n )\n w = _WorkItem(future, func, args, kwargs)\n self._work_queue.put(w)\n self._adjust_thread_count()\n self._all_futures.add(future)\n return future"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsubmit a function to the pool.", "response": "def submit(self, func, *args, **kwargs):\n \"\"\"Submit a function to the pool, `self.submit(function,arg1,arg2,arg3=3)`\"\"\"\n\n with self._shutdown_lock:\n if PY3 and self._broken:\n raise BrokenProcessPool(\n \"A child process terminated \"\n \"abruptly, the process pool is not usable anymore\"\n )\n if self._shutdown_thread:\n raise RuntimeError(\"cannot schedule new futures after shutdown\")\n callback = kwargs.pop(\"callback\", self.default_callback)\n future = NewFuture(\n self._timeout,\n args,\n kwargs,\n callback=callback,\n catch_exception=self.catch_exception,\n )\n w = _WorkItem(future, func, args, kwargs)\n self._pending_work_items[self._queue_count] = w\n self._work_ids.put(self._queue_count)\n self._queue_count += 1\n self._result_queue.put(None)\n self._start_queue_management_thread()\n if PY2:\n self._adjust_process_count()\n self._all_futures.add(future)\n return future"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ninvoke all the callbacks that have been set for this instance.", "response": "def _invoke_callbacks(self):\n \"\"\"Record the task_end_time & task_cost_time, set result for self._callback_result.\"\"\"\n self.task_end_time = time.time()\n self.task_cost_time = self.task_end_time - self.task_start_time\n with self._condition:\n for callback in self._done_callbacks:\n try:\n result = callback(self)\n if callback in self._user_callbacks:\n self._callback_result = result\n except Exception as e:\n Config.main_logger.error(\"exception calling callback for %s\" % e)\n self._condition.notify_all()"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nblocking the main thead until the future finish return the future. callback_result.", "response": "def callback_result(self):\n \"\"\"Block the main thead until future finish, return the future.callback_result.\"\"\"\n if self._state in [PENDING, RUNNING]:\n self.x\n if self._user_callbacks:\n return self._callback_result\n else:\n return self.x"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nblocks the main thead until the future finish return the future. result method.", "response": "def x(self):\n \"\"\"Block the main thead until future finish, return the future.result().\"\"\"\n with self._condition:\n result = None\n if not self.done():\n self._condition.wait(self._timeout)\n if not self.done():\n # timeout\n self.set_exception(TimeoutError())\n if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:\n # cancelled\n result = CancelledError()\n elif self._state == FINISHED:\n # finished\n if self._exception:\n result = self._exception\n else:\n result = self._result\n if isinstance(result, Exception):\n if self.catch_exception:\n result = FailureException(result)\n return result\n else:\n raise result\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nclose session shutdown pool.", "response": "def close(self, wait=False):\n \"\"\"Close session, shutdown pool.\"\"\"\n self.session.close()\n self.pool.shutdown(wait=wait)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef request(self, method, url, callback=None, retry=0, **kwargs):\n return self.pool.submit(\n self._request,\n method=method,\n url=url,\n retry=retry,\n callback=callback or self.default_callback,\n **kwargs\n )", "response": "Similar to requests. request but return as NewFuture."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get(self, url, params=None, callback=None, retry=0, **kwargs):\n kwargs.setdefault(\"allow_redirects\", True)\n return self.request(\n \"get\", url=url, params=params, callback=callback, retry=retry, **kwargs\n )", "response": "Similar to requests. get but return as NewFuture."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef post(self, url, data=None, json=None, callback=None, retry=0, **kwargs):\n return self.request(\n \"post\",\n url=url,\n data=data,\n json=json,\n callback=callback,\n retry=retry,\n **kwargs\n )", "response": "Similar to requests. post but return as NewFuture."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef getLinkInfo(self, wanInterfaceId=1, timeout=1):\n namespace = Wan.getServiceType(\"getLinkInfo\") + str(wanInterfaceId)\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetInfo\", timeout=timeout)\n\n return WanLinkInfo(results)", "response": "Execute GetInfo action to get basic WAN link information s."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef getLinkProperties(self, wanInterfaceId=1, timeout=1):\n namespace = Wan.getServiceType(\"getLinkProperties\") + str(wanInterfaceId)\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetCommonLinkProperties\", timeout=timeout)\n\n return WanLinkProperties(results)", "response": "Execute GetCommonLinkProperties action to get WAN link properties."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nexecuting GetInfo action to get basic ADSL information s.", "response": "def getADSLInfo(self, wanInterfaceId=1, timeout=1):\n \"\"\"Execute GetInfo action to get basic ADSL information's.\n\n :param int wanInterfaceId: the id of the WAN device\n :param float timeout: the timeout to wait for the action to be executed\n :return: ADSL informations.\n :rtype: ADSLInfo\n \"\"\"\n namespace = Wan.getServiceType(\"getADSLInfo\") + str(wanInterfaceId)\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetInfo\", timeout=timeout)\n\n return ADSLInfo(results)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef getEthernetLinkStatus(self, wanInterfaceId=1, timeout=1):\n namespace = Wan.getServiceType(\"getEthernetLinkStatus\") + str(wanInterfaceId)\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetEthernetLinkStatus\", timeout=timeout)\n\n return results[\"NewEthernetLinkStatus\"]", "response": "Execute GetEthernetLinkStatus action to get the status of the ethernet link."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nexecute GetTotalBytesSent & GetTotalBytesReceived actions to get WAN statistics.", "response": "def getByteStatistic(self, wanInterfaceId=1, timeout=1):\n \"\"\"Execute GetTotalBytesSent&GetTotalBytesReceived actions to get WAN statistics.\n\n :param int wanInterfaceId: the id of the WAN device\n :param float timeout: the timeout to wait for the action to be executed\n :return: a tuple of two values, total bytes sent and total bytes received\n :rtype: list[int]\n \"\"\"\n namespace = Wan.getServiceType(\"getByteStatistic\") + str(wanInterfaceId)\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetTotalBytesSent\", timeout=timeout)\n results2 = self.execute(uri, namespace, \"GetTotalBytesReceived\", timeout=timeout)\n\n return [int(results[\"NewTotalBytesSent\"]),\n int(results2[\"NewTotalBytesReceived\"])]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nexecuting GetInfo action to get WAN connection information s.", "response": "def getConnectionInfo(self, wanInterfaceId=1, timeout=1):\n \"\"\"Execute GetInfo action to get WAN connection information's.\n\n :param int wanInterfaceId: the id of the WAN device\n :param float timeout: the timeout to wait for the action to be executed\n :return: WAN connection information's.\n :rtype: ConnectionInfo\n \"\"\"\n namespace = Wan.getServiceType(\"getConnectionInfo\") + str(wanInterfaceId)\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetInfo\", timeout=timeout)\n\n return ConnectionInfo(results)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef setEnable(self, status, wanInterfaceId=1, timeout=1):\n namespace = Wan.getServiceType(\"setEnable\") + str(wanInterfaceId)\n uri = self.getControlURL(namespace)\n\n if status:\n setStatus = 1\n else:\n setStatus = 0\n\n self.execute(uri, namespace, \"SetEnable\", timeout=timeout, NewEnable=setStatus)", "response": "Set enable status for a WAN interface."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nrequesting the connection to be established", "response": "def requestConnection(self, wanInterfaceId=1, timeout=1):\n \"\"\"Request the connection to be established\n\n :param int wanInterfaceId: the id of the WAN interface\n :param float timeout: the timeout to wait for the action to be executed\n \"\"\"\n namespace = Wan.getServiceType(\"requestConnection\") + str(wanInterfaceId)\n uri = self.getControlURL(namespace)\n\n self.execute(uri, namespace, \"RequestConnection\", timeout=timeout)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nwrites a config object to filename", "response": "def write_config(filename, config, mode=\"w\"):\n '''use configparser to write a config object to filename\n '''\n with open(filename, mode) as filey:\n config.write(filey)\n return filename"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef generate_temporary_file(folder='/tmp', prefix='helpme', ext='json'):\n '''write a temporary file, in base directory with a particular extension.\n \n Parameters\n ==========\n folder: the base directory to write in. \n prefix: the prefix to use\n ext: the extension to use.\n\n ''' \n tmp = next(tempfile._get_candidate_names())\n return '%s/%s.%s.%s' %(folder, prefix, tmp, ext)", "response": "generate a temporary file in base directory with a particular extension"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncopy a file from a source to its destination.", "response": "def copyfile(source, destination, force=True):\n '''copy a file from a source to its destination.\n '''\n if os.path.exists(destination) and force is True:\n os.remove(destination)\n shutil.copyfile(source, destination)\n return destination"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a full inventory with all the keys and values", "response": "def full_inventory(self):\n \"\"\"\n Returns a full inventory\n Some additional work required to provide consistent and consumable\n output.\n Inventory output only contains values, no keys - Add the keys to\n the output so that it can be consumed more easily.\n \"\"\"\n if self._full_inventory:\n return self._full_inventory\n\n resp, inventory = self.get('Inventory')\n\n keys = ['host_id', 'hostname', 'ip_address', 'chassis',\n 'used_count', 'current_state', 'comment', 'distro',\n 'rel', 'centos_version', 'architecture', 'node_pool',\n 'console_port', 'flavor']\n\n real_inventory = dict()\n for host in inventory:\n real_inventory[host[1]] = dict()\n for key in keys:\n real_inventory[host[1]][key] = host[keys.index(key)]\n\n self._full_inventory = real_inventory\n\n return self._full_inventory"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget the self - inventory for the current API key.", "response": "def self_inventory(self):\n \"\"\"\n Inventory output will only contain the server name and the session ID\n when a key is provided. Provide the same format as with the full\n inventory instead for consistency.\n \"\"\"\n if self.api_key is None:\n return {}\n\n if self._self_inventory:\n return self._self_inventory\n\n resp, self_inventory = self.get('Inventory?key=%s' % self.api_key)\n real_self_inventory = dict()\n\n for host in self_inventory:\n real_self_inventory[host[0]] = self.full_inventory[host[0]]\n\n self._self_inventory = real_self_inventory\n\n return self._self_inventory"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfilters an inventory to only return servers matching ssid", "response": "def _ssid_inventory(self, inventory, ssid):\n \"\"\"\n Filters an inventory to only return servers matching ssid\n \"\"\"\n matching_hosts = {}\n for host in inventory:\n if inventory[host]['comment'] == ssid:\n matching_hosts[host] = inventory[host]\n\n return matching_hosts"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef inventory(self, all=False, ssid=None):\n if all or self.api_key is None:\n if ssid is not None:\n return self._ssid_inventory(self.full_inventory, ssid)\n else:\n return self.full_inventory\n else:\n if ssid is not None:\n return self._ssid_inventory(self.self_inventory, ssid)\n else:\n return self.self_inventory", "response": "Returns a node inventory."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef node_get(self, arch=None, ver=None, flavor=None, count=1,\n retry_count=1, retry_interval=10):\n \"\"\"\n Requests specified number of nodes with the provided parameters.\n\n :param arch: Server architecture (ex: x86_64)\n :param ver: CentOS version (ex: 7)\n :param count: Number of servers (ex: 2)\n :parma flavor: The flavor of machine to use (multi-arch only)\n :param retry_count: Number of times to retry in case of failure (ex: 5)\n :param retry_interval: Wait in seconds between each retry (ex: 30)\n :return: [ [ requested_hosts ], ssid ]\n \"\"\"\n if self.api_key is None:\n raise exceptions.ApiKeyRequired\n\n args = \"key=%s\" % self.api_key\n if arch is not None:\n args += \"&arch=%s\" % arch\n if ver is not None:\n args += \"&ver=%s\" % ver\n if flavor is not None:\n args += \"&flavor=%s\" % flavor\n args += \"&count=%s\" % count\n\n resp, body = self.get('Node/get?%s' % args)\n if not body:\n for _ in range(retry_count):\n time.sleep(retry_interval)\n resp, body = self.get('Node/get?%s' % args)\n if body:\n break\n\n if not body:\n raise exceptions.NoInventory\n\n # Get the hosts that were requested.\n # Note: We have to iterate over full inventory instead of just the\n # hosts we got back from the response because the reply contains the\n # fqdn of the host while the full inventory only contains a short name.\n requested_hosts = dict()\n for host in self.full_inventory:\n for full_host in body['hosts']:\n if host in full_host:\n requested_hosts[host] = self.full_inventory[host]\n\n return requested_hosts, body['ssid']", "response": "Get the hosts and ssid of the nodes with the provided parameters."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreleasing the servers for the specified ssid.", "response": "def node_done(self, ssid=None):\n \"\"\"\n Release the servers for the specified ssid.\n The API doesn't provide any kind of output, try to be helpful by\n providing the list of servers to be released.\n\n :param ssid: ssid of the server pool\n :return: [ requested_hosts ]\n \"\"\"\n if self.api_key is None:\n raise exceptions.ApiKeyRequired\n\n if ssid is None:\n raise exceptions.SsidRequired\n\n # There is no body replied in this call so at least get the hosts for\n # the specified ssid to return them.\n requested_hosts = dict()\n for host in self.self_inventory:\n if ssid == self.self_inventory[host]['comment']:\n requested_hosts[host] = self.full_inventory[host]\n\n args = \"key={key}&ssid={ssid}\".format(key=self.api_key, ssid=ssid)\n\n resp, body = self.get('Node/done?%s' % args)\n\n return requested_hosts"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nupdates Redis state for a feature flag.", "response": "def update_redis(project: str, environment: str, feature: str, state: str) \\\n -> None:\n \"\"\"\n Update redis state for a feature flag.\n\n :param project: LaunchDarkly project key.\n :param environment: LaunchDarkly environment key.\n :param feature: LaunchDarkly feature key.\n :param state: State for a feature flag.\n \"\"\"\n try:\n hosts = RedisWrapper.connection_string_parser(\n os.environ.get('REDIS_HOSTS'))\n except RuntimeError as ex:\n LOG.error(ex)\n sys.exit(1)\n\n for host in hosts:\n LOG.info(\"connecting to %s:%s\", host.host, host.port)\n try:\n if valid_state(state):\n new_state = state.lower()\n redis = RedisWrapper(\n host.host,\n host.port,\n project,\n environment\n )\n redis.update_flag_record(new_state, feature)\n create_file(project, environment, feature, new_state)\n LOG.info(\"%s was successfully updated.\", feature)\n else:\n raise Exception('Invalid state: {0}, -s needs \\\n to be either on or off.'.format(state))\n except KeyError as ex:\n LOG.error(\"unable to update %s. Exception: %s\",\n host.host,\n ex)\n sys.exit(1)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nexecutes command against the LaunchDarkly API.", "response": "def update_ld_api(project: str, environment: str, feature: str, state: str):\n \"\"\"\n Execute command against the LaunchDarkly API.\n\n This command is generally not used directly, instead it is called as a\n part of running the ``playback()`` function.\n\n :param project: LaunchDarkly project key.\n :param environment: LaunchDarkly environment key.\n :param feature: LaunchDarkly feature key.\n :param state: State for a feature flag.\n \"\"\"\n ld_api = LaunchDarklyApi(\n os.environ.get('LD_API_KEY'),\n project,\n environment\n )\n\n if valid_state(state):\n if state.lower() == 'off':\n new_state = False\n else:\n new_state = True\n ld_api.update_flag(new_state, feature)\n else:\n raise ValueError('Invalid state: {0}, -s needs to be either \\\n on or off.'.format(state))"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngenerates Relay Proxy Configuration.", "response": "def generate_relay_config(project):\n \"\"\"\n Generate Relay Proxy Configuration.\n\n Generate a ld-relay.conf file to quickly spin up a relay proxy.\n Right now this is mostly used for integration testing.\n\n :param project: LaunchDarkly project key\n \"\"\"\n ld_api = LaunchDarklyApi(\n os.environ.get('LD_API_KEY'),\n project_key=project\n )\n config = ConfigGenerator()\n\n envs = ld_api.get_environments(project)\n config.generate_relay_config(envs)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef dicomdir_info(dirpath, *args, **kwargs):\n dr = DicomReader(dirpath=dirpath, *args, **kwargs)\n info = dr.dicomdirectory.get_stats_of_series_in_dir()\n return info", "response": "Get information about series in dir"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef is_dicom_dir(datapath):\n # Second tries open files\n # with dicom module.\n\n retval = False\n datapath = op.expanduser(datapath)\n for f in os.listdir(datapath):\n if f.endswith((\".dcm\", \".DCM\")):\n retval = True\n return True\n # @todo not working and I dont know why\n try:\n pydicom.read_file(os.path.join(datapath, f))\n\n retval = True\n # except pydicom.errors.InvalidDicomError:\n # logger.debug(\"Invalid Dicom while reading file \" + str(f))\n except Exception as e:\n logger.warning(\"Unable to read dicom file \" + str(f))\n logger.warning(e)\n # import traceback\n # traceback.print_exc()\n\n if retval:\n return True\n return False", "response": "Check if in dir is one or more dicom files."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfunctions generates list of files from specific directory", "response": "def files_in_dir(dirpath, wildcard=\"*\", startpath=None):\n \"\"\"\n Function generates list of files from specific dir\n\n files_in_dir(dirpath, wildcard=\"*.*\", startpath=None)\n\n dirpath: required directory\n wilcard: mask for files\n startpath: start for relative path\n\n Example\n files_in_dir('medical/jatra-kiv','*.dcm', '~/data/')\n \"\"\"\n\n import glob\n\n filelist = []\n\n if startpath is not None:\n completedirpath = os.path.join(startpath, dirpath)\n else:\n completedirpath = dirpath\n\n if os.path.exists(completedirpath):\n logger.info('completedirpath = ' + completedirpath)\n\n else:\n logger.error('Wrong path: ' + completedirpath)\n raise Exception('Wrong path : ' + completedirpath)\n\n for infile in glob.glob(os.path.join(completedirpath, wildcard)):\n filelist.append(infile)\n\n if len(filelist) == 0:\n logger.error('No required files in path: ' + completedirpath)\n raise Exception('No required file in path: ' + completedirpath)\n\n return filelist"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_slice_location(dcmdata, teil=None):\n slice_location = None\n if hasattr(dcmdata, 'SliceLocation'):\n # print(dcmdata.SliceLocation)\n # print(type(dcmdata.SliceLocation))\n try:\n slice_location = float(dcmdata.SliceLocation)\n except Exception as exc:\n logger.info(\"It is not possible to use SliceLocation\")\n logger.debug(traceback.format_exc())\n if slice_location is None and hasattr(dcmdata, \"SliceThickness\") and teil is not None:\n logger.debug(\n \"Estimating SliceLocation wiht image number and SliceThickness\"\n )\n\n # from builtins import map\n i = list(map(int, re.findall('\\d+', teil)))\n i = i[-1]\n try:\n slice_location = float(i * float(dcmdata.SliceThickness))\n except ValueError as e:\n print(type(dcmdata.SliceThickness))\n print(dcmdata.SliceThickness)\n logger.debug(traceback.format_exc())\n logger.debug(\"SliceThickness problem\")\n\n if slice_location is None and hasattr(dcmdata, \"ImagePositionPatient\") and hasattr(dcmdata,\n \"ImageOrientationPatient\"):\n if dcmdata.ImageOrientationPatient == [1, 0, 0, 0, 1, 0]:\n slice_location = dcmdata.ImagePositionPatient[2]\n else:\n logger.warning(\"Unknown ImageOrientationPatient\")\n if slice_location is None:\n logger.warning(\"Problem with slice location\")\n\n return slice_location", "response": "get location of the slice in a single object"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nadding attribute to dict if it exists.", "response": "def attr_to_dict(obj, attr, dct):\n \"\"\"\n Add attribute to dict if it exists.\n :param dct:\n :param obj: object\n :param attr: object attribute name\n :return: dict\n \"\"\"\n if hasattr(obj, attr):\n dct[attr] = getattr(obj, attr)\n return dct"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_series_number_by_guess_for_liver(dcmreader, counts, bins, qt_app=None):\n series_info = dcmreader.dicomdirectory.get_stats_of_series_in_dir()\n print(dcmreader.print_series_info(series_info))\n import pandas as pd\n df = pd.DataFrame(list(series_info.values()))\n\n #select CT\n df = df[df[\"Modality\"].str.lower().str.contains(\"ct\") == True]\n # select just venous\n df = df[df[\"SeriesDescription\"].str.lower().str.contains(\"ven\") == True]\n # remove saggittal\n df = df[df[\"SeriesDescription\"].str.lower().str.contains(\"sag\") == False]\n # remove cor\n df = df[df[\"SeriesDescription\"].str.lower().str.contains(\"cor\") == False]\n df[\"dst_to_200\"] = np.abs(200 - df.Count)\n dfs = df.sort_values(by=\"dst_to_200\", ascending=True)\n sn = list(dfs.SeriesNumber)[0]\n\n return sn", "response": "Select the series number that is used for liver."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfunctions make 3D data from dicom file slices.", "response": "def get_overlay(self):\n \"\"\"\n Function make 3D data from dicom file slices. There are usualy\n more overlays in the data.\n \"\"\"\n overlay = {}\n dcmlist = self.files_in_serie\n\n for i in range(len(dcmlist)):\n onefile = dcmlist[i]\n logger.info(\"reading '%s'\" % onefile)\n data = self._read_file(onefile)\n\n if len(overlay) == 0:\n # first there is created dictionary with\n # avalible overlay indexes\n for i_overlay in range(0, 50):\n try:\n # overlay index\n data2d = decode_overlay_slice(data, i_overlay)\n # mport pdb; pdb.set_trace()\n shp2 = data2d.shape\n overlay[i_overlay] = np.zeros([len(dcmlist), shp2[0],\n shp2[1]], dtype=np.int8)\n overlay[i_overlay][-i - 1, :, :] = data2d\n\n except Exception:\n # exception is exceptetd. We are trying numbers 0-50\n # logger.exception('Problem with overlay image number ' +\n # str(i_overlay))\n pass\n\n else:\n for i_overlay in overlay.keys():\n try:\n data2d = decode_overlay_slice(data, i_overlay)\n overlay[i_overlay][-i - 1, :, :] = data2d\n except Exception:\n logger.warning('Problem with overlay number ' +\n str(i_overlay))\n\n return overlay"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfunctions make 3D data from dicom files", "response": "def get_3Ddata(self, start=0, stop=None, step=1):\n \"\"\"\n Function make 3D data from dicom file slices\n \"\"\"\n data3d = []\n dcmlist = self.files_in_serie\n # print('stsp ', start, stop, step)\n\n # raw_max = None\n # raw_min = None\n # slope = None\n # inter = None\n\n # get shape 2d\n\n # sometimes there is render in series\n if len(self.files_in_serie) > 1:\n data = self._read_file(dcmlist[0])\n data2d1 = data.pixel_array\n data = self._read_file(dcmlist[1])\n data2d2 = data.pixel_array\n if (data2d1.shape[0] == data2d2.shape[0]) and (data2d1.shape[1] == data2d2.shape[1]):\n pass\n else:\n dcmlist.pop(0)\n\n if stop is None:\n stop = len(dcmlist)\n\n # printRescaleWarning = False\n for i in xrange(start, stop, step):\n onefile = dcmlist[i]\n data = self._read_file(onefile)\n new_data2d = data.pixel_array\n # new_data2d, slope, inter = dcmtools.get_pixel_array_from_pdcm(data)\n # mport pdb; pdb.set_trace()\n\n if len(data3d) == 0:\n shp2 = new_data2d.shape\n data3d = np.zeros([len(dcmlist), shp2[0], shp2[1]],\n dtype=new_data2d.dtype)\n slope, inter = dcmtools.get_slope_and_intercept_from_pdcm(data)\n\n\n\n # first readed slide is at the end\n\n if (data3d.shape[1] == new_data2d.shape[0]) and (data3d.shape[2] == new_data2d.shape[1]):\n data3d[-i - 1, :, :] = new_data2d\n else:\n msg = \"Problem with shape \" + \\\n \"Data size: \" + str(data3d.nbytes) + \\\n ', shape: ' + str(shp2) + 'x' + str(len(dcmlist)) + \\\n ' file ' + onefile\n logger.warning(msg)\n print(msg)\n\n logger.debug(\"Data size: \" + str(data3d.nbytes)\n + ', shape: ' + str(shp2) + 'x' + str(len(dcmlist))\n + ' file ' + onefile)\n data3d = misc.use_economic_dtype(data3d, slope=slope, inter=inter)\n # if original_dtype == np.uint16 and data3d.dtype == np.int16:\n # data3d = data3d.astype(np.int32)\n # or just force set slope=0.5, inter = 0\n # new_data2d = rescale_pixel_array(data2d, slope, inter)\n # if printRescaleWarning:\n # print(\"Automatic Rescale with slope 0.5\")\n # logger.warning(\"Automatic Rescale with slope 0.5\")\n # data3d = dcmtools.rescale_pixel_array(data3d, slope=slope, inter=inter)\n\n return data3d"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_standard_dicomdir_info(self):\n dicomdir_filepath = os.path.join(self.dirpath, self.standard_dicomdir_filename)\n if not os.path.exists(dicomdir_filepath):\n self.create_standard_dicomdir()\n return self.read_standard_dicomdir_info()", "response": "Read DICOMDIR crate if necessary."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_metadata_new(self, series_number):\n # TODO implement simplier metadata function\n # automatic test is prepared\n\n files, files_with_info = self.get_sorted_series_files(series_number=series_number, return_files_with_info=True)\n metadata = {\n # 'voxelsize_mm': voxelsize_mm,\n # 'Modality': data1.Modality,\n # 'SeriesNumber': series_number,\n # 'SeriesDescription' = data1.SeriesDescription,\n # 'ImageComments' : data1.ImageComments,\n # \"AcquisitionDate\": metadata,\n # \"StudyDate\": metadata,\n # \"StudyDescription\": metadata,\n # \"RequestedProcedureDescription\", metadata\n }\n return metadata", "response": "Return series metadata.\n Output condatin information about voxelsize_mm, series_number and modality.\n If it is possible, the ImageComment, AcquisitionDate and few other dicom tags are also in output dict.\n :param series_number:\n :return: metadata dict with voxelsize_mm, SeriesNumber and other dicom tags"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting metadata for a neighboorhoding slice.", "response": "def get_metaData(self, dcmlist, series_number):\n \"\"\"\n Get metadata.\n Voxel size is obtained from PixelSpacing and difference of\n SliceLocation of two neighboorhoding slices (first have index ifile).\n Files in are used.\n \"\"\"\n # if dcmlist is None:\n # dcmlist = self.files_in_serie\n\n # number of slice where to extract metadata inforamtion\n ifile = 0\n if len(dcmlist) == 0:\n return {}\n\n logger.debug(\"Filename: \" + dcmlist[ifile])\n data1 = self._read_file(dcmlist[ifile])\n try:\n # try to get difference from the beginning and also from the end\n voxeldepth = self._get_slice_location_difference(dcmlist, ifile)\n voxeldepth_end = self._get_slice_location_difference(dcmlist, -2)\n if voxeldepth != voxeldepth_end:\n logger.warning(\"Depth of slices is not the same in beginning and end of the sequence\")\n voxeldepth_1 = self._get_slice_location_difference(dcmlist, 1)\n voxeldepth = np.median([voxeldepth, voxeldepth_end, voxeldepth_1])\n\n\n\n # head1, teil1 = os.path.split(dcmlist[ifile])\n # head2, teil2 = os.path.split(dcmlist[ifile + 1])\n #\n # data2 = self._read_file(dcmlist[ifile + 1])\n # loc1 = get_slice_location(data1, teil1)\n # loc2 = get_slice_location(data2, teil2)\n # voxeldepth = float(np.abs(loc1 - loc2))\n except Exception:\n logger.warning('Problem with voxel depth. Using SliceThickness')\n logger.debug(traceback.format_exc())\n # + ' SeriesNumber: ' + str(data1.SeriesNumber))\n\n try:\n voxeldepth = float(data1.SliceThickness)\n except Exception:\n logger.warning('Probem with SliceThicknes, setting zero. '\n + traceback.format_exc())\n voxeldepth = 0\n\n try:\n pixelsize_mm = data1.PixelSpacing\n except:\n logger.warning('Problem with PixelSpacing. Using [1,1]')\n pixelsize_mm = [1, 1]\n voxelsize_mm = [\n voxeldepth,\n float(pixelsize_mm[0]),\n float(pixelsize_mm[1]),\n ]\n metadata = {'voxelsize_mm': voxelsize_mm,\n 'Modality': data1.Modality,\n 'SeriesNumber': series_number\n }\n\n try:\n metadata['SeriesDescription'] = data1.SeriesDescription\n\n except:\n logger.info(\n 'Problem with tag SeriesDescription, SeriesNumber: ' +\n str(data1.SeriesNumber))\n try:\n metadata['ImageComments'] = data1.ImageComments\n except:\n logger.info(\n 'Problem with tag ImageComments')\n # , SeriesNumber: ' +\n # str(data1.SeriesNumber))\n try:\n metadata['Modality'] = data1.Modality\n except:\n logger.info(\n 'Problem with tag Modality')\n # SeriesNumber: ' +\n # str(data1.SeriesNumber))\n metadata = attr_to_dict(data1, \"AcquisitionDate\", metadata)\n metadata = attr_to_dict(data1, \"StudyDate\", metadata)\n metadata = attr_to_dict(data1, \"StudyID\", metadata)\n metadata = attr_to_dict(data1, \"StudyDescription\", metadata)\n metadata = attr_to_dict(data1, \"RequestedProcedureDescription\", metadata)\n metadata = attr_to_dict(data1, \"PatientSex\", metadata)\n metadata = attr_to_dict(data1, \"PatientAge\", metadata)\n metadata = attr_to_dict(data1, \"PatientID\", metadata)\n metadata = attr_to_dict(data1, \"PatientName\", metadata)\n # metadata = attr_to_dict(data1, \"AcquisitionTime\", metadata)\n\n metadata['dcmfilelist'] = dcmlist\n return metadata"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_stats_of_series_in_dir(self, study_id=None):\n if study_id is not None:\n logger.error(\"study_id tag is not implemented yet\")\n return\n import numpy as np\n dcmdir = self.files_with_info\n # get series number\n # vytvo\u0159en\u00ed slovn\u00edku, kde je kl\u00ed\u010dem \u010d\u00edslo s\u00e9rie a hodnotou jsou v\u0161echny\n # informace z dicomdir\n series_info = {line['SeriesNumber']: line for line in dcmdir}\n\n # po\u010d\u00edt\u00e1n\u00ed velikosti s\u00e9rie\n try:\n dcmdirseries = [line['SeriesNumber'] for line in dcmdir]\n\n except:\n logger.debug('Dicom tag SeriesNumber not found')\n series_info = {0: {'Count': 0}}\n return series_info\n # eturn [0],[0]\n\n bins, counts = np.unique(dcmdirseries, return_counts=True)\n\n # sestaven\u00ed informace o velikosti s\u00e9rie a slovn\u00edku\n\n for i in range(0, len(bins)):\n series_info[bins[i]]['Count'] = counts[i]\n\n # adding information from files\n lst = self.get_sorted_series_files(series_number=bins[i])\n metadata = self.get_metaData(dcmlist=lst, series_number=bins[i])\n # adding dictionary metadata to series_info dictionary\n series_info[bins[i]] = dict(\n list(series_info[bins[i]].items()) +\n list(metadata.items())\n )\n\n return series_info", "response": "get series staticstics from dicomdir"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef print_series_info(self, series_info, minimal_series_number=1):\n strinfo = ''\n if len(series_info) > minimal_series_number:\n for serie_number in series_info.keys():\n strl = get_one_serie_info(series_info, serie_number)\n strinfo = strinfo + strl + '\\n'\n # rint strl\n\n return strinfo", "response": "Print series_info from dcmdirstats\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nprepare info from dicomdir file", "response": "def __prepare_info_from_dicomdir_file(self, writedicomdirfile=True):\n \"\"\"\n Check if exists dicomdir file and load it or cerate it\n\n dcmdir = get_dir(dirpath)\n\n dcmdir: list with filenames, SeriesNumber and SliceLocation\n \"\"\"\n createdcmdir = True\n\n dicomdirfile = os.path.join(self.dirpath, self.dicomdir_filename)\n ftype = 'pickle'\n # if exist dicomdir file and is in correct version, use it\n if os.path.exists(dicomdirfile):\n try:\n dcmdirplus = misc.obj_from_file(dicomdirfile, ftype)\n if dcmdirplus['version'] == __version__:\n createdcmdir = False\n dcmdir = dcmdirplus['filesinfo']\n except Exception:\n logger.debug('Found dicomdir.pkl with wrong version')\n createdcmdir = True\n\n if createdcmdir or self.force_create_dicomdir:\n dcmdirplus = self._create_dicomdir_info()\n dcmdir = dcmdirplus['filesinfo']\n if (writedicomdirfile) and len(dcmdir) > 0:\n # obj_to_file(dcmdirplus, dicomdirfile, ftype)\n try:\n misc.obj_to_file(dcmdirplus, dicomdirfile, ftype)\n except:\n logger.warning('Cannot write dcmdir file')\n traceback.print_exc()\n\n # bj_to_file(dcmdir, dcmdiryamlpath )\n\n dcmdir = dcmdirplus['filesinfo']\n self.dcmdirplus = dcmdirplus\n self.files_with_info = dcmdir\n return dcmdir"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef series_in_dir(self):\n\n # none_count = 0\n countsd = {}\n # dcmdirseries = []\n for line in self.files_with_info:\n if \"SeriesNumber\" in line:\n sn = line['SeriesNumber']\n else:\n sn = None\n if sn in countsd:\n countsd[sn] += 1\n else:\n countsd[sn] = 1\n\n bins = list(countsd)\n counts = list(countsd.values())\n\n\n # try:\n # dcmdirseries = [line['SeriesNumber'] for line in self.files_with_info]\n # except:\n # return [0], [0]\n\n # bins, counts = np.unique(dcmdirseries, return_counts=True)\n # binslist = bins.tolist()\n\n # if None in binslist:\n # if len(binslist) == 1:\n # return [0], [0]\n # else:\n # logger.warning\n\n # kv\u016fli spr\u00e1vn\u00fdm interval\u016fm mezi biny je nutno jeden p\u0159idat na konce\n # mxb = np.max(bins)\n # if mxb is None:\n # mxb = 1\n # else:\n # mxb = mxb + 1\n #\n # binslist.append(mxb)\n # counts, binsvyhodit = np.histogram(dcmdirseries, bins=binslist)\n\n # return counts.tolist(), bins.tolist()\n return counts, bins", "response": "return list of series names and counts of all files in dirpath"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_sorted_series_files(self, startpath=\"\", series_number=None, return_files_with_info=False,\n sort_keys=\"SliceLocation\", return_files=True, remove_doubled_slice_locations=True):\n \"\"\"\n Function returns sorted list of dicom files. File paths are organized\n by SeriesUID, StudyUID and FrameUID\n\n\n :param startpath: path prefix. E.g. \"~/data\"\n :param series_number: ID of series used for filtering the data\n :param return_files_with_info: return more complex information about sorted files\n :param return_files: return simple list of sorted files\n :type sort_keys: One key or list of keys used for sorting method by the order of keys.\n\n \"\"\"\n dcmdir = self.files_with_info[:]\n\n # select sublist with SeriesNumber\n if series_number is not None:\n dcmdir = [\n line for line in dcmdir if line['SeriesNumber'] == series_number\n ]\n dcmdir = sort_list_of_dicts(dcmdir, keys=sort_keys)\n\n logger.debug('SeriesNumber: ' + str(series_number))\n\n if remove_doubled_slice_locations:\n dcmdir = self._remove_doubled_slice_locations(dcmdir)\n\n filelist = []\n for onefile in dcmdir:\n filelist.append(os.path.join(startpath,\n self.dirpath, onefile['filename']))\n # head, tail = os.path.split(onefile['filename'])\n\n\n retval = []\n if return_files:\n retval.append(filelist)\n if return_files_with_info:\n retval.append(dcmdir)\n\n if len(retval) == 0:\n retval = None\n elif len(retval) == 1:\n retval = retval[0]\n else:\n retval = tuple(retval)\n\n return retval", "response": "Function returns sorted list of dicom files."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _create_dicomdir_info(self):\n\n filelist = files_in_dir(self.dirpath)\n files = []\n metadataline = {}\n\n for filepath in filelist:\n head, teil = os.path.split(filepath)\n dcmdata = None\n if os.path.isdir(filepath):\n logger.debug(\"Subdirectory found in series dir is ignored: \" + str(filepath))\n continue\n try:\n dcmdata = pydicom.read_file(filepath)\n\n except pydicom.errors.InvalidDicomError as e:\n # some files doesnt have DICM marker\n try:\n dcmdata = pydicom.read_file(filepath, force=self.force_read)\n\n # if e.[0].startswith(\"File is missing \\\\'DICM\\\\' marker. Use force=True to force reading\")\n except Exception as e:\n if teil != self.dicomdir_filename:\n # print('Dicom read problem with file ' + filepath)\n logger.info('Dicom read problem with file ' + filepath)\n import traceback\n logger.debug(traceback.format_exc())\n if hasattr(dcmdata, \"DirectoryRecordSequence\"):\n # file is DICOMDIR - metainfo about files in directory\n # we are not using this info\n dcmdata = None\n\n if dcmdata is not None:\n metadataline = _prepare_metadata_line(dcmdata, teil)\n files.append(metadataline)\n\n # if SliceLocation is None, it is sorted to the end\n # this is not necessary it can be deleted\n files.sort(key=lambda x: (x['SliceLocation'] is None, x[\"SliceLocation\"]))\n\n dcmdirplus = {'version': __version__, 'filesinfo': files, }\n if \"StudyDate\" in metadataline:\n dcmdirplus[\"StudyDate\"] = metadataline[\"StudyDate\"]\n return dcmdirplus", "response": "Function crates list of all files in dicom dir with all IDs"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn one item from seq or None by default.", "response": "def get_one(seq, default=None, skip_string_iter=True):\n \"\"\"\n Return one item from seq or None(by default).\n \"\"\"\n if skip_string_iter and isinstance(seq, (str, unicode, bytes, bytearray)):\n return seq\n if not seq:\n return ''\n try:\n return next(iter(seq))\n except TypeError:\n # not hasattr __iter__/__getitem__\n return default"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nensuring that the object is a list of the available cache entries.", "response": "def ensure_list(obj):\n \"\"\"\n null obj -> return []; \n\n str, unicode, bytes, bytearray -> [obj];\n\n else -> list(obj)\n \"\"\"\n if not obj:\n return []\n elif isinstance(obj, (str, unicode, bytes, bytearray)):\n return [obj]\n elif hasattr(obj, '__iter__') or hasattr(obj, '__getitem__'):\n return list(obj)\n else:\n return [obj]"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\noperates a python obj", "response": "def python_parser(self, obj, *args):\n \"\"\"operate a python obj\"\"\"\n attr, args = args[0], args[1:]\n item = getattr(obj, attr)\n if callable(item):\n item = item(*args)\n return [item]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef html_parser(self, scode, *args):\n allow_method = ('text', 'html', 'xml')\n css_path, method = args\n assert method in allow_method or method.startswith(\n '@'), 'method allow: %s or @attr' % allow_method\n result = self.ensure_list(\n self._fromstring(scode,\n parser=self._html_parser).cssselect(css_path))\n if method.startswith('@'):\n result = [item.get(method[1:]) for item in result]\n else:\n result = [\n self._tostring(\n item, method=method, with_tail=0, encoding='unicode')\n for item in result\n ]\n return result", "response": "parse a single HTML code into a list of items"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef xml_parser(self, scode, *args):\n allow_method = ('text', 'html', 'xml')\n xpath_string, method = args\n assert method in allow_method, 'method allow: %s' % allow_method\n result = self.ensure_list(\n self._fromstring(scode,\n parser=self._xml_parser).xpath(xpath_string))\n result = [\n self._tostring(\n item, method=method, with_tail=0, encoding='unicode')\n for item in result\n ]\n return result", "response": "Parse the XML code."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nparsing the scode and return a list of objects.", "response": "def parse(self, scode, args_chain=None, join_with=None, default=''):\n \"\"\"\n single arg:\n [one_to_many, parser_name, *args]\n args_chain: \n [['1-n', 're', 'search', '(<.*?>)', '\\\\1']]\n [['1-n', 'html', 'p', 'html'], ['n-n', 'html', 'p', 'text']]\n \"\"\"\n assert args_chain and isinstance(\n args_chain, (list, tuple)\n ) and isinstance(args_chain[0], (list, tuple)), ValueError(\n 'args_chain type should be list of list, like: [[\"1-n\", \"html\", \"p\", \"html\"], [\"n-n\", \"html\", \"p\", \"text\"]].'\n )\n for arg in args_chain:\n # py2 not support * unpack\n one_to_many, parser_name, parse_args = (arg[0], arg[1], arg[2:])\n assert self._re.match(\n '^[1n]-[1n]$',\n one_to_many), 'one_to_many should be one of 1-1, 1-n, n-n, n-1'\n input_count, output_count = one_to_many.split('-')\n parser = self._choose_parser(parser_name)\n # input data to parse.\n if input_count == 'n':\n scode = list(map(lambda item: parser(item, *parse_args), scode))\n if input_count == '1':\n if parser not in (self.jsonpath_parser, self.objectpath_parser,\n self.python_parser):\n # json may remain multi-items\n scode = get_one(scode, default=default)\n scode = parser(scode, *parse_args)\n # ensure result match n or 1 after parsing.\n if parser in (self.objectpath_parser,):\n # objectpath not need\n continue\n if output_count == '1':\n # 1-1 or n-1\n scode = get_one(scode, default=default)\n elif input_count == 'n':\n # n-n\n scode = [get_one(i, default=default) for i in scode]\n else:\n # 1-n\n scode = list(scode)\n if join_with:\n scode = join_with.join(map(str, scode))\n return scode"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef parse(self):\n # Build the HTML tree\n self.tree = self._build_tree(self.html_contents)\n\n # Parse the CSS contents\n self.stylesheet = self.parser.parse_stylesheet(self.css_contents)\n\n # Get the cleaned CSS contents\n self.cleaned_css = self._clean_css()", "response": "Parses the CSS contents and returns the cleaned CSS as a string"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef rel_to_abs(self, base_url):\n self.cleaned_css = self.rel_to_abs_re.sub(\n lambda match: \"url('%s')\" % urljoin(\n base_url, match.group('path').strip('\\'\"')),\n self.cleaned_css)", "response": "Converts relative links from css contents to absolute links\n ."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nclean the CSS rules and contents of the tree and return the cleaned CSS", "response": "def _clean_css(self):\n \"\"\"\n Returns the cleaned CSS\n\n :param stylesheet: The Stylesheet object to parse\n :type stylesheet: tinycss.css21.Stylesheet\n \"\"\"\n # Init the cleaned CSS rules and contents string\n css_rules = []\n\n # For every rule in the CSS\n for rule in self.stylesheet.rules:\n\n try:\n # Clean the CSS rule\n cleaned_rule = self._clean_rule(rule)\n\n # Append the rule to matched CSS rules\n if cleaned_rule is not None:\n css_rules.append(cleaned_rule)\n\n except:\n # On error, assume the rule matched the tree\n css_rules.append(rule)\n\n return self._build_css(css_rules)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _token_list_matches_tree(self, token_list):\n try:\n parsed_selector = cssselect.parse(\n ''.join(token.as_css() for token in token_list))[0]\n\n return bool(\n self.tree.xpath(\n self.xpath_translator.selector_to_xpath(parsed_selector)))\n except:\n # On error, assume the selector matches the tree\n return True", "response": "Returns True if the token list matches the HTML tree"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts a tinycss rule to a formatted CSS string", "response": "def _rule_as_string(self, rule):\n \"\"\"\n Converts a tinycss rule to a formatted CSS string\n\n :param rule: The rule to format\n :type rule: tinycss Rule object\n :returns: The Rule as a CSS string\n :rtype: str\n \"\"\"\n if isinstance(rule, RuleSet):\n # Simple CSS rule : a { color: red; }\n return '%s{%s}' % (\n self._selector_as_string(rule.selector),\n self._declarations_as_string(rule.declarations))\n\n elif isinstance(rule, ImportRule):\n # @import rule\n return \"@import url('%s') %s;\" % (\n rule.uri, ','.join(rule.media))\n\n elif isinstance(rule, FontFaceRule):\n # @font-face rule\n return \"@font-face{%s}\" % self._declarations_as_string(rule.declarations)\n\n elif isinstance(rule, MediaRule):\n # @media rule\n return \"@media %s{%s}\" % (\n ','.join(rule.media),\n ''.join(self._rule_as_string(r) for r in rule.rules))\n\n elif isinstance(rule, PageRule):\n # @page rule\n selector, pseudo = rule.selector\n\n return \"@page%s%s{%s}\" % (\n ' %s' % selector if selector else '',\n ' :%s' % pseudo if pseudo else '',\n self._declarations_as_string(rule.declarations))\n\n return ''"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a selector as a CSS string for the selector", "response": "def _selector_as_string(self, selector):\n \"\"\"\n Returns a selector as a CSS string\n\n :param selector: A list of tinycss Tokens\n :type selector: list\n :returns: The CSS string for the selector\n :rtype: str\n \"\"\"\n return ','.join(\n ''.join(token.as_css() for token in strip_whitespace(token_list))\n for token_list in split_on_comma(selector))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _declarations_as_string(self, declarations):\n return ''.join('%s:%s%s;' % (\n d.name,\n d.value.as_css(),\n ' !' + d.priority if d.priority else '') for d in declarations)", "response": "Returns a list of declarations as a formatted CSS string for the keys in the list."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngenerate a markdown list of a list of environment variables", "response": "def envars_to_markdown(envars, title = \"Environment\"):\n '''generate a markdown list of a list of environment variable tuples\n\n Parameters\n ==========\n title: A title for the section (defaults to \"Environment\"\n envars: a list of tuples for the environment, e.g.:\n\n [('TERM', 'xterm-256color'),\n ('SHELL', '/bin/bash'),\n ('USER', 'vanessa'),\n ('LD_LIBRARY_PATH', ':/usr/local/pulse')]\n\n '''\n markdown = ''\n if envars not in [None, '', []]:\n markdown += '\\n## %s\\n' % title\n for envar in envars:\n markdown += ' - **%s**: %s\\n' %(envar[0], envar[1])\n return markdown"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\nasync def create_vm(self, preset_name: str, image: str, flavor: str, security_groups: List=None,\n userdata: Dict=None, key_name: str=None, availability_zone: str=None,\n subnets: List=None) -> Any:\n \"\"\"\n Create (boot) a new server.\n\n :arg string preset_name: Name of vm group where vm is created.\n :arg string image: Image name.\n :arg string flavor: Flavor (or instance_type in AWS) name.\n :arg list security_groups: A list of security group names.\n :arg dict userdata: A dict of arbitrary key/value metadata to store in grains.\n :arg string key_name: (optional extension) name of previously created\n keypair to inject into the instance.\n :arg string availability_zone: Name of the availability zone for instance\n placement.\n :arg string subnets: List of the subnets for instance placement.\n\n Returns Any vm_id.\n \"\"\"\n\n raise NotImplementedError", "response": "Create a new server."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_default_commands(self):\n commands = Application.get_default_commands(self)\n\n self.add(ConstantsCommand())\n self.add(LoaderCommand())\n self.add(PyStratumCommand())\n self.add(WrapperCommand())\n\n return commands", "response": "Returns the default commands of this application."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef discover(service=\"ssdp:all\", timeout=1, retries=2, ipAddress=\"239.255.255.250\", port=1900):\n\n socket.setdefaulttimeout(timeout)\n\n messages = []\n\n if isinstance(service, str):\n services = [service]\n elif isinstance(service, list):\n services = service\n\n for service in services:\n message = 'M-SEARCH * HTTP/1.1\\r\\nMX: 5\\r\\nMAN: \"ssdp:discover\"\\r\\nHOST: ' + \\\n ipAddress + ':' + str(port) + '\\r\\n'\n message += \"ST: \" + service + \"\\r\\n\\r\\n\"\n messages.append(message)\n\n responses = {}\n\n for _ in range(retries):\n\n # setup the socket\n sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)\n sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 2)\n\n # noinspection PyAssignmentToLoopOrWithParameter\n for _ in range(2):\n\n # send the messages with different service types\n for message in messages:\n # send message more often to make sure all devices will get it\n sock.sendto(message.encode('utf-8'), (ipAddress, port))\n\n while True:\n try:\n # read the message until timeout\n data = sock.recv(1024)\n except socket.timeout:\n break\n else:\n # no time out, read the response data and create response object\n response = DiscoveryResponse(data)\n # filter duplicated responses\n responses[response.location] = response\n\n # return a list of all responses\n return list(responses.values())", "response": "Discover UPnP devices in the local network."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef discoverParticularHost(host, service=\"ssdp:all\", deviceDefinitionURL=None, timeout=1, retries=2,\n ipAddress=\"239.255.255.250\", port=1900, proxies=None):\n \"\"\"Discover a particular host and find the best response.\n\n This tries to find the most specific discovery result for the given host. Only the discovery result contains\n the URL to the XML tree which initializes the device definition. If an URL is already known it should be\n provided to avoid additional latency for a broader first device discovery.\n This method also do some magic to find the best result for the given host as UPnP devices behave sometimes\n strangely. This call is costly the result if any should be cached.\n\n :param str host: the host to find\n :param service: the service type or list of service types if known to search for\n :type service: str or list[str]\n :param str deviceDefinitionURL: if provided it is used to skip a first device discovery\n :param float timeout: the time to wait for each retry\n :param int retries: the amount of times how often the device is tried to discover\n :param str ipAddress: the multicast ip address to discover devices\n :param int port: the port to discover devices\n :param str proxies: proxy definition as defined here:\n `Proxy definition `_\n :return: If the device have been found the response is returned otherwise None\n :rtype: DiscoveryResponse\n :raises ValueError: if problems with reading or parsing the xml device definition occurs\n :raises requests.exceptions.ConnectionError: when the device definitions can not be downloaded\n :raises requests.exceptions.ConnectTimeout: when download time out\n\n Example:\n ::\n\n proxies = {\"http\": \"http://localhost:8888\"}\n result = discoverParticularHost(\"192.168.0.1\", proxies=proxies)\n if result is not None:\n print(\"Host: \" + result.locationHost + \" Port: \" + result.locationPort + \" Device definitions: \" + \\\\\n result.location)\n\n .. seealso::\n\n :class:`~simpletr64.DiscoveryResponse`, :meth:`~simpletr64.Discover.discover`\n \"\"\"\n\n # get all IP addresses for the given host\n ipResults = socket.getaddrinfo(host, 80)\n\n if len(ipResults) == 0:\n return None\n\n ipAddresses = []\n\n # remember all ip addresses for the given host\n for ipAdrTupple in ipResults:\n ipAddresses.append(ipAdrTupple[4][0])\n\n bestPick = None\n services = []\n\n if deviceDefinitionURL is None:\n # no xml definition given, so lets search for one\n\n # search for all devices first\n discoverResults = Discover.discover(service=service, timeout=timeout, retries=retries,\n ipAddress=ipAddress, port=port)\n\n for result in discoverResults:\n if result.locationHost in ipAddresses:\n # now we found a result for that host, pick the best service type if multiple results for the host\n # are found\n if Discover.rateServiceTypeInResult(result) > Discover.rateServiceTypeInResult(bestPick):\n bestPick = result\n\n # remember all services\n if result.service not in services:\n services.append(result.service)\n\n if bestPick is None:\n return None\n else:\n # create response with given parameter\n bestPick = DiscoveryResponse.create(deviceDefinitionURL, service=service)\n\n # some routers do not advice their TR64 capabilities but their UPnp which is only a subset of actions.\n # Try to find out if the given XML definition path will give us a better service type.\n # load xml definition\n\n # some devices response differently without a User-Agent\n headers = {\"User-Agent\": \"Mozilla/5.0; SimpleTR64-3\"}\n\n request = requests.get(bestPick.location, proxies=proxies, headers=headers, timeout=float(timeout))\n\n if request.status_code != 200:\n errorStr = DeviceTR64._extractErrorString(request)\n raise ValueError('Could not get CPE definitions for \"' + bestPick.location + '\": ' +\n str(request.status_code) + ' - ' + request.reason + \" -- \" + errorStr)\n\n # parse xml\n try:\n root = ET.fromstring(request.text.encode('utf-8'))\n except Exception as e:\n raise ValueError(\"Could not parse CPE definitions for '\" + bestPick.location + \"': \" + str(e))\n\n # find the first deviceType in the document tree\n for element in root.getiterator():\n # check if element tag name ends on deviceType, skip XML namespace\n if element.tag.lower().endswith(\"devicetype\"):\n\n serviceFound = element.text\n\n # remember the service found if it does not exist yet\n if serviceFound not in services:\n services.append(serviceFound)\n\n # create a specific service just to check if we found it already\n serviceFound = serviceFound.replace(\"schemas-upnp-org\", \"dslforum-org\")\n\n # test if we already have the best service type then we dont need to do an other discovery request\n if serviceFound == bestPick.service:\n return bestPick\n\n for service in services:\n # we search for the specific device tyoe version as of specified in TR64 protocol.\n # some devices returns different results depending on the given service type, so lets be\n # very specific\n specificService = service.replace(\"schemas-upnp-org\", \"dslforum-org\")\n\n if specificService not in services:\n services.append(specificService)\n\n # we do an other discovery request with more specific service/device type\n discoverResultsSpecific = Discover.discover(service=services, timeout=float(timeout), retries=retries,\n ipAddress=ipAddress, port=port)\n\n # iterate through all results to find the most specific one\n evenBetterPick = None\n\n for specificResult in discoverResultsSpecific:\n if specificResult.locationHost in ipAddresses:\n if Discover.rateServiceTypeInResult(specificResult) > \\\n Discover.rateServiceTypeInResult(evenBetterPick):\n evenBetterPick = specificResult\n\n if evenBetterPick is not None:\n # best we could find\n return evenBetterPick\n\n # we found first deviceType tag in the XML structure, no need to go further\n break\n\n if deviceDefinitionURL is not None:\n # we created our own response, so no result found\n return None\n\n # we found only an unspecific result, return it anyway\n return bestPick", "response": "Discover a particular host and find the most specific response."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrating the service type in a result.", "response": "def rateServiceTypeInResult(discoveryResponse):\n \"\"\"Gives a quality rating for a given service type in a result, higher is better.\n\n Several UpnP devices reply to a discovery request with multiple responses with different service type\n announcements. To find the most specific one we need to be able rate the service types against each other.\n Usually this is an internal method and just exported for convenience reasons.\n\n :param DiscoveryResponse discoveryResponse: the response to rate\n :return: a rating of the quality of the given service type\n :rtype: int\n \"\"\"\n if discoveryResponse is None:\n return 0\n\n serviceType = discoveryResponse.service\n\n if serviceType.startswith(\"urn:dslforum-org:device\"):\n return 11\n if serviceType.startswith(\"urn:dslforum-org:service\"):\n return 10\n if serviceType.startswith(\"urn:dslforum-org:\"):\n return 9\n if serviceType.startswith(\"urn:schemas-upnp-org:device\"):\n return 8\n if serviceType.startswith(\"urn:schemas-upnp-org:service\"):\n return 7\n if serviceType.startswith(\"urn:schemas-upnp-org:\"):\n return 6\n if serviceType.startswith(\"urn:schemas-\"): # other schemas, schema-any-com for example\n return 5\n if serviceType.startswith(\"urn:\"):\n return 4\n if serviceType.startswith(\"upnp:rootdevice\"):\n return 3\n if serviceType.startswith(\"uuid:\"): # no service, just the uuid given\n return 2\n return 1"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndownloads the image from the url and return the name of the downloaded file", "response": "def download(self, url,\n file_name,\n headers=None,\n show_progress=True):\n\n '''stream to a temporary file, rename on successful completion\n\n Parameters\n ==========\n file_name: the file name to stream to\n url: the url to stream from\n headers: additional headers to add\n force: If the final image exists, don't overwrite\n\n '''\n\n fd, tmp_file = tempfile.mkstemp(prefix=(\"%s.tmp.\" % file_name)) \n os.close(fd)\n\n # Should we verify the request?\n verify = self._verify()\n\n # Check here if exists\n if requests.head(url, verify=verify).status_code in [200, 401]:\n response = self.stream(url, headers=headers, stream_to=tmp_file)\n\n if isinstance(response, HTTPError):\n bot.error(\"Error downloading %s, exiting.\" %url)\n sys.exit(1)\n shutil.move(tmp_file, file_name)\n else:\n bot.error(\"Invalid url or permissions %s\" %url)\n return file_name"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nstream the response to a file.", "response": "def stream_response(self, response, stream_to=None):\n '''\n stream response is one level higher up than stream, starting with a \n response object and then performing the stream without making the\n requests.get. The expectation is that the request was successful \n (status code 20*).\n\n Parameters\n ==========\n response: a response that is ready to be iterated over to download in\n streamed chunks\n stream_to: the file to stream to\n\n\n '''\n\n if response.status_code == 200:\n\n # Keep user updated with Progress Bar\n content_size = None\n if 'Content-Length' in response.headers:\n progress = 0\n content_size = int(response.headers['Content-Length'])\n bot.show_progress(progress, content_size, length=35)\n\n chunk_size = 1 << 20\n with open(stream_to, 'wb') as filey:\n for chunk in response.iter_content(chunk_size=chunk_size):\n filey.write(chunk)\n if content_size is not None:\n progress+=chunk_size\n bot.show_progress(iteration=progress,\n total=content_size,\n length=35,\n carriage_return=False)\n\n # Newline to finish download\n sys.stdout.write('\\n')\n\n return stream_to \n\n bot.error(\"Problem with stream, response %s\" %(response.status_code))\n sys.exit(1)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the SCDP URL for a given service type and namespace. If no SCDP URL is given the default value is returned.", "response": "def getSCDPURL(self, serviceType, default=None):\n \"\"\"Returns the SCDP (Service Control Protocol Document) URL for a given service type.\n\n When the device definitions have been loaded with :meth:`~simpletr64.DeviceTR64.loadDeviceDefinitions` this\n method returns for a given service type/namespace the associated URL to the SCDP. If the device definitions\n have not been loaded a default value can be given which gets returned instead. The SCDP specifies all the\n interaction functionality a device provides.\n\n :param serviceType: the service type to look up for\n :param default: the default return value in case the service type is not found and device definitions are not\n loaded\n :type default: str or None\n :return: the URL/URI\n :rtype: str or None\n :raises ValueError: if the device did load device definitions and the service type is not known.\n\n .. seealso::\n\n :meth:`~simpletr64.DeviceTR64.loadDeviceDefinitions`\n \"\"\"\n if serviceType in self.__deviceServiceDefinitions.keys():\n return self.__deviceServiceDefinitions[serviceType][\"scpdURL\"]\n\n # check if definitions have been loaded, then dont return the default\n if self.__deviceXMLInitialized:\n raise ValueError(\"Device do not support given serviceType: \" + serviceType)\n\n return default"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef getControlURL(self, serviceType, default=None):\n if serviceType in self.__deviceServiceDefinitions.keys():\n return self.__deviceServiceDefinitions[serviceType][\"controlURL\"]\n\n # check if definitions have been loaded, then dont return the default\n if self.__deviceXMLInitialized:\n raise ValueError(\"Device do not support given serviceType: \" + serviceType)\n\n return default", "response": "Returns the control URL for a given service type and namespace."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the event URL for a given service type and namespace. If no event URL is given the associated event URL is returned. If no event URL is given the associated event URL is returned.", "response": "def getEventSubURL(self, serviceType, default=None):\n \"\"\"Returns the event URL for a given service type.\n\n When the device definitions have been loaded with :meth:`~simpletr64.DeviceTR64.loadDeviceDefinitions` this\n method returns for a given service type/namespace the associated event URL. If the device definitions have\n not been loaded a default value can be given which gets returned instead.\n\n :param serviceType: the service type to look up for\n :param default: the default return value in case the service type is not found and device definitions are not\n loaded\n :type default: str or None\n :return: the URL/URI\n :rtype: str or None\n :raises ValueError: if the device did load device definitions and the service type is not known.\n\n .. seealso::\n\n :meth:`~simpletr64.DeviceTR64.loadDeviceDefinitions`\n \"\"\"\n if serviceType in self.__deviceServiceDefinitions.keys():\n return self.__deviceServiceDefinitions[serviceType][\"eventSubURL\"]\n\n # check if definitions have been loaded, then dont return the default\n if self.__deviceXMLInitialized:\n raise ValueError(\"Device do not support given serviceType: \" + serviceType)\n\n return default"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef execute(self, uri, namespace, action, timeout=2, **kwargs):\n\n if not uri:\n raise ValueError(\"No action URI has been defined.\")\n\n if not namespace:\n raise ValueError(\"No namespace has been defined.\")\n\n if not action:\n raise ValueError(\"No action has been defined.\")\n\n # soap headers\n header = {'Content-Type': 'text/xml; charset=\"UTF-8\"',\n 'Soapaction': '\"' + namespace + \"#\" + action + '\"'}\n\n # build SOAP body\n body = '''\n\n \n \\n'''\n\n body += \" \\n'\n\n arguments = {}\n\n for key in kwargs.keys():\n body += \" <\" + key + \">\" + str(kwargs[key]) + \"\\n\"\n arguments[key] = str(kwargs[key])\n\n body += \" \\n\"\n body += '''\n'''\n\n # setup proxies\n proxies = {}\n if self.__httpsProxy:\n proxies = {\"https\": self.__httpsProxy}\n\n if self.__httpProxy:\n proxies = {\"http\": self.__httpProxy}\n\n # setup authentication\n auth = None\n if self.__password:\n auth = HTTPDigestAuth(self.__username, self.__password)\n\n # build the URL\n location = self.__protocol + \"://\" + self.__hostname + \":\" + str(self.port) + uri\n\n # Post http request\n request = requests.post(location, data=body, headers=header, auth=auth, proxies=proxies, timeout=float(timeout),\n verify=self.__verify)\n\n if request.status_code != 200:\n errorStr = DeviceTR64._extractErrorString(request)\n raise ValueError('Could not execute \"' + action + str(arguments) + '\": ' + str(request.status_code) +\n ' - ' + request.reason + \" -- \" + errorStr)\n\n # parse XML return\n try:\n root = ET.fromstring(request.text.encode('utf-8'))\n except Exception as e:\n raise ValueError(\"Can not parse results for the action: \" + str(e))\n\n # iterate in the XML structure to get the action result\n actionNode = root[0][0]\n\n # we need to remove XML namespace for the action node\n namespaceLength = len(namespace) + 2 # add braces\n tag = actionNode.tag[namespaceLength:]\n\n if tag != (action + \"Response\"):\n raise ValueError('Soap result structure is wrong, expected action \"' + action +\n 'Response\" got \"' + tag + '\".')\n\n # pack all the received results\n results = {}\n\n for resultNode in actionNode:\n results[resultNode.tag] = resultNode.text\n\n return results", "response": "Executes an action on a device and returns the results of the action."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nextracting an error string from a failed UPnP call.", "response": "def _extractErrorString(request):\n \"\"\"Extract error string from a failed UPnP call.\n\n :param request: the failed request result\n :type request: requests.Response\n :return: an extracted error text or empty str\n :rtype: str\n \"\"\"\n errorStr = \"\"\n\n tag = None\n\n # noinspection PyBroadException\n try:\n # parse XML return\n root = ET.fromstring(request.text.encode('utf-8'))\n tag = root[0][0]\n except:\n # return an empty string as we can not parse the structure\n return errorStr\n\n for element in tag.getiterator():\n tagName = element.tag.lower()\n\n if tagName.endswith(\"string\"):\n errorStr += element.text + \" \"\n elif tagName.endswith(\"description\"):\n errorStr += element.text + \" \"\n\n return errorStr"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef setupTR64Device(self, deviceType):\n\n if deviceType.lower() != \"fritz.box\":\n raise ValueError(\"Unknown device type given.\")\n\n self.__deviceServiceDefinitions = {}\n self.__deviceXMLInitialized = False\n\n # Fritz.box setup\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:DeviceConfig:1\"] = {\n \"controlURL\": \"/upnp/control/deviceconfig\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:ManagementServer:1\"] = {\n \"controlURL\": \"/upnp/control/mgmsrv\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:LANConfigSecurity:1\"] = {\n \"controlURL\": \"/upnp/control/lanconfigsecurity\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:Time:1\"] = {\"controlURL\": \"/upnp/control/time\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:LANHostConfigManagement:1\"] = {\n \"controlURL\": \"/upnp/control/lanhostconfigmgm\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:UserInterface:1\"] = {\n \"controlURL\": \"/upnp/control/userif\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:DeviceInfo:1\"] = {\n \"controlURL\": \"/upnp/control/deviceinfo\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:X_AVM-DE_TAM:1\"] = {\"controlURL\": \"/upnp/control/x_tam\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:X_AVM-DE_MyFritz:1\"] = {\n \"controlURL\": \"/upnp/control/x_myfritz\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:X_AVM-DE_RemoteAccess:1\"] = {\n \"controlURL\": \"/upnp/control/x_remote\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:WLANConfiguration:1\"] = {\n \"controlURL\": \"/upnp/control/wlanconfig1\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:WLANConfiguration:3\"] = {\n \"controlURL\": \"/upnp/control/wlanconfig3\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:WLANConfiguration:2\"] = {\n \"controlURL\": \"/upnp/control/wlanconfig2\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:X_AVM-DE_WebDAVClient:1\"] = {\n \"controlURL\": \"/upnp/control/x_webdav\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:WANDSLLinkConfig:1\"] = {\n \"controlURL\": \"/upnp/control/wandsllinkconfig1\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:Hosts:1\"] = {\"controlURL\": \"/upnp/control/hosts\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:X_VoIP:1\"] = {\"controlURL\": \"/upnp/control/x_voip\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:LANEthernetInterfaceConfig:1\"] = {\n \"controlURL\": \"/upnp/control/lanethernetifcfg\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:Layer3Forwarding:1\"] = {\n \"controlURL\": \"/upnp/control/layer3forwarding\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:WANIPConnection:1\"] = {\n \"controlURL\": \"/upnp/control/wanipconnection1\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:X_AVM-DE_OnTel:1\"] = {\n \"controlURL\": \"/upnp/control/x_contact\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:WANCommonInterfaceConfig:1\"] = {\n \"controlURL\": \"/upnp/control/wancommonifconfig1\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:X_AVM-DE_UPnP:1\"] = {\n \"controlURL\": \"/upnp/control/x_upnp\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:WANDSLInterfaceConfig:1\"] = {\n \"controlURL\": \"/upnp/control/wandslifconfig1\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:WANPPPConnection:1\"] = {\n \"controlURL\": \"/upnp/control/wanpppconn1\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:X_AVM-DE_Storage:1\"] = {\n \"controlURL\": \"/upnp/control/x_storage\"}\n self.deviceServiceDefinitions[\"urn:dslforum-org:service:WANEthernetLinkConfig:1\"] = {\n \"controlURL\": \"/upnp/control/wanethlinkconfig1\"}", "response": "Setup actions for known devices."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef loadDeviceDefinitions(self, urlOfXMLDefinition, timeout=3):\n\n # setup proxies\n proxies = {}\n if self.__httpsProxy:\n proxies = {\"https\": self.__httpsProxy}\n\n if self.__httpProxy:\n proxies = {\"http\": self.__httpProxy}\n\n # some devices response differently without a User-Agent\n headers = {\"User-Agent\": \"Mozilla/5.0; SimpleTR64-1\"}\n\n # setup authentication\n auth = None\n if self.__password:\n auth = HTTPDigestAuth(self.__username, self.__password)\n\n # get the content\n request = requests.get(urlOfXMLDefinition, proxies=proxies, headers=headers, timeout=float(timeout),\n auth=auth, verify=self.__verify)\n\n if request.status_code != 200:\n errorStr = DeviceTR64._extractErrorString(request)\n raise ValueError('Could not get CPE definitions \"' + urlOfXMLDefinition + '\" : ' +\n str(request.status_code) + ' - ' + request.reason + \" -- \" + errorStr)\n\n # parse XML return\n xml = request.text.encode('utf-8')\n\n return self._loadDeviceDefinitions(urlOfXMLDefinition, xml)", "response": "Loads the device definitions from a given URL which points to the root XML in the device."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _iterateToFindSCPDElements(self, element, baseURIPath):\n for child in element.getchildren():\n tagName = child.tag.lower()\n if tagName.endswith('servicelist'):\n self._processServiceList(child,baseURIPath)\n\n elif tagName.endswith('devicetype'):\n if \"deviceType\" not in self.__deviceInformations.keys():\n self.__deviceInformations[\"deviceType\"] = child.text\n elif tagName.endswith('friendlyname'):\n if \"friendlyName\" not in self.__deviceInformations.keys():\n self.__deviceInformations[\"friendlyName\"] = child.text\n elif tagName.endswith('manufacturer'):\n if \"manufacturer\" not in self.__deviceInformations.keys():\n self.__deviceInformations[\"manufacturer\"] = child.text\n elif tagName.endswith('manufacturerurl'):\n if \"manufacturerURL\" not in self.__deviceInformations.keys():\n self.__deviceInformations[\"manufacturerURL\"] = child.text\n elif tagName.endswith('modeldescription'):\n if \"modelDescription\" not in self.__deviceInformations.keys():\n self.__deviceInformations[\"modelDescription\"] = child.text\n elif tagName.endswith('modelname'):\n if \"modelName\" not in self.__deviceInformations.keys():\n self.__deviceInformations[\"modelName\"] = child.text\n elif tagName.endswith('modelurl'):\n if \"modelURL\" not in self.__deviceInformations.keys():\n self.__deviceInformations[\"modelURL\"] = child.text\n elif tagName.endswith('modelnumber'):\n if \"modelNumber\" not in self.__deviceInformations.keys():\n self.__deviceInformations[\"modelNumber\"] = child.text\n elif tagName.endswith('serialnumber'):\n if \"serialNumber\" not in self.__deviceInformations.keys():\n self.__deviceInformations[\"serialNumber\"] = child.text\n elif tagName.endswith('presentationurl'):\n if \"presentationURL\" not in self.__deviceInformations.keys():\n self.__deviceInformations[\"presentationURL\"] = child.text\n elif tagName.endswith('udn'):\n if \"UDN\" not in self.__deviceInformations.keys():\n self.__deviceInformations[\"UDN\"] = child.text\n elif tagName.endswith('upc'):\n if \"UPC\" not in self.__deviceInformations.keys():\n self.__deviceInformations[\"UPC\"] = child.text\n elif tagName.endswith('iconlist') or tagName.endswith('specversion'):\n # skip these items\n pass\n else:\n if not tagName.endswith('device') and not tagName.endswith('devicelist'):\n self.__deviceUnknownKeys[child.tag] = child.text\n\n self._iterateToFindSCPDElements(child, baseURIPath)", "response": "This method is used to iterate through the device definition XML tree and populate the object with the information from the device information."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef loadSCPD(self, serviceType=None, timeout=3, ignoreFailures=False):\n\n if serviceType is not None:\n self._loadSCPD(serviceType, float(timeout))\n else:\n self.__deviceSCPD = {}\n for serviceType in self.__deviceServiceDefinitions.keys():\n # remove any previous error\n self.__deviceServiceDefinitions[serviceType].pop(\"error\", None)\n\n try:\n self._loadSCPD(serviceType, float(timeout))\n except ValueError as e:\n if not ignoreFailures:\n # we not ignoring this so rethrow last exception\n raise\n else:\n # add a message in the structure\n self.__deviceServiceDefinitions[serviceType][\"error\"] = str(e)", "response": "Load the SCPD for a given service type."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _parseSCPDActions(self, actionListElement, actions, variableParameterDict):\n\n # go through all action elements in this list\n for actionElement in actionListElement.getchildren():\n\n action = {}\n\n # go through all elements in this action\n for inActionElement in actionElement.getchildren():\n tagName = inActionElement.tag.lower()\n\n if tagName.endswith(\"name\"):\n # action name\n action[\"name\"] = inActionElement.text\n elif tagName.endswith(\"argumentlist\"):\n # parse the arguments of this action\n for argumentElement in inActionElement.getchildren():\n\n argument = {}\n\n # go through the argument definition\n for inArgumentElement in argumentElement.getchildren():\n tagName = inArgumentElement.tag.lower()\n\n if tagName.endswith(\"name\"):\n # remember the argument name\n argument[\"name\"] = inArgumentElement.text\n elif tagName.endswith(\"direction\"):\n # is it an in or out argument\n argument[\"direction\"] = inArgumentElement.text\n elif tagName.endswith(\"relatedstatevariable\"):\n # remember the argument and safe it under the variable name to dereference later\n argument[\"variable\"] = inArgumentElement.text\n\n if argument[\"variable\"] not in variableParameterDict.keys():\n variableParameterDict[argument[\"variable\"]] = []\n\n variableParameterDict[argument[\"variable\"]].append(argument)\n\n if \"name\" not in argument.keys():\n raise ValueError(\"Parameter definition does not contain a name.\")\n\n if \"direction\" not in argument.keys():\n raise ValueError(\"Parameter definition does not contain a direction: \" + argument[\"name\"])\n\n direction = argument[\"direction\"] + \"Parameter\"\n\n # store the actual argument in the action\n if direction not in action.keys():\n action[direction] = {}\n\n action[direction][argument[\"name\"]] = argument\n\n # cleanup, we stored the argument we dont need these values in there anymore otherwise they\n # would be redundant\n del argument[\"name\"]\n del argument[\"direction\"]\n\n if \"name\" not in action.keys():\n raise ValueError(\"Action has not a name assigned.\")\n\n if action[\"name\"] in actions.keys():\n raise ValueError(\"Action name defined more than ones: \" + action[\"name\"])\n\n # save the action under its name\n actions[action[\"name\"]] = action\n\n # cleanup, as we have saved the action under its name in the container it would be redundant\n del action[\"name\"]", "response": "Internal method to parse the SCPD definitions."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _parseSCPDVariableTypes(self, variableListElement, variableTypes):\n\n # iterate through all variables\n for variableElement in variableListElement.getchildren():\n\n variable = {}\n\n # iterate through the variable definition\n for inVariableElement in variableElement.getchildren():\n tagName = inVariableElement.tag.lower()\n\n if tagName.endswith(\"name\"):\n variable[\"name\"] = inVariableElement.text\n elif tagName.endswith(\"datatype\"):\n variable[\"dataType\"] = inVariableElement.text\n elif tagName.endswith(\"defaultvalue\"):\n variable[\"defaultValue\"] = inVariableElement.text\n\n if \"name\" not in variable.keys():\n raise ValueError(\"Variable has no name defined.\")\n\n if \"dataType\" not in variable.keys():\n raise ValueError(\"No dataType was defined by variable: \" + variable[\"name\"])\n\n if variable[\"name\"] in variableTypes.keys():\n raise ValueError(\"Variable has been defined multiple times: \" + variable[\"name\"])\n\n variableTypes[variable[\"name\"]] = variable", "response": "Internal method to parse the variable types in the SCPD definitions."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef createFromURL(urlOfXMLDefinition):\n url = urlparse(urlOfXMLDefinition)\n\n if not url.port:\n if url.scheme.lower() == \"https\":\n port = 443\n else:\n port = 80\n else:\n port = url.port\n\n return Wifi(url.hostname, port, url.scheme)", "response": "Factory method to create a Wifi object from an XML device definition."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef getWifiInfo(self, wifiInterfaceId=1, timeout=1):\n namespace = Wifi.getServiceType(\"getWifiInfo\") + str(wifiInterfaceId)\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetInfo\", timeout=timeout)\n\n return WifiBasicInfo(results)", "response": "Execute GetInfo action to get Wifi basic information s."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nexecutes GetTotalAssociations action to get the amount of associated Wifi clients.", "response": "def getTotalAssociations(self, wifiInterfaceId=1, timeout=1):\n \"\"\"Execute GetTotalAssociations action to get the amount of associated Wifi clients.\n\n :param int wifiInterfaceId: the id of the Wifi interface\n :param float timeout: the timeout to wait for the action to be executed\n :return: the amount of Wifi clients\n :rtype: int\n\n .. seealso:: :meth:`~simpletr64.actions.Wifi.getGenericAssociatedDeviceInfo`\n \"\"\"\n namespace = Wifi.getServiceType(\"getTotalAssociations\") + str(wifiInterfaceId)\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetTotalAssociations\", timeout=timeout)\n\n return int(results[\"NewTotalAssociations\"])"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nexecuting GetGenericAssociatedDeviceInfo action to get detailed information about a Wifi client.", "response": "def getGenericAssociatedDeviceInfo(self, index, wifiInterfaceId=1, timeout=1):\n \"\"\"Execute GetGenericAssociatedDeviceInfo action to get detailed information about a Wifi client.\n\n :param int index: the number of the client\n :param int wifiInterfaceId: the id of the Wifi interface\n :param float timeout: the timeout to wait for the action to be executed\n :return: the detailed information's about a Wifi client\n :rtype: WifiDeviceInfo\n\n .. seealso:: :meth:`~simpletr64.actions.Wifi.getTotalAssociations`\n \"\"\"\n namespace = Wifi.getServiceType(\"getGenericAssociatedDeviceInfo\") + str(wifiInterfaceId)\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetGenericAssociatedDeviceInfo\", timeout=timeout,\n NewAssociatedDeviceIndex=index)\n\n return WifiDeviceInfo(results)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nexecute GetSpecificAssociatedDeviceInfo action to get detailed information about a Wifi client.", "response": "def getSpecificAssociatedDeviceInfo(self, macAddress, wifiInterfaceId=1, timeout=1):\n \"\"\"Execute GetSpecificAssociatedDeviceInfo action to get detailed information about a Wifi client.\n\n :param str macAddress: MAC address in the form ``38:C9:86:26:7E:38``; be aware that the MAC address might\n be case sensitive, depending on the router\n :param int wifiInterfaceId: the id of the Wifi interface\n :param float timeout: the timeout to wait for the action to be executed\n :return: the detailed information's about a Wifi client\n :rtype: WifiDeviceInfo\n\n .. seealso:: :meth:`~simpletr64.actions.Wifi.getGenericAssociatedDeviceInfo`\n \"\"\"\n namespace = Wifi.getServiceType(\"getSpecificAssociatedDeviceInfo\") + str(wifiInterfaceId)\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetSpecificAssociatedDeviceInfo\", timeout=timeout,\n NewAssociatedDeviceMACAddress=macAddress)\n\n return WifiDeviceInfo(results, macAddress=macAddress)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nset enable status for a Wifi interface.", "response": "def setEnable(self, status, wifiInterfaceId=1, timeout=1):\n \"\"\"Set enable status for a Wifi interface, be careful you don't cut yourself off.\n\n :param bool status: enable or disable the interface\n :param int wifiInterfaceId: the id of the Wifi interface\n :param float timeout: the timeout to wait for the action to be executed\n \"\"\"\n namespace = Wifi.getServiceType(\"setEnable\") + str(wifiInterfaceId)\n uri = self.getControlURL(namespace)\n\n if status:\n setStatus = 1\n else:\n setStatus = 0\n\n self.execute(uri, namespace, \"SetEnable\", timeout=timeout, NewEnable=setStatus)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nset the channel of this Wifi interface", "response": "def setChannel(self, channel, wifiInterfaceId=1, timeout=1):\n \"\"\"Set the channel of this Wifi interface\n\n :param int channel: the channel number\n :param int wifiInterfaceId: the id of the Wifi interface\n :param float timeout: the timeout to wait for the action to be executed\n \"\"\"\n namespace = Wifi.getServiceType(\"setChannel\") + str(wifiInterfaceId)\n uri = self.getControlURL(namespace)\n\n self.execute(uri, namespace, \"SetChannel\", timeout=timeout, NewChannel=channel)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef setSSID(self, ssid, wifiInterfaceId=1, timeout=1):\n namespace = Wifi.getServiceType(\"setChannel\") + str(wifiInterfaceId)\n uri = self.getControlURL(namespace)\n\n self.execute(uri, namespace, \"SetChannel\", timeout=timeout, NewSSID=ssid)", "response": "Set the SSID of the Wifi network"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\naccesses decorator which checks if a RPC method is enabled by our configuration", "response": "def enabled_checker(func):\n \"\"\" Access decorator which checks if a RPC method is enabled by our configuration\n \"\"\"\n @wraps(func)\n def wrap(self, *args, **kwargs):\n if self.allowed_methods and isinstance(self.allowed_methods, list) and func.__name__ not in self.allowed_methods:\n raise Exception(\"Method {} is disabled\".format(func.__name__))\n return func(self, *args, **kwargs)\n return wrap"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\nasync def list_presets(self):\n presets = await self.request.app.vmshepherd.preset_manager.list_presets()\n return list(presets.keys())", "response": "List all available presets."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\nasync def list_vms(self, preset):\n vmshepherd = self.request.app.vmshepherd\n preset = vmshepherd.preset_manager.get_preset(preset)\n result_vms = {vm.id: {'ip': vm.ip[0], 'state': vm.state.value} for vm in preset.vms}\n return preset.count, result_vms", "response": "List virtual machines in a given preset"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\nasync def terminate_vm(self, preset, vm_id):\n vmshepherd = self.request.app.vmshepherd\n preset = vmshepherd.preset_manager.get_preset(preset)\n await preset.iaas.terminate_vm(vm_id)\n return 'OK'", "response": "Discard vm in specified preset"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget VM metadata for a given VM", "response": "async def get_vm_metadata(self, preset, vm_id):\n \"\"\" Get vm metadata\n\n :arg string preset: preset name\n :arg int vm_id: Virtual Machine id\n :return: Metadata for Virtual Machine\n :rtype: dict\n\n Sample response:\n ``{ 'time_shutdown' : \"12312312321' }``\n \"\"\"\n vmshepherd = self.request.app.vmshepherd\n preset = vmshepherd.preset_manager.get_preset(preset)\n vm_info = await preset.iaas.get_vm(vm_id)\n ret_info = copy.deepcopy(vm_info.metadata) if vm_info.metadata else {}\n ret_info['tags'] = vm_info.tags\n ret_info['iaas_shutdown'] = vm_info.timed_shutdown_at\n return ret_info"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef linkify(text, attrs={}):\n def separate_parentheses(s):\n start = re_find(r'^\\(*', s)\n end = re_find(r'\\)*$', s)\n n = min(len(start), len(end))\n if n:\n return s[:n], s[n:-n], s[-n:]\n else:\n return '', s, ''\n\n def link_repl(url, proto='http://'):\n opening, url, closing = separate_parentheses(url)\n\n punct = re_find(punct_re, url)\n if punct:\n url = url[:-len(punct)]\n\n if re.search(proto_re, url):\n href = url\n else:\n href = proto + url\n href = escape_url(href)\n\n repl = u'{0!s}{3!s}{4!s}{5!s}'\n return repl.format(opening,\n href, attrs_text, url, punct,\n closing)\n\n def repl(match):\n matches = match.groupdict()\n if matches['url']:\n return link_repl(matches['url'])\n else:\n return link_repl(matches['email'], proto='mailto:')\n\n # Prepare attrs\n attr = ' {0!s}=\"{1!s}\"'\n attrs_text = ''.join(starmap(attr.format, attrs.items()))\n\n # Make replaces\n return re.sub(combined_re, repl, force_unicode(text))", "response": "Convert URL - like and email - like strings into links."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nforcing unicode string s to be unicode.", "response": "def force_unicode(s, encoding='utf-8', errors='strict'):\n \"\"\"\n Similar to smart_text, except that lazy instances are resolved to\n strings, rather than kept as lazy objects.\n \"\"\"\n # Handle the common case first, saves 30-40% when s is an instance of\n # six.text_type. This function gets called often in that setting.\n if isinstance(s, six.text_type):\n return s\n if not isinstance(s, six.string_types):\n if six.PY3:\n if isinstance(s, bytes):\n s = six.text_type(s, encoding, errors)\n else:\n s = six.text_type(s)\n else:\n s = six.text_type(bytes(s), encoding, errors)\n else:\n # Note: We use .decode() here, instead of six.text_type(s,\n # encoding, errors), so that if s is a SafeBytes, it ends up being\n # a SafeText at the end.\n s = s.decode(encoding, errors)\n return s"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef save_dict_to_hdf5(dic, filename):\n with h5py.File(filename, 'w') as h5file:\n rf = recursively_save_dict_contents_to_group(h5file, '/', dic)\n h5_rf = h5file.create_group(\"_reconstruction_flags\")\n # h5_rf = h5file.create_group(\"_reconstruction_key_flags\")\n for k, v in rf.items():\n h5_rf.create_dataset(\"/_reconstruction_flags\" + k, data=v)", "response": "Save dictionary to HDF5 file."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef recursively_load_dict_contents_from_group(h5file, path):\n rf = h5file[\"_reconstruction_flags\"]\n # rkf = h5file[\"_reconstruction_key_flags\"]\n ans = {}\n for key, item in h5file[path].items():\n dest_key = key\n # if key in (\"_reconstruction_flags\", \"_reconstruction_key_flags\"):\n if key in \"_reconstruction_flags\":\n continue\n kkey = key + \"_key_\"\n tkey = key + \"_typ_\"\n if kkey in rf:\n flag = rf[kkey]\n if flag.value == \"json_key\":\n import json\n dest_key = json.loads(key)\n # import pickle\n # dest_key = pickle.loads(key.encode(\"ascii\"))\n # logger.debug(\"unpickling key\")\n\n if tkey in rf:\n flag = rf[tkey]\n if flag.value == \"list\":\n dict_to_output = recursively_load_dict_contents_from_group(h5file, path + key + '/')\n ans[dest_key] = list(dict_to_output.values())\n continue\n if flag.value == \"tuple\":\n dict_to_output = recursively_load_dict_contents_from_group(h5file, path + key + '/')\n ans[dest_key] = tuple(dict_to_output.values())\n continue\n elif flag.value == \"json_value\":\n import json\n ans[dest_key] = json.loads(item.value)\n continue\n elif flag.value == \"float\":\n ans[dest_key] = float(item.value)\n continue\n elif flag.value == \"int\":\n ans[dest_key] = int(item.value)\n continue\n\n if isinstance(item, h5py._hl.dataset.Dataset):\n ans[dest_key] = item.value\n elif isinstance(item, h5py._hl.group.Group):\n ans[dest_key] = recursively_load_dict_contents_from_group(h5file, path + key + '/')\n return ans", "response": "Recursively loads the contents of a group into a single dict."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse_args(*args):\n parser = argparse.ArgumentParser(\n description=\"Send push notifications for a feed\")\n\n parser.add_argument('--version', action='version',\n version=\"%(prog)s \" + __version__.__version__)\n\n parser.add_argument('feeds', type=str, nargs='*', metavar='feed_url',\n help='A URL for a feed to process')\n parser.add_argument('--cache', '-c', type=str, dest='cache_dir',\n help='Cache storage directory',\n required=False)\n\n parser.add_argument(\"-v\", \"--verbosity\", action=\"count\",\n help=\"increase output verbosity\",\n default=0)\n parser.add_argument(\"-e\", \"--entry\", nargs='+',\n help='URLs to entries/pages to index directly',\n metavar='entry_url',\n dest='entries')\n parser.add_argument(\"-s\", \"--websub-only\", nargs='+',\n help='URLs/feeds to only send WebSub notifications for',\n metavar='feed_url', dest='websub_only')\n parser.add_argument('--timeout', '-t', type=int, dest='timeout',\n help='Connection timeout, in seconds',\n default=120)\n\n parser.add_argument('--max-connections', type=int, dest='max_connections',\n help='Maximum number of connections to have open at once',\n default=100)\n parser.add_argument('--max-per-host', type=int, dest='max_per_host',\n help='Maximum number of connections per host',\n default=0)\n\n parser.add_argument('--rel-whitelist', '-w', dest='rel_whitelist', type=str,\n help=\"Comma-separated list of link RELs to whitelist\"\n + \" for sending webmentions\")\n parser.add_argument('--rel-blacklist', '-b', dest='rel_blacklist', type=str,\n help=\"Comma-separated list of link RELs to blacklist\"\n + \" from sending webmentions\",\n default=\"nofollow\")\n\n parser.add_argument('--max-time', '-m', dest='max_time', type=float,\n help=\"Maximum time (in seconds) to spend on this\", default=1800)\n\n parser.add_argument('--user-agent', dest='user_agent', type=str,\n help=\"User-agent string to send\", default=__version__.USER_AGENT)\n\n feature = parser.add_mutually_exclusive_group(required=False)\n feature.add_argument('--keepalive', dest='keepalive', action='store_true',\n help=\"Keep TCP connections alive\")\n feature.add_argument('--no-keepalive', dest='keepalive', action='store_false',\n help=\"Don't keep TCP connections alive\")\n feature.set_defaults(keepalive=False)\n\n feature = parser.add_mutually_exclusive_group(required=False)\n feature.add_argument('--archive', '-a', dest='archive', action='store_true',\n help='Process archive links in the feed per RFC 5005')\n feature.add_argument('--no-archive', dest='archive', action='store_false',\n help='Do not process archive links in the feed')\n feature.set_defaults(archive=False)\n\n feature = parser.add_mutually_exclusive_group(required=False)\n feature.add_argument('--recurse', '-r',\n help=\"Recursively check other discovered feeds\",\n action='store_true', dest='recurse')\n feature.add_argument('--no-recurse', dest='recurse',\n action='store_false',\n help=\"Do not recurse into other feeds\")\n feature.set_defaults(recurse=False)\n\n return parser.parse_args(*args)", "response": "Parse the arguments for the command line"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nretrieving all environments for a given project.", "response": "def get_environments(self, project_key: str) -> dict:\n \"\"\"\n Retrieve all environments for a given project.\n\n Includes name, key, and mobile key.\n\n :param project_key: Key for project.\n\n :returns: dictionary of environments.\n \"\"\"\n try:\n resp = self.client.get_project(project_key)\n except launchdarkly_api.rest.ApiException as ex:\n msg = \"Unable to get environments.\"\n resp = \"API response was {0} {1}.\".format(ex.status, ex.reason)\n LOG.error(\"%s %s\", msg, resp)\n sys.exit(1)\n\n envs = []\n\n for env in resp.environments:\n env = dict(\n key=env.key,\n api_key=env.api_key,\n client_id=env.id\n )\n envs.append(env)\n\n return envs"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nupdates the status of a feature flag.", "response": "def update_flag(self, state: str, feature_key: str) \\\n -> launchdarkly_api.FeatureFlag:\n \"\"\"\n Update the flag status for the specified feature flag.\n\n :param state: New feature flag state\n :param featureKey: Feature flag key\n\n :returns: FeatureFlag object.\n \"\"\"\n build_env = \"/environments/\" + self.environment_key + \"/on\"\n patch_comment = [{\"op\": \"replace\", \"path\": build_env, \"value\": state}]\n\n try:\n resp = self.feature.patch_feature_flag(\n self.project_key, feature_key, patch_comment)\n except launchdarkly_api.rest.ApiException as ex:\n msg = \"Unable to update flag.\"\n resp = \"API response was {0} {1}.\".format(ex.status, ex.reason)\n LOG.error(\"%s %s\", msg, resp)\n sys.exit(1)\n\n return resp"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _write_result_handler(self, routine):\n self._write_line('ret = {}')\n self._write_execute_rows(routine)\n self._write_line('for row in rows:')\n\n num_of_dict = len(routine['columns'])\n\n i = 0\n while i < num_of_dict:\n value = \"row['{0!s}']\".format(routine['columns'][i])\n\n stack = ''\n j = 0\n while j < i:\n stack += \"[row['{0!s}']]\".format(routine['columns'][j])\n j += 1\n line = 'if {0!s} in ret{1!s}:'.format(value, stack)\n self._write_line(line)\n i += 1\n\n line = \"raise Exception('Duplicate key for %s.' % str(({0!s})))\". \\\n format(\", \".join([\"row['{0!s}']\".format(column_name) for column_name in routine['columns']]))\n\n self._write_line(line)\n self._indent_level_down()\n\n i = num_of_dict\n while i > 0:\n self._write_line('else:')\n\n part1 = ''\n j = 0\n while j < i - 1:\n part1 += \"[row['{0!s}']]\".format(routine['columns'][j])\n j += 1\n part1 += \"[row['{0!s}']]\".format(routine['columns'][j])\n\n part2 = ''\n j = i - 1\n while j < num_of_dict:\n if j + 1 != i:\n part2 += \"{{row['{0!s}']: \".format(routine['columns'][j])\n j += 1\n part2 += \"row\" + ('}' * (num_of_dict - i))\n\n line = \"ret{0!s} = {1!s}\".format(part1, part2)\n self._write_line(line)\n self._indent_level_down()\n if i > 1:\n self._indent_level_down()\n i -= 1\n\n self._write_line()\n self._write_line('return ret')", "response": "Generates code for calling the stored routine in the wrapper method."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _log_query(query):\n query = query.strip()\n\n if os.linesep in query:\n # Query is a multi line query\n MetadataDataLayer.io.log_very_verbose('Executing query:')\n MetadataDataLayer.io.log_very_verbose('{0}'.format(query))\n else:\n # Query is a single line query.\n MetadataDataLayer.io.log_very_verbose('Executing query: {0}'.format(query))", "response": "Logs the query on the console."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef request_token(self, board):\n '''send a public key to request a token. When we call this function,\n we already have an RSA key at self.key\n\n board: the discourse board to post to\n '''\n nonce = str(uuid.uuid4())\n\n data = {'scopes': 'write',\n 'client_id': self.client_id,\n 'application_name': 'HelpMe',\n 'public_key': self.public_key.replace(\"'\",\"\"),\n 'nonce': nonce }\n\n url = (board + \"/user-api-key/new?scopes=write&application_name=HelpMe&public_key=\" + \n self.public_key.replace(\"'\", \"\") + \n \"&client_id=\" + self.client_id +\n \"&nonce=\" + nonce )\n\n bot.newline()\n bot.info('Open browser to:')\n bot.info(url)\n bot.newline()\n \n # the user will open browser, get a token, and then have it saved here.\n bot.info('Copy paste token, press Ctrl-D to save it:')\n\n lines = []\n\n # The message is multiple lines\n while True:\n try:\n line = enter_input()\n except EOFError:\n break\n if line:\n lines.append(line)\n\n message = \"\\n\".join(lines)\n\n # Write to temporary file, we only need to get key\n tmpfile = mktemp()\n with open(tmpfile, 'w') as filey:\n filey.write(message)\n\n # Read in again, and get token **important** is binary\n with open(tmpfile, 'rb') as filey:\n message = filey.read()\n\n # uses pycryptodome (3.7.2)\n cipher = Cipher_PKCS1_v1_5.new(self.key)\n decrypted = json.loads(cipher.decrypt(b64decode(message), None).decode())\n\n # Validate nonce is in response\n if \"nonce\" not in decrypted:\n bot.exit('Missing nonce field in response for token, invalid.')\n\n # Must return nonce that we sent\n if decrypted['nonce'] != nonce:\n bot.exit('Invalid nonce, exiting.')\n\n return decrypted['key']", "response": "send a public key to request a token"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a Discourse post given a title body board and token.", "response": "def create_post(self, title, body, board, category, username):\n '''create a Discourse post, given a title, body, board, and token.\n\n Parameters\n ==========\n title: the issue title\n body: the issue body\n board: the discourse board to post to\n\n '''\n\n category_url = \"%s/categories.json\" % board\n response = requests.get(category_url)\n\n if response.status_code != 200:\n print('Error with retrieving %s' % category_url)\n sys.exit(1)\n\n # Get a list of all categories\n categories = response.json()['category_list']['categories']\n categories = {c['name']:c['id'] for c in categories}\n\n # And if not valid, warn the user\n if category not in categories:\n bot.warning('%s is not valid, will use default' % category)\n\n category_id = categories.get(category, None)\n\n headers = {\"Content-Type\": \"application/json\",\n \"User-Api-Client-Id\": self.client_id,\n \"User-Api-Key\": self.token }\n\n\n # First get the category ids\n data = {'title': title,\n 'raw': body,\n 'category': category_id}\n\n response = requests.post(\"%s/posts.json\" % board,\n headers=headers,\n data=json.dumps(data))\n\n if response.status_code in [200, 201, 202]:\n topic = response.json()\n url = \"%s/t/%s/%s\" %(board, topic['topic_slug'], topic['topic_id'])\n bot.info(url)\n return url\n\n elif response.status_code == 404:\n bot.error('Cannot post to board, not found. Do you have permission?')\n sys.exit(1)\n\n else:\n bot.error('Cannot post to board %s' % board)\n bot.error(response.content)\n sys.exit(1)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef main(self, config_filename, file_names=None):\n self._io.title('Loader')\n\n if file_names:\n self.__load_list(config_filename, file_names)\n else:\n self.__load_all(config_filename)\n\n if self.error_file_names:\n self.__log_overview_errors()\n return 1\n else:\n return 0", "response": "Load the stored routines into the current schema."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef __log_overview_errors(self):\n if self.error_file_names:\n self._io.warning('Routines in the files below are not loaded:')\n self._io.listing(sorted(self.error_file_names))", "response": "Log information about stored routines that were not loaded successfully."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nadd a replace part to the map of replace pairs.", "response": "def _add_replace_pair(self, name, value, quote):\n \"\"\"\n Adds a replace part to the map of replace pairs.\n\n :param name: The name of the replace pair.\n :param value: The value of value of the replace pair.\n \"\"\"\n key = '@' + name + '@'\n key = key.lower()\n\n class_name = value.__class__.__name__\n\n if class_name in ['int', 'float']:\n value = str(value)\n elif class_name in ['bool']:\n value = '1' if value else '0'\n elif class_name in ['str']:\n if quote:\n value = \"'\" + value + \"'\"\n else:\n self._io.log_verbose(\"Ignoring constant {} which is an instance of {}\".format(name, class_name))\n\n self._replace_pairs[key] = value"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef __load_list(self, config_filename, file_names):\n self._read_configuration_file(config_filename)\n self.connect()\n self.find_source_files_from_list(file_names)\n self._get_column_type()\n self.__read_stored_routine_metadata()\n self.__get_constants()\n self._get_old_stored_routine_info()\n self._get_correct_sql_mode()\n self.__load_stored_routines()\n self.__write_stored_routine_metadata()\n self.disconnect()", "response": "Loads all stored routines in a list into the RDBMS instance."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nloads all stored routines into the RDBMS instance.", "response": "def __load_all(self, config_filename):\n \"\"\"\n Loads all stored routines into the RDBMS instance.\n\n :param str config_filename: string The filename of the configuration file.\n \"\"\"\n self._read_configuration_file(config_filename)\n self.connect()\n self.__find_source_files()\n self._get_column_type()\n self.__read_stored_routine_metadata()\n self.__get_constants()\n self._get_old_stored_routine_info()\n self._get_correct_sql_mode()\n self.__load_stored_routines()\n self._drop_obsolete_routines()\n self.__remove_obsolete_metadata()\n self.__write_stored_routine_metadata()\n self.disconnect()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreading the parameters from the configuration file.", "response": "def _read_configuration_file(self, config_filename):\n \"\"\"\n Reads parameters from the configuration file.\n\n :param str config_filename: The name of the configuration file.\n \"\"\"\n config = configparser.ConfigParser()\n config.read(config_filename)\n\n self._source_directory = config.get('loader', 'source_directory')\n self._source_file_extension = config.get('loader', 'extension')\n self._source_file_encoding = config.get('loader', 'encoding')\n self.__shadow_directory = config.get('loader', 'shadow_directory', fallback=None)\n\n self._pystratum_metadata_filename = config.get('wrapper', 'metadata')\n\n self._constants_class_name = config.get('constants', 'class')"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef __find_source_files(self):\n for dir_path, _, files in os.walk(self._source_directory):\n for name in files:\n if name.lower().endswith(self._source_file_extension):\n basename = os.path.splitext(os.path.basename(name))[0]\n relative_path = os.path.relpath(os.path.join(dir_path, name))\n\n if basename in self._source_file_names:\n self._io.error(\"Files '{0}' and '{1}' have the same basename.\".\n format(self._source_file_names[basename], relative_path))\n self.error_file_names.add(relative_path)\n else:\n self._source_file_names[basename] = relative_path", "response": "Searches recursively for all source files in a directory and adds them to the internal list of error files."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef __read_stored_routine_metadata(self):\n if os.path.isfile(self._pystratum_metadata_filename):\n with open(self._pystratum_metadata_filename, 'r') as file:\n self._pystratum_metadata = json.load(file)", "response": "Reads the metadata of stored routines from the metadata file."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nload all stored routines into the RDBMS instance.", "response": "def __load_stored_routines(self):\n \"\"\"\n Loads all stored routines into the RDBMS instance.\n \"\"\"\n self._io.writeln('')\n\n for routine_name in sorted(self._source_file_names):\n if routine_name in self._pystratum_metadata:\n old_metadata = self._pystratum_metadata[routine_name]\n else:\n old_metadata = None\n\n if routine_name in self._rdbms_old_metadata:\n old_routine_info = self._rdbms_old_metadata[routine_name]\n else:\n old_routine_info = None\n\n routine_loader_helper = self.create_routine_loader_helper(routine_name, old_metadata, old_routine_info)\n routine_loader_helper.shadow_directory = self.__shadow_directory\n\n metadata = routine_loader_helper.load_stored_routine()\n\n if not metadata:\n self.error_file_names.add(self._source_file_names[routine_name])\n if routine_name in self._pystratum_metadata:\n del self._pystratum_metadata[routine_name]\n else:\n self._pystratum_metadata[routine_name] = metadata"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nremove obsolete entries from the metadata of all stored routines.", "response": "def __remove_obsolete_metadata(self):\n \"\"\"\n Removes obsolete entries from the metadata of all stored routines.\n \"\"\"\n clean = {}\n for key, _ in self._source_file_names.items():\n if key in self._pystratum_metadata:\n clean[key] = self._pystratum_metadata[key]\n\n self._pystratum_metadata = clean"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef __write_stored_routine_metadata(self):\n with open(self._pystratum_metadata_filename, 'w') as stream:\n json.dump(self._pystratum_metadata, stream, indent=4, sort_keys=True)", "response": "Writes the metadata of all stored routines to the metadata file."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfinding all source files that actually exists from a list of file names.", "response": "def find_source_files_from_list(self, file_names):\n \"\"\"\n Finds all source files that actually exists from a list of file names.\n\n :param list[str] file_names: The list of file names.\n \"\"\"\n for file_name in file_names:\n if os.path.exists(file_name):\n routine_name = os.path.splitext(os.path.basename(file_name))[0]\n if routine_name not in self._source_file_names:\n self._source_file_names[routine_name] = file_name\n else:\n self._io.error(\"Files '{0}' and '{1}' have the same basename.\".\n format(self._source_file_names[routine_name], file_name))\n self.error_file_names.add(file_name)\n else:\n self._io.error(\"File not exists: '{0}'\".format(file_name))\n self.error_file_names.add(file_name)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef __get_constants(self):\n helper = ConstantClass(self._constants_class_name, self._io)\n helper.reload()\n constants = helper.constants()\n\n for name, value in constants.items():\n self._add_replace_pair(name, value, True)\n\n self._io.text('Read {0} constants for substitution from {1}'.\n format(len(constants), helper.file_name()))", "response": "Gets the constants from the class that acts like a namespace for constants and adds them to the replace pairs."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef init_logger(\n name=\"\",\n handler_path_levels=None,\n level=logging.INFO,\n formatter=None,\n formatter_str=None,\n datefmt=\"%Y-%m-%d %H:%M:%S\",\n):\n \"\"\"Add a default handler for logger.\n\n Args:\n\n name = '' or logger obj.\n\n handler_path_levels = [['loggerfile.log',13],['','DEBUG'],['','info'],['','notSet']] # [[path,level]]\n\n level = the least level for the logger.\n\n formatter = logging.Formatter(\n '%(levelname)-7s %(asctime)s %(name)s (%(filename)s: %(lineno)s): %(message)s',\n \"%Y-%m-%d %H:%M:%S\")\n\n formatter_str = '%(levelname)-7s %(asctime)s %(name)s (%(funcName)s: %(lineno)s): %(message)s'\n\n custom formatter:\n %(asctime)s %(created)f %(filename)s %(funcName)s %(levelname)s %(levelno)s %(lineno)s %(message)s %(module)s %(name)s %(pathname)s %(process)s %(relativeCreated)s %(thread)s %(threadName)s \n \"\"\"\n levels = {\n \"NOTSET\": logging.NOTSET,\n \"DEBUG\": logging.DEBUG,\n \"INFO\": logging.INFO,\n \"WARNING\": logging.WARNING,\n \"ERROR\": logging.ERROR,\n \"CRITICAL\": logging.CRITICAL,\n }\n if not formatter:\n if formatter_str:\n formatter_str = formatter_str\n else:\n formatter_str = \"%(asctime)s %(levelname)-5s [%(name)s] %(filename)s(%(lineno)s): %(message)s\"\n formatter = logging.Formatter(formatter_str, datefmt=datefmt)\n logger = name if isinstance(name, logging.Logger) else logging.getLogger(str(name))\n logger.setLevel(level)\n handler_path_levels = handler_path_levels or [[\"\", \"INFO\"]]\n # ---------------------------------------\n for each_handler in handler_path_levels:\n path, handler_level = each_handler\n handler = logging.FileHandler(path) if path else logging.StreamHandler()\n handler.setLevel(\n levels.get(handler_level.upper(), 1)\n if isinstance(handler_level, str)\n else handler_level\n )\n handler.setFormatter(formatter)\n logger.addHandler(handler)\n return logger", "response": "Initialize a new logger object."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef print_info(*messages, **kwargs):\n sep = kwargs.pop(\"sep\", \" \")\n frame = sys._getframe(1)\n ln = frame.f_lineno\n _file = frame.f_globals.get(\"__file__\", \"\")\n fn = os.path.split(_file)[-1]\n return print_logger.info(\n sep.join(map(unicode, messages)), extra={\"ln\": ln, \"fn\": fn}\n )", "response": "Simple print use logger print with time and file line number."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a list of field names for a Knockout model.", "response": "def get_fields(model):\n \"\"\"\n Returns a Model's knockout_fields,\n or the default set of field names.\n \"\"\"\n\n try:\n if hasattr(model, \"knockout_fields\"):\n fields = model.knockout_fields()\n else:\n try:\n fields = model_to_dict(model).keys()\n except Exception as e:\n fields = model._meta.get_fields()\n return fields\n\n # Crash proofing\n except Exception as e:\n logger.exception(e)\n return []"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngives an object and a list of fields recursively build an object for serialization.", "response": "def get_object_data(obj, fields, safe):\n \"\"\"\n Given an object and a list of fields, recursively build an object for serialization.\n\n Returns a dictionary.\n \"\"\"\n\n temp_dict = dict()\n for field in fields:\n\n try:\n attribute = getattr(obj, str(field))\n if isinstance(attribute, list) and all([isinstance(item, models.Model) for item in attribute]):\n temp_dict[field] = []\n for item in attribute:\n temp_dict[field].append(get_object_data(item, get_fields(item), safe)) # Recur\n elif isinstance(attribute, models.Model):\n attribute_fields = get_fields(attribute)\n object_data = get_object_data(attribute, attribute_fields, safe) # Recur\n temp_dict[field] = object_data\n else:\n if not safe:\n if isinstance(attribute, basestring):\n attribute = cgi.escape(attribute)\n temp_dict[field] = attribute\n\n except Exception as e:\n logger.info(\"Unable to get attribute.\")\n logger.error(e)\n continue\n\n return temp_dict"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the Knockout Model and Knockout ViewModel.", "response": "def ko_model(model, field_names=None, data=None):\n \"\"\"\n Given a model, returns the Knockout Model and the Knockout ViewModel.\n Takes optional field names and data.\n \"\"\"\n\n try:\n if isinstance(model, str):\n modelName = model\n else:\n modelName = model.__class__.__name__\n\n if field_names:\n fields = field_names\n else:\n fields = get_fields(model)\n\n if hasattr(model, \"comparator\"):\n comparator = str(model.comparator())\n else:\n comparator = 'id'\n\n modelViewString = render_to_string(\n \"knockout_modeler/model.js\",\n {'modelName': modelName, 'fields': fields, 'data': data, 'comparator': comparator}\n )\n\n return modelViewString\n except Exception as e:\n logger.exception(e)\n return ''"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the Knockout data bindings for a given model", "response": "def ko_bindings(model):\n \"\"\"\n Given a model, returns the Knockout data bindings.\n \"\"\"\n\n try:\n if isinstance(model, str):\n modelName = model\n else:\n modelName = model.__class__.__name__\n\n modelBindingsString = \"ko.applyBindings(new \" + modelName + \"ViewModel(), $('#\" + modelName.lower() + \"s')[0]);\"\n return modelBindingsString\n\n except Exception as e:\n logger.error(e)\n return ''"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngiving a QuerySet return just the serialized representation of the Knockout object.", "response": "def ko_json(queryset, field_names=None, name=None, safe=False):\n \"\"\"\n Given a QuerySet, return just the serialized representation\n based on the knockout_fields. Useful for middleware/APIs.\n\n Convenience method around ko_data.\n\n \"\"\"\n return ko_data(queryset, field_names, name, safe, return_json=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngiving a QuerySet return just the serialized representation of the knockout_fields as JavaScript.", "response": "def ko_data(queryset, field_names=None, name=None, safe=False, return_json=False):\n \"\"\"\n Given a QuerySet, return just the serialized representation\n based on the knockout_fields as JavaScript.\n\n \"\"\"\n\n try:\n try:\n # Get an inital instance of the QS.\n queryset_instance = queryset[0]\n except TypeError as e:\n # We are being passed an object rather than a QuerySet.\n # That's naughty, but we'll survive.\n queryset_instance = queryset\n queryset = [queryset]\n except IndexError as e:\n if not isinstance(queryset, list):\n # This is an empty QS - get the model directly.\n queryset_instance = queryset.model\n else:\n # We have been given an empty list.\n # Return nothing.\n return '[]'\n\n modelName = queryset_instance.__class__.__name__\n modelNameData = []\n\n if field_names is not None:\n fields = field_names\n else:\n fields = get_fields(queryset_instance)\n\n for obj in queryset:\n object_data = get_object_data(obj, fields, safe)\n modelNameData.append(object_data)\n\n if name:\n modelNameString = name\n else:\n modelNameString = modelName + \"Data\"\n\n dthandler = lambda obj: obj.isoformat() if isinstance(obj, (datetime.date, datetime.datetime)) else None\n dumped_json = json.dumps(modelNameData, default=dthandler)\n\n if return_json:\n return dumped_json\n return \"var \" + modelNameString + \" = \" + dumped_json + ';'\n except Exception as e:\n logger.exception(e)\n return '[]'"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef ko(queryset, field_names=None):\n\n try:\n koDataString = ko_data(queryset, field_names)\n koModelString = ko_model(queryset[0].__class__.__name__, field_names, data=True)\n koBindingsString = ko_bindings(queryset[0])\n\n koString = koDataString + '\\n' + koModelString + '\\n' + koBindingsString\n\n return koString\n except Exception as e:\n logger.error(e)\n return ''", "response": "Converts a Django QuerySet into a complete Knockout implementation."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef load_keypair(keypair_file):\n '''load a keypair from a keypair file. We add attributes key (the raw key)\n and public_key (the url prepared public key) to the client.\n\n Parameters\n ==========\n keypair_file: the pem file to load.\n '''\n from Crypto.PublicKey import RSA\n\n # Load key\n with open(keypair_file, 'rb') as filey:\n key = RSA.import_key(filey.read())\n\n return quote_plus(key.publickey().exportKey().decode('utf-8'))", "response": "load a keypair from a keypair file"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a string that can be used to construct the exception message.", "response": "def __message(expected_row_count, actual_row_count, query):\n \"\"\"\n Composes the exception message.\n\n :param str expected_row_count: The expected row count.\n :param int actual_row_count: The actual row count.\n :param str query: The query.\n\n :rtype: str\n \"\"\"\n query = query.strip()\n\n message = 'Wrong number of rows selected'\n message += os.linesep\n message += 'Expected number of rows: {}'.format(expected_row_count)\n message += os.linesep\n message += 'Actual number of rows: {}'.format(actual_row_count)\n message += os.linesep\n message += 'Query:'\n message += os.linesep if os.linesep in query else ' '\n message += query\n\n return message"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsubmitting the issue to github.", "response": "def _submit(self):\n '''submit the issue to github. When we get here we should have:\n \n {'user_prompt_issue': 'I want to do the thing.', \n 'user_prompt_repo': 'vsoch/hello-world', \n 'user_prompt_title': 'Error with this thing',\n 'record_asciinema': '/tmp/helpme.93o__nt5.json',\n 'record_environment': ((1,1),(2,2)...(N,N))}\n\n self.token should be propogated with the personal access token\n ''' \n body = self.data['user_prompt_issue']\n title = self.data['user_prompt_title']\n repo = self.data['user_prompt_repo']\n\n # Step 1: Environment\n\n envars = self.data.get('record_environment')\n body = body + envars_to_markdown(envars)\n\n # Step 2: Asciinema\n\n asciinema = self.data.get('record_asciinema')\n if asciinema not in [None, '']:\n url = upload_asciinema(asciinema)\n \n # If the upload is successful, add a link to it.\n\n if url is not None:\n body += \"\\n[View Asciinema Recording](%s)\" %url\n \n # Add other metadata about client\n\n body += \"\\n\\ngenerated by [HelpMe](https://vsoch.github.io/helpme/)\"\n body += \"\\nHelpMe Github Issue: %s\" %(self.run_id)\n\n # Submit the issue\n\n issue = create_issue(title, body, repo, self.token)\n return issue"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nresolving JSON references in a given OpenAPI spec.", "response": "def resolve_refs(uri, spec):\n \"\"\"Resolve JSON references in a given dictionary.\n\n OpenAPI spec may contain JSON references to its nodes or external\n sources, so any attempt to rely that there's some expected attribute\n in the spec may fail. So we need to resolve JSON references before\n we use it (i.e. replace with referenced object). For details see:\n\n https://tools.ietf.org/html/draft-pbryan-zyp-json-ref-02\n\n The input spec is modified in-place despite being returned from\n the function.\n \"\"\"\n resolver = jsonschema.RefResolver(uri, spec)\n\n def _do_resolve(node):\n if isinstance(node, collections.Mapping) and '$ref' in node:\n with resolver.resolving(node['$ref']) as resolved:\n result = deepcopy(resolved)\n for key in resolved:\n if key in node:\n merge(result[key], node[key])\n return result\n elif isinstance(node, collections.Mapping):\n for k, v in node.items():\n node[k] = _do_resolve(v)\n elif isinstance(node, (list, tuple)):\n for i in range(len(node)):\n node[i] = _do_resolve(node[i])\n return node\n\n return _do_resolve(spec)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef main(args, extras):\n '''This is the actual driver for the helper.\n '''\n\n from helpme.main import get_helper\n name = args.command\n\n if name in HELPME_HELPERS:\n\n # Get the helper, do the recording, submit\n\n helper = get_helper(name=name)\n\n if args.asciinema is not None:\n if os.path.exists(args.asciinema):\n helper.data['record_asciinema'] = args.asciinema\n \n helper.run(positionals=extras)", "response": "This is the main function for the helper. It is the main function for the helper."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nextracting the cleaned html tree as a string and only css rules matching the cleaned html tree.", "response": "def extract(self, html_contents, css_contents=None, base_url=None):\n \"\"\"\n Extracts the cleaned html tree as a string and only\n css rules matching the cleaned html tree\n\n :param html_contents: The HTML contents to parse\n :type html_contents: str\n :param css_contents: The CSS contents to parse\n :type css_contents: str\n :param base_url: The base page URL to use for relative to absolute links\n :type base_url: str\n\n :returns: cleaned HTML contents, cleaned CSS contents\n :rtype: str or tuple\n \"\"\"\n # Clean HTML\n html_extractor = self.html_extractor(\n html_contents, self._xpaths_to_keep, self._xpaths_to_discard)\n has_matches = html_extractor.parse()\n\n if has_matches:\n\n # Relative to absolute URLs\n if base_url is not None:\n html_extractor.rel_to_abs(base_url)\n\n # Convert ElementTree to string\n cleaned_html = html_extractor.to_string()\n\n else:\n cleaned_html = None\n\n # Clean CSS\n if css_contents is not None:\n\n if cleaned_html is not None:\n\n css_extractor = self.css_extractor(css_contents, cleaned_html)\n css_extractor.parse()\n\n # Relative to absolute URLs\n if base_url is not None:\n css_extractor.rel_to_abs(base_url)\n\n cleaned_css = css_extractor.to_string()\n\n else:\n cleaned_css = None\n\n else:\n return cleaned_html\n\n return (cleaned_html, cleaned_css)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nadding a Xpath expression to the dest list", "response": "def __add(self, dest, xpath):\n \"\"\"\n Adds a Xpath expression to the dest list\n\n :param dest: The destination list to add the Xpath\n :type dest: list\n :param xpath: The Xpath expression to add\n :type xpath: str\n \"\"\"\n assert isinstance(xpath, string_types)\n dest.append(xpath)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nappends a line of code to the generated code and adjusts the indent level of the generated code.", "response": "def _write_line(self, line=None):\n \"\"\"\n Appends a line of code to the generated code and adjust the indent level of the generated code.\n\n :param line: The line of code (with out LF) that must be appended.\n \"\"\"\n if line is None:\n self._write(\"\\n\")\n if self.__indent_level > 1:\n self.__indent_level -= 1\n elif line == '':\n self._write(\"\\n\")\n else:\n line = (' ' * 4 * self.__indent_level) + line\n if line[-1:] == ':':\n self.__indent_level += 1\n self._write(line + \"\\n\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nwriting a separator line at the top of the generated code.", "response": "def _write_separator(self):\n \"\"\"\n Inserts a horizontal (commented) line tot the generated code.\n \"\"\"\n tmp = self._page_width - ((4 * self.__indent_level) + 2)\n self._write_line('# ' + ('-' * tmp))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef write_routine_method(self, routine):\n if self._lob_as_string_flag:\n return self._write_routine_method_without_lob(routine)\n else:\n if self.is_lob_parameter(routine['parameters']):\n return self._write_routine_method_with_lob(routine)\n else:\n return self._write_routine_method_without_lob(routine)", "response": "Returns a complete wrapper method."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _write_docstring_parameters(self, routine):\n if routine['pydoc']['parameters']:\n self._write_line('')\n\n for param in routine['pydoc']['parameters']:\n lines = param['description'].split(os.linesep)\n self._write_line(':param {0} {1}: {2}'.format(param['python_type'],\n param['parameter_name'],\n lines[0]))\n del lines[0]\n\n tmp = ':param {0} {1}:'.format(param['python_type'], param['parameter_name'])\n indent = ' ' * len(tmp)\n for line in lines:\n self._write_line('{0} {1}'.format(indent, line))\n\n self._write_line('{0} {1}'.format(indent, param['data_type_descriptor']))", "response": "Writes the parameters part of the docstring for the wrapper method of a stored routine."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nwrite the return type part of the docstring for the stored routine.", "response": "def __write_docstring_return_type(self):\n \"\"\"\n Writes the return type part of the docstring for the wrapper method of a stored routine.\n \"\"\"\n rtype = self._get_docstring_return_type()\n if rtype:\n self._write_line('')\n self._write_line(':rtype: {0}'.format(rtype))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nwrite the docstring for the wrapper method of a stored routine.", "response": "def __write_docstring(self, routine):\n \"\"\"\n Writes the docstring for the wrapper method of a stored routine.\n\n :param dict routine: The metadata of the stored routine.\n \"\"\"\n self._write_line('\"\"\"')\n\n self.__write_docstring_description(routine)\n self._write_docstring_parameters(routine)\n self.__write_docstring_return_type()\n\n self._write_line('\"\"\"')"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_wrapper_args(routine):\n ret = ''\n\n for parameter_info in routine['parameters']:\n if ret:\n ret += ', '\n\n ret += parameter_info['name']\n\n return ret", "response": "Returns code for the parameters of the wrapper method for the stored routine."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ngenerate the ld - relay. conf file.", "response": "def generate_relay_config(self, environments: list) -> None:\n \"\"\"Generate ld-relay.conf file.\n\n Given a list of environments of a project, this will generate a\n ``ld-relay.conf`` file in the current working directory. The conf file\n follows the specification that is documented in the main `ld-relay`_\n documentation.\n\n .. _ld-relay: https://github.com/launchdarkly/ld-relay#configuration-file-format\n\n :param environments: list of LaunchDarkly environments.\n \"\"\"\n template = self.env.get_template('ld-relay.conf.jinja')\n\n with open('ld-relay.conf', 'w') as ld_relay_config:\n template = template.render(\n envs=environments\n )\n ld_relay_config.write(template)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nload dynamically the class that acts like a namespace for constants.", "response": "def __load(self):\n \"\"\"\n Loads dynamically the class that acts like a namespace for constants.\n \"\"\"\n parts = self.__class_name.split('.')\n module_name = \".\".join(parts[:-1])\n module = __import__(module_name)\n modules = []\n for comp in parts[1:]:\n module = getattr(module, comp)\n modules.append(module)\n\n self.__module = modules[-2]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef constants(self):\n ret = {}\n\n name = self.__class_name.split('.')[-1]\n constant_class = getattr(self.__module, name)\n for name, value in constant_class.__dict__.items():\n if re.match(r'^[A-Z][A-Z0-9_]*$', name):\n ret[name] = value\n\n return ret", "response": "Returns the constants from the class that acts like a namespace for constants."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nextracting the following info from the source of the module with the class that acts like a namespace for constants.", "response": "def __extract_info(self, lines):\n \"\"\"\n Extracts the following info from the source of the module with the class that acts like a namespace for\n constants:\n * Start line with constants\n * Last line with constants\n * Indent for constants\n\n :param list[str] lines: The source of the module with the class that acts like a namespace for constants.\n\n :rtype: dict\n \"\"\"\n ret = {'start_line': 0,\n 'last_line': 0,\n 'indent': ''}\n\n mode = 1\n count = 0\n for line in lines:\n if mode == 1:\n if line.strip() == self.__annotation:\n ret['start_line'] = count + 1\n ret['last_line'] = count + 1\n parts = re.match(r'^(\\s+)', line)\n ret['indent'] = parts.group(1)\n mode = 2\n\n elif mode == 2:\n if line.strip() == '' or line.strip()[0:1] == '#':\n mode = 3\n else:\n ret['last_line'] = count + 1\n\n else:\n break\n\n count += 1\n\n if mode != 3:\n raise RuntimeError(\"Unable to find '{}' in file {}\".format(self.__annotation, self.source()))\n\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the source of the module with the class that acts like a namespace for constants with new constants.", "response": "def source_with_constants(self, constants):\n \"\"\"\n Returns the source of the module with the class that acts like a namespace for constants with new constants.\n\n :param dict[str,int] constants: The new constants.\n\n :rtype: str\n \"\"\"\n old_lines = self.source().split(\"\\n\")\n info = self.__extract_info(old_lines)\n\n new_lines = old_lines[0:info['start_line']]\n\n for constant, value in sorted(constants.items()):\n new_lines.append(\"{0}{1} = {2}\".format(info['indent'], str(constant), str(value)))\n\n new_lines.extend(old_lines[info['last_line']:])\n\n return \"\\n\".join(new_lines)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_configfile_user():\n '''return the full path for the user configuration file. If doesn't\n exist, create it for the user.\n '''\n from helpme.defaults import HELPME_CLIENT_SECRETS\n\n # The inital file has a funny username\n\n if not os.path.exists(HELPME_CLIENT_SECRETS):\n bot.debug('Generating settings file at %s' % HELPME_CLIENT_SECRETS)\n config_dir = os.path.dirname(HELPME_CLIENT_SECRETS)\n\n # The configuration directory might be needed for different clients\n if not os.path.exists(config_dir):\n mkdir_p(config_dir)\n\n name = RobotNamer().generate()\n\n # Generate the user config\n\n config = configparser.ConfigParser()\n config['DEFAULT'] = {'Alias': name }\n write_config(HELPME_CLIENT_SECRETS, config)\n\n return HELPME_CLIENT_SECRETS", "response": "return the full path for the user configuration file. If doesn t exist create it."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _remove_setting(section, name, configfile, save=False):\n '''remove a setting from the global config\n '''\n removed = False\n config = _load_config(configfile)\n if section in config:\n if name.lower() in config[section]:\n removed = config.remove_option(section, name)\n\n # Does the user want to save the file?\n if removed and save:\n write_config(configfile, config)\n\n return removed", "response": "remove a setting from the global config file"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef remove_setting(self, section, name, save=False):\n '''remove a setting from the global config\n '''\n configfile = get_configfile()\n return _remove_setting(section, name, configfile, save)", "response": "remove a setting from the global config"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef remove_user_setting(self, section, name, save=False):\n '''remove a setting from the user config\n '''\n configfile = get_configfile_user()\n return _remove_setting(section, name, configfile, save)", "response": "remove a setting from the user config"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nload a tuple of environment variables to add to the user settings", "response": "def load_envars(self, items):\n '''load a tuple of environment variables, to add to the user settings\n \n Example:\n \n items = [('HELPME_DISCOURSE_BOARD', 'user_prompt_board'),\n ('HELPME_DISCOURSE_CATEGORY', 'user_prompt_category')]\n\n # Note that it's added to the client with an underscore:\n self._load_envars()\n ''' \n for item in items:\n envar = item[0]\n key = item[1]\n value = self._get_and_update_setting(envar)\n\n if value != None:\n self.data[key] = value\n self.config.remove_option(self.name, key)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a setting from the environment then secrets file or main config.", "response": "def get_setting(self, name, section=None, default=None, user=True):\n '''return a setting from the environment (first priority) and then\n secrets (second priority) if one can be found. If not, return None.\n\n Parameters\n ==========\n section: the section in the config, defaults to self.name\n name: they key (index) of the setting to look up\n default: (optional) if not found, return default instead.\n user: if True, load from user config. Otherwise, main config\n\n ''' \n loader = self._load_config_user\n if not user:\n loader = self._load_config\n\n if section is None:\n section = self.name\n\n # First priority is the environment\n\n setting = os.environ.get(name)\n\n # Second priority is the secrets file\n\n if setting is None:\n\n # Loads user config on level of helper (already indexed)\n config = loader()\n if section in config:\n if name.lower() in config[section]:\n setting = config[section][name.lower()]\n\n # Third priority, return a default\n\n if setting is None and default is not None:\n setting = default\n\n return setting"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget all settings for a client", "response": "def get_settings(self):\n '''get all settings for a client, if defined in config.\n '''\n config = self._load_config_user()\n if self.name in config:\n return config[self.name]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nupdating client secrets will update the data structure for a particular token authentication.", "response": "def update_settings(self, updates, config=None):\n '''update client secrets will update the data structure for a particular\n authentication. This should only be used for a (quasi permanent) token\n or similar. The secrets file, if found, is updated and saved by default.\n\n Parameters\n ==========\n helper: the name of the helper to look up in the config\n updates: a dictionary of key:value pairs to add to the config\n config: a configparser.ConfigParser(), if already loaded\n\n '''\n if config is None:\n config = self._load_config_user()\n\n if self.name not in config:\n config[self.name] = {}\n\n config[self.name].update(updates)\n\n # Update the saved file\n\n configfile = get_configfile_user()\n write_config(configfile, config)\n return config"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_and_update_setting(self, name, default=None, user=True):\n '''Look for a setting in the environment (first priority) and then\n the settings file (second). If something is found, the settings\n file is updated. The order of operations works as follows:\n\n 1. The user config file is used as a cache for the variable\n 2. the environment variable always takes priority to cache, and if\n found, will update the cache.\n 3. If the variable is not found and the cache is set, we are good\n 5. If the variable is not found and the cache isn't set, return\n default (default is None)\n\n So the user of the function can assume a return of None equates to\n not set anywhere, and take the appropriate action.\n ''' \n\n setting = self._get_setting(name, user=user)\n\n if setting is None and default is not None:\n setting = default\n\n # If the setting is found, update the client secrets\n if setting is not None:\n updates = {name : setting}\n self._update_settings(updates)\n\n return setting", "response": "Look for a setting in the environment and then update the client secrets with the new values."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef run(app, argv=sys.argv, extra_args=None):\n if app not in argv[:2]:\n # app is automatically added if not present\n argv.insert(1, app)\n if len(argv) < 3 and 'test' not in argv[:2]:\n # test argument is given if not argument is passed\n argv.insert(2, 'test')\n if extra_args:\n argv.extend(extra_args)\n return runner(argv)", "response": "Run commands in a plain django environment"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nrunning commands in a django CMS environment", "response": "def cms(app, argv=sys.argv, extra_args=None):\n \"\"\"\n Run commands in a django cMS environment\n\n :param app: application\n :param argv: arguments (default to sys.argv)\n :param extra_args: list of extra arguments\n \"\"\"\n try:\n import cms # NOQA # nopyflakes\n except ImportError:\n print('runner.cms is available only if django CMS is installed')\n raise\n if app not in argv[:2]:\n # app is automatically added if not present\n argv.insert(1, app)\n if len(argv) < 3 and 'test' not in argv[:2]:\n # test argument is given if not argument is passed\n argv.insert(2, 'test')\n if '--cms' not in argv:\n # this is the cms runner, just add the cms argument\n argv.append('--cms')\n if extra_args:\n argv.extend(extra_args)\n return runner(argv)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef setup(app, helper_module, extra_args=None, use_cms=False):\n helper = helper_module.__file__\n argv = [os.path.basename(helper), app, 'setup', '--extra-settings={0}'.format(helper)]\n if use_cms:\n argv.append('--cms')\n if extra_args:\n argv.extend(extra_args)\n return runner(argv)", "response": "Setup the Django environment and return the environment settings."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nparsing the HTML and returns a cleaned lxml ElementTree with the elements removed.", "response": "def parse(self):\n \"\"\"\n Returns a cleaned lxml ElementTree\n\n :returns: Whether the cleaned HTML has matches or not\n :rtype: bool\n \"\"\"\n # Create the element tree\n self.tree = self._build_tree(self.html_contents)\n\n # Get explicits elements to keep and discard\n self.elts_to_keep = self._get_elements_to_keep()\n self.elts_to_discard = self._get_elements_to_discard()\n\n # Init an empty list of Elements to remove\n self.elts_to_remove = []\n\n # Check if the root is a match or if there is any matches\n is_root = self._is_keep(self.tree)\n has_descendant = self._has_keep_elt_in_descendants(self.tree)\n\n if not(is_root or has_descendant):\n return False\n\n # Parse and clean the ElementTree\n self._parse_element(self.tree, parent_is_keep=is_root)\n self._remove_elements(self.elts_to_remove)\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef rel_to_abs(self, base_url):\n # Delete target attributes\n strip_attributes(self.tree, 'target')\n\n # Absolute links\n self.tree.rewrite_links(\n lambda link: urljoin(base_url, link)\n if not link.startswith(self.rel_to_abs_excluded_prefixes) else link)\n\n # Extra attributes\n onclick_elements = self.tree.xpath('//*[@onclick]')\n\n for element in onclick_elements:\n # Replace attribute with absolute URL\n element.set('onclick', self.javascript_open_re.sub(\n lambda match: '%s%s%s' % (match.group('opening'),\n urljoin(base_url, match.group('url')),\n match.group('ending')),\n element.get('onclick')))", "response": "Converts relative links from html contents to absolute links"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the list of HtmlElements for the source", "response": "def _get_elements(self, source):\n \"\"\"\n Returns the list of HtmlElements for the source\n\n :param source: The source list to parse\n :type source: list\n :returns: A list of HtmlElements\n :rtype: list\n \"\"\"\n return list(chain(*[self.tree.xpath(xpath) for xpath in source]))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _parse_element(self, elt, parent_is_keep=False):\n for e in elt.iterchildren():\n\n is_discard_element = self._is_discard(e)\n is_keep_element = self._is_keep(e)\n\n # Element is an explicit one to discard, flag it and continue\n if is_discard_element and not is_keep_element:\n self.elts_to_remove.append(e)\n continue\n\n if not parent_is_keep:\n # Parent element is not an explicit keep, normal process\n # Element is an explicit one to keep, inspect it\n if is_keep_element:\n self._parse_element(e, parent_is_keep=True)\n continue\n\n # Has a descendant to keep, inspect it\n if self._has_keep_elt_in_descendants(e):\n self._parse_element(e)\n continue\n\n # Element did not match anything, remove it\n self.elts_to_remove.append(e)\n\n else:\n # Element is a child of a keep element, only check explicit discards\n self._parse_element(e, parent_is_keep=True)", "response": "Parses an element recursively and returns a list of all the related elements."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns whether the element has a descendant to keep or not", "response": "def _has_keep_elt_in_descendants(self, elt):\n \"\"\"\n Returns whether the element has a descendant to keep or not\n\n :param elt: The HtmlElement to check\n :type elt: lxml.html.HtmlElement\n :returns: True if the element has a keep element in its descendants\n :rtype: bool\n \"\"\"\n # iterdescendants is a generator, don't cast it as a list to avoid\n # parsing the whole descendants tree if not necessary\n for d in elt.iterdescendants():\n if d in self.elts_to_keep:\n return True\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nremoves flagged elements from the ElementTree.", "response": "def _remove_elements(self, elts_to_remove):\n \"\"\"\n Removes flagged elements from the ElementTree\n \"\"\"\n for e in elts_to_remove:\n\n # Get the element parent\n parent = e.getparent()\n\n # lxml also remove the element tail, preserve it\n if e.tail and e.tail.strip():\n parent_text = parent.text or ''\n parent.text = parent_text + e.tail\n\n # Remove the element\n e.getparent().remove(e)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef getSystemInfo(self, timeout=1):\n namespace = System.getServiceType(\"getSystemInfo\")\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetInfo\", timeout=timeout)\n\n return SystemInfo(results)", "response": "Execute GetInfo action to get information about the System on the device."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef getTimeInfo(self, timeout=1):\n namespace = System.getServiceType(\"getTimeInfo\")\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetInfo\", timeout=timeout)\n\n return TimeInfo(results)", "response": "Execute GetInfo action to get information about the time on the device."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning if a software update is available for this service.", "response": "def softwareUpdateAvailable(self, timeout=1):\n \"\"\"Returns if a software update is available\n\n :return: if a software update is available\n :rtype: bool\n \"\"\"\n namespace = System.getServiceType(\"softwareUpdateAvailable\")\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetInfo\", timeout=timeout)\n\n return bool(int(results[\"NewUpgradeAvailable\"]))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef make_headers(headers):\n out = {}\n if 'etag' in headers:\n out['if-none-match'] = headers['etag']\n if 'last-modified' in headers:\n out['if-modified-since'] = headers['last-modified']\n return out", "response": "Make the cache control headers based on a previous request s\n response headers\n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get(self, prefix, url, schema_version=None):\n if not self.cache_dir:\n return None\n\n filename = self._get_cache_file(prefix, url)\n\n try:\n with open(filename, 'rb') as file:\n item = pickle.load(file)\n if schema_version and schema_version != item.schema:\n LOGGER.debug(\"Cache get %s %s: Wanted schema %d, got %d\",\n prefix, url,\n schema_version, item.schema)\n return None\n return item\n except FileNotFoundError:\n pass\n except Exception: # pylint:disable=broad-except\n _, msg, _ = sys.exc_info()\n LOGGER.warning(\"Cache get %s %s failed: %s\", prefix, url, msg)\n\n return None", "response": "Get the cached object for a given url."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set(self, prefix, url, obj):\n if not self.cache_dir:\n return\n\n filename = self._get_cache_file(prefix, url)\n\n try:\n os.makedirs(os.path.join(self.cache_dir, prefix))\n except OSError:\n pass\n\n with open(filename, 'wb') as file:\n pickle.dump(obj, file)", "response": "Add an object into the cache"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\njoining input path to sample data path", "response": "def join_path(*path_to_join, **kwargs):\n \"\"\"Join input path to sample data path (usually in ~/lisa_data)\n\n :param path_to_join: one or more paths\n :param get_root: return dataset root path. If false, the path would be into \"medical/orig\"\n :return: joined path\n \"\"\"\n if \"get_root\" in kwargs:\n get_root = kwargs[\"get_root\"]\n else:\n # default value\n get_root = False\n sdp = dataset_path(get_root=get_root)\n pth = os.path.join(sdp, *path_to_join)\n logger.debug(\"sample_data_path\" + str(sdp))\n logger.debug(\"path \" + str(pth))\n return pth"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_dataset_path(path, cache=None, cachefile=\"~/.io3d_cache.yaml\"):\n if cachefile is not None:\n cache = cachef.CacheFile(cachefile)\n cache.update(\"local_dataset_dir\", path)", "response": "Sets path to dataset."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting dataset path. :param cache: CacheFile object :param cachefile: cachefile path, default '~/.io3d_cache.yaml' :return: path to dataset", "response": "def dataset_path(cache=None, cachefile=\"~/.io3d_cache.yaml\", get_root=False):\n \"\"\"Get dataset path.\n\n :param cache: CacheFile object\n :param cachefile: cachefile path, default '~/.io3d_cache.yaml'\n :return: path to dataset\n \"\"\"\n local_data_dir = local_dir\n\n if cachefile is not None:\n cache = cachef.CacheFile(cachefile)\n # cache.update('local_dataset_dir', head)\n if cache is not None:\n local_data_dir = cache.get_or_save_default(\"local_dataset_dir\", local_dir)\n\n if get_root:\n local_data_dir\n else:\n logger.warning(\"Parameter\")\n local_data_dir = op.join(local_data_dir, \"medical\", \"orig\")\n\n return op.expanduser(local_data_dir)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ngive you metadata for dataset chosen via label", "response": "def get_dataset_meta(label):\n \"\"\"Gives you metadata for dataset chosen via 'label' param\n\n :param label: label = key in data_url dict (that big dict containing all possible datasets)\n :return: tuple (data_url, url, expected_hash, hash_path, relative_download_dir)\n relative_download_dir says where will be downloaded the file from url and eventually unzipped\n\n \"\"\"\n data_url = data_urls[label]\n if type(data_url) == str:\n # back compatibility\n data_url = [data_url]\n if type(data_url) == list:\n data_url.extend([None, None, None, None])\n data_url = data_url[:4]\n url, expected_hash, hash_path, relative_donwload_dir = data_url\n if hash_path is None:\n hash_path = label\n # elif type(data_url) == dict:\n return data_url, url, expected_hash, hash_path, relative_donwload_dir"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns list of possible packages contained in dataset in multi dataset eg. lisa.", "response": "def _expand_dataset_packages(dataset_label_dict):\n \"\"\"Returns list of possible packages contained in dataset, in case the dataset is multi dataset, eg. 'lisa'.\n\n In case the param is not pointing to multidataset returns only that label in a list.\n\n :param str dataset_label_dict: label of multi dataset\n :return: list of labels\n \"\"\"\n new_dataset_label_dict = []\n for label in dataset_label_dict:\n dataset_metadata = data_urls[label]\n if type(dataset_metadata) == dict and \"package\" in dataset_metadata:\n new_dataset_label_dict.extend(dataset_metadata[\"package\"])\n else:\n new_dataset_label_dict.append(label)\n return new_dataset_label_dict"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndownloads sample data by label.", "response": "def download(dataset_label=None, destination_dir=None, dry_run=False):\n \"\"\"Download sample data by data label. Warning: function with side effect!\n\n Labels can be listed by sample_data.data_urls.keys(). Returns downloaded files.\n\n :param dataset_label: label of data. If it is set to None, all data are downloaded\n :param destination_dir: output dir for data\n :param dry_run: runs function without downloading anything\n \"\"\"\n if destination_dir is None:\n destination_dir = op.join(dataset_path(get_root=True), \"medical\", \"orig\")\n\n destination_dir = op.expanduser(destination_dir)\n logger.info(\"destination dir: {}\".format(destination_dir))\n\n if dataset_label is None:\n dataset_label = data_urls.keys()\n\n if type(dataset_label) == str:\n dataset_label = [dataset_label]\n\n dataset_label = _expand_dataset_packages(dataset_label)\n\n for label in dataset_label:\n # make all data:url have length 3\n data_url, url, expected_hash, hash_path, relative_download_dir = get_dataset_meta(\n label\n )\n if relative_download_dir is None:\n label_destination_dir = destination_dir\n else:\n label_destination_dir = op.join(destination_dir, relative_download_dir)\n if not op.exists(label_destination_dir):\n logger.debug(\"creating directory {}\".format(label_destination_dir))\n os.makedirs(label_destination_dir)\n\n if hash_path is None:\n hash_path = label\n\n path_to_hash = os.path.join(label_destination_dir, hash_path)\n try:\n computed_hash = checksum(path_to_hash)\n except Exception as e:\n # there is probably no checksumdir module\n logger.warning(e)\n logger.warning(\"problem with sample_data.checksum()\")\n computed_hash = None\n\n logger.info(\"dataset: '\" + label + \"'\")\n logger.info(\"path to hash: {}\".format(path_to_hash))\n logger.info(\"expected hash: '\" + str(expected_hash) + \"'\")\n logger.info(\"computed hash: '\" + str(computed_hash) + \"'\")\n if (computed_hash is not None) and (expected_hash == computed_hash):\n logger.info(\"match ok - no download needed\")\n else:\n logger.info(\"downloading\")\n if not dry_run:\n downzip(url, destination=label_destination_dir)\n logger.info(\"finished\")\n downloaded_hash = checksum(\n os.path.join(label_destination_dir, hash_path)\n )\n logger.info(\"downloaded hash: '\" + str(downloaded_hash) + \"'\")\n if downloaded_hash != expected_hash:\n logger.warning(\n \"downloaded hash is different from expected hash\\n\"\n + \"expected hash: '\"\n + str(expected_hash)\n + \"'\\n\"\n + \"downloaded hash: '\"\n + str(downloaded_hash)\n + \"'\\n\"\n )\n else:\n logger.debug(\"dry run\")"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_old(dataset_label, data_id, destination_dir=None):\n # TODO implement\n if destination_dir is None:\n destination_dir = op.join(dataset_path(get_root=True), \"medical\", \"orig\")\n\n destination_dir = op.expanduser(destination_dir)\n data_url, url, expected_hash, hash_path, relative_output_path = get_dataset_meta(\n dataset_label\n )\n paths = glob.glob(os.path.join(destination_dir, hash_path))\n paths.sort()\n import fnmatch\n\n print(paths)\n print(data_id)\n pathsf = fnmatch.filter(paths, data_id)\n print(pathsf)\n datap = io3d.read(pathsf[0], dataplus_format=True)\n return datap", "response": "Get the 3D data from specified dataset with specified id."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns checksum of files given by path.", "response": "def checksum(path, hashfunc=\"md5\"):\n \"\"\"Return checksum of files given by path.\n\n Wildcards can be used in check sum. Function is strongly dependent on checksumdir package by 'cakepietoast'.\n\n :param path: path of files to get hash from\n :param hashfunc: function used to get hash, default 'md5'\n :return: (str) hash of the file/files given by path\n \"\"\"\n import checksumdir\n\n hash_func = checksumdir.HASH_FUNCS.get(hashfunc)\n if not hash_func:\n raise NotImplementedError(\"{} not implemented.\".format(hashfunc))\n\n if os.path.isdir(path):\n return checksumdir.dirhash(path, hashfunc=hashfunc)\n\n hashvalues = []\n path_list = list(sorted(glob.glob(path)))\n logger.debug(\"path_list: len: %i\", len(path_list))\n if len(path_list) > 0:\n logger.debug(\"first ... last: %s ... %s\", str(path_list[0]), str(path_list[-1]))\n\n for path in path_list:\n if os.path.isfile(path):\n hashvalues.append(checksumdir._filehash(path, hashfunc=hash_func))\n logger.debug(\"one hash per file: len: %i\", len(hashvalues))\n if len(path_list) > 0:\n logger.debug(\"first ... last: %s ... %s\", str(hashvalues[0]), str(hashvalues[-1]))\n checksum_hash = checksumdir._reduce_hash(hashvalues, hashfunc=hash_func)\n logger.debug(\"total hash: {}\".format(str(checksum_hash)))\n return checksum_hash"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef generate_donut():\n segmentation = np.zeros([20, 30, 40])\n # generate test data\n segmentation[6:10, 7:24, 10:37] = 1\n segmentation[6:10, 7, 10] = 0\n segmentation[6:10, 23, 10] = 0\n segmentation[6:10, 7, 36] = 0\n segmentation[6:10, 23, 36] = 0\n segmentation[2:18, 12:19, 18:28] = 2\n\n data3d = segmentation * 100 + np.random.random(segmentation.shape) * 30\n voxelsize_mm = [3, 2, 1]\n\n datap = {\n \"data3d\": data3d,\n \"segmentation\": segmentation,\n \"voxelsize_mm\": voxelsize_mm,\n }\n # io3d.write(datap, \"donut.pklz\")\n return datap", "response": "Generate donut like shape with stick inside\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef generate_round_data(\n sz=32, offset=0, radius=7, seedsz=3, add_object_without_seeds=False\n):\n \"\"\"\n Generate data with two sphere objects.\n :param sz: output data shape is [sz, sz+1, sz+2]\n :param offset:\n :param radius:\n :param seedsz:\n :param add_object_without_seeds: Add also one cube-like object in the corner.\n :return:\n \"\"\"\n\n import scipy.ndimage\n\n # seedsz= int(sz/10)\n space = 2\n seeds = np.zeros([sz, sz + 1, sz + 2], dtype=np.int8)\n xmin = radius + seedsz + offset + 2\n ymin = radius + seedsz + offset + 6\n seeds[offset + 12, xmin + 3 : xmin + 7 + seedsz, ymin : ymin + 2] = 1\n seeds[offset + 20, xmin + 7 : xmin + 12 + seedsz, ymin + 5 : ymin + 7] = 1\n\n # add temp seed\n if add_object_without_seeds:\n seeds[-3, -3, -3] = 1\n img = np.ones([sz, sz + 1, sz + 2])\n img = img - seeds\n\n seeds[2 : 10 + seedsz, 2 : 9 + seedsz, 2 : 3 + seedsz] = 2\n\n # remove temo seed\n if add_object_without_seeds:\n seeds[-3, -3, -3] = 0\n\n img = scipy.ndimage.morphology.distance_transform_edt(img)\n segm = img < radius\n img = (100 * segm + 80 * np.random.random(img.shape)).astype(np.uint8)\n return img, segm, seeds", "response": "Generate data with two sphere objects."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate synthetic data. There is some liver and porta -like object. :return data3d, segmentation, voxelsize_mm, slab, seeds_liver, seeds_porta:", "response": "def generate_synthetic_liver(return_dataplus=False):\n \"\"\"\n Create synthetic data. There is some liver and porta -like object.\n :return data3d, segmentation, voxelsize_mm, slab, seeds_liver, seeds_porta:\n \"\"\"\n # data\n slab = {\"none\": 0, \"liver\": 1, \"porta\": 2}\n voxelsize_mm = np.array([1.0, 1.0, 1.2])\n\n segm = np.zeros([80, 256, 250], dtype=np.int16)\n\n # liver\n segm[30:60, 70:180, 40:190] = slab[\"liver\"]\n\n # porta\n segm[40:45, 120:130, 70:190] = slab[\"porta\"]\n segm[41:44, 122:127, 68:70] = slab[\n \"porta\"\n ] # hack to fix stability of skeleton algorithm\n #\n segm[40:45, 80:130, 100:110] = slab[\"porta\"]\n segm[42:44, 77:80, 103:106] = slab[\n \"porta\"\n ] # hack to fix stability of skeleton algorithm\n # segm[41:44, 78:80, 101:109] = slab['porta']\n # vertical branch under main branch\n segm[40:44, 120:170, 130:135] = slab[\"porta\"]\n\n data3d = np.zeros(segm.shape)\n data3d[segm == slab[\"liver\"]] = 146\n data3d[segm == slab[\"porta\"]] = 206\n noise = np.random.normal(0, 10, segm.shape) # .astype(np.int16)\n data3d = (data3d + noise).astype(np.int16)\n\n seeds_liver = np.zeros(data3d.shape, np.int8)\n seeds_liver[40:55, 90:120, 70:110] = 1\n seeds_liver[30:45, 190:200, 40:90] = 2\n\n seeds_porta = np.zeros(data3d.shape, np.int8)\n seeds_porta[40:45, 121:139, 80:95] = 1\n\n if return_dataplus:\n datap = {\n \"data3d\": data3d,\n \"voxelsize_mm\": voxelsize_mm,\n \"slab\": slab,\n \"seeds_liver\": seeds_liver,\n \"seeds_porta\": seeds_porta,\n \"segmentation\": segm,\n }\n return datap\n else:\n return data3d, segm, voxelsize_mm, slab, seeds_liver, seeds_porta"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating 2D binar face with 2D smiles.", "response": "def _get_face2(shape=None, face_r=1.0, smile_r1=0.5, smile_r2=0.7, eye_r=0.2):\n \"\"\"\n Create 2D binar face\n :param shape:\n :param face_r:\n :param smile_r1:\n :param smile_r2:\n :param eye_r:\n :return:\n \"\"\"\n\n # data3d = np.zeros([1,7,7], dtype=np.int16)\n if shape is None:\n shape = [32, 32]\n\n center = (np.asarray(shape) - 1) / 2.0\n r = np.min(center) * face_r\n\n # np.min(np.asarray(shape) / 2.0)\n # shape = data3d.shape[1:]\n # data3d[center[0], center[1], center[2]] = 1\n x, y = np.meshgrid(range(shape[1]), range(shape[0]))\n\n head = (x - center[0]) ** 2 + (y - center[1]) ** 2 < r ** 2\n\n smile = (\n ((x - center[0]) ** 2 + (y - center[1]) ** 2 < (r * smile_r2) ** 2)\n & (y > (center[1] + 0.3 * r))\n & ((x - center[0]) ** 2 + (y - center[1]) ** 2 >= (r * smile_r1) ** 2)\n )\n smile\n e1c = center + r * np.array([-0.35, -0.2])\n e2c = center + r * np.array([0.35, -0.2])\n\n eyes = (x - e1c[0]) ** 2 + (y - e1c[1]) ** 2 <= (r * eye_r) ** 2\n eyes += (x - e2c[0]) ** 2 + (y - e1c[1]) ** 2 <= (r * eye_r) ** 2\n\n face = head & ~smile & ~eyes\n return face"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef generate_face(shape=None, face_r=1.0, smile_r1=0.5, smile_r2=0.7, eye_r=0.2):\n # TODO add axis (ax=0)\n if shape is None:\n shape = [32, 32]\n nd = len(shape)\n if nd == 2:\n sh2 = shape\n else:\n sh2 = shape[1:]\n fc2 = _get_face2(\n sh2, face_r=face_r, smile_r1=smile_r1, smile_r2=smile_r2, eye_r=eye_r\n )\n\n if nd == 2:\n return fc2\n else:\n fc3 = np.zeros(shape)\n for i in range(fc3.shape[0]):\n fc3[i, :, :] = fc2\n return fc3", "response": "Generate 2D or 3D binar data with smile face."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfunctioning attempts to remove file from the system", "response": "def remove(local_file_name):\n \"\"\"Function attempts to remove file, if failure occures -> print exception\n\n :param local_file_name: name of file to remove\n \"\"\"\n try:\n os.remove(local_file_name)\n except Exception as e:\n print(\n \"Cannot remove file '\" + local_file_name + \"'. Please remove it manually.\"\n )\n print(e)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndownloads unzip and delete.", "response": "def downzip(url, destination=\"./sample_data/\"):\n \"\"\"Download, unzip and delete. Warning: function with strong side effects!\n\n Returns downloaded data.\n\n :param str url: url from which data should be donloaded\n :param destination: destination to which data should be downloaded\n \"\"\"\n\n # url = \"http://147.228.240.61/queetech/sample-data/jatra_06mm_jenjatra.zip\"\n logmsg = \"downloading from '\" + url + \"' to '\" + destination + \"'\"\n print(logmsg)\n logger.info(logmsg)\n tmp_filename = url[url.rfind(\"/\") + 1 :]\n # tmp_filename = \"tmp.zip\"\n # urllibr.urlretrieve(url, zip_file_name)\n from . import network\n\n network.download_file(url, destination, filename=tmp_filename)\n zip_file_name = os.path.join(destination, tmp_filename)\n base, ext = op.splitext(tmp_filename)\n # print(ext)\n # print(tmp_filename)\n if ext in (\".zip\"):\n unzip_recursive(zip_file_name)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nunzips one file and deletes it.", "response": "def unzip_one(local_file_name):\n \"\"\"Unzips one file and deletes it. Warning: function with side effects!\n\n :param str local_file_name: file name of zip file\n :return: list of archive members by name.\n \"\"\"\n local_file_name = op.expanduser(local_file_name)\n destination = op.dirname(local_file_name)\n datafile = zipfile.ZipFile(local_file_name)\n namelist = datafile.namelist()\n datafile.extractall(destination)\n datafile.close()\n remove(local_file_name)\n\n fullnamelist = []\n for fn in namelist:\n fullnamelist.append(op.join(destination, fn))\n return fullnamelist"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nunzip file with all recursive zip files inside and delete zip files after that.", "response": "def unzip_recursive(zip_file_name):\n \"\"\"Unzip file with all recursive zip files inside and delete zip files after that.\n\n :param zip_file_name: file name of zip file\n :return: list of archive members by name.\n \"\"\"\n logger.debug(\"unzipping \" + zip_file_name)\n fnlist = unzip_one(zip_file_name)\n for fn in fnlist:\n if zipfile.is_zipfile(fn):\n local_fnlist = unzip_recursive(fn)\n fnlist.extend(local_fnlist)\n return fnlist"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef upload_asciinema(filename):\n '''a wrapper around generation of an asciinema.api.Api to call the \n upload command given an already existing asciinema file. \n\n Parameters\n ==========\n filename: the asciinema file to upload, can be generated with \n function record_asciinema in record.py\n\n '''\n if os.path.exists(filename):\n\n import asciinema.config as aconfig\n from asciinema.api import Api\n\n # Load the API class\n\n cfg = aconfig.load()\n api = Api(cfg.api_url, os.environ.get(\"USER\"), cfg.install_id)\n\n # Perform the upload, return the url\n\n uploader = UploadCommand(api, filename)\n \n try:\n url, warn = uploader.api.upload_asciicast(filename)\n if warn:\n uploader.print_warning(warn)\n\n # Extract just the url, if provided (always is https)\n if url:\n match = re.search('https://.+', url)\n if match: \n url = match.group() \n return url\n\n except:\n bot.error('Problem with upload, skipping')\n\n else:\n bot.warning('Cannot find %s, skipping submission.' %filename)", "response": "a wrapper around generation of an API to call the UploadCommand to upload an already existing asciinema file and return the url of the uploaded file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating TLS configuration object.", "response": "def make_tls_config(app_config):\n \"\"\"Creates TLS configuration object.\"\"\"\n\n if not app_config['DOCKER_TLS']:\n return False\n\n cert_path = app_config['DOCKER_TLS_CERT_PATH']\n if cert_path:\n client_cert = '{0}:{1}'.format(\n os.path.join(cert_path, 'cert.pem'),\n os.path.join(cert_path, 'key.pem'))\n ca_cert = os.path.join(cert_path, 'ca.pem')\n else:\n client_cert = app_config['DOCKER_TLS_CLIENT_CERT']\n ca_cert = app_config['DOCKER_TLS_CA_CERT']\n\n client_cert = parse_client_cert_pair(client_cert)\n return TLSConfig(\n client_cert=client_cert,\n ca_cert=ca_cert,\n verify=app_config['DOCKER_TLS_VERIFY'],\n ssl_version=app_config['DOCKER_TLS_SSL_VERSION'],\n assert_hostname=app_config['DOCKER_TLS_ASSERT_HOSTNAME'])"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse_client_cert_pair(config_value):\n if not config_value:\n return\n client_cert = config_value.split(':')\n if len(client_cert) != 2:\n tips = ('client_cert should be formatted like '\n '\"/path/to/cert.pem:/path/to/key.pem\"')\n raise ValueError('{0!r} is invalid.\\n{1}'.format(config_value, tips))\n return tuple(client_cert)", "response": "Parses the client cert pair from config item."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ninitialize an application for using the Docker client.", "response": "def init_app(self, app):\n \"\"\"Initializes an application for using :class:`docker.Client`.\n\n :param app: an instance of :class:`~flask.Flask`.\n \"\"\"\n app.extensions = getattr(app, 'extensions', {})\n app.extensions['docker.client'] = None\n\n app.config.setdefault('DOCKER_URL', None)\n app.config.setdefault('DOCKER_VERSION', '1.16')\n app.config.setdefault('DOCKER_TIMEOUT', 30)\n app.config.setdefault('DOCKER_TLS', False)\n app.config.setdefault('DOCKER_TLS_VERIFY', True)\n app.config.setdefault('DOCKER_TLS_SSL_VERSION', None)\n app.config.setdefault('DOCKER_TLS_ASSERT_HOSTNAME', None)\n\n app.config.setdefault('DOCKER_TLS_CERT_PATH', None)\n app.config.setdefault('DOCKER_TLS_CLIENT_CERT', None)\n app.config.setdefault('DOCKER_TLS_CA_CERT', None)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the original docker. Client object.", "response": "def client(self):\n \"\"\"The original :class:`docker.Client` object. All docker operation\n calling will be forwarded here. ::\n\n docker.create_container('ubuntu')\n docker.client.create_container('ubuntu') # equivalent\n \"\"\"\n if not self.app.config['DOCKER_URL']:\n raise RuntimeError('\"DOCKER_URL\" must be specified')\n\n if not self.app.extensions['docker.client']:\n self.app.extensions['docker.client'] = Client(\n base_url=self.app.config['DOCKER_URL'],\n version=self.app.config['DOCKER_VERSION'],\n timeout=self.app.config['DOCKER_TIMEOUT'],\n tls=make_tls_config(self.app.config))\n return self.app.extensions['docker.client']"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nprocesses a feed and schedule the entries.", "response": "async def process_feed(self, url, send_mentions=True):\n \"\"\" process a feed \"\"\"\n\n self._feed_domains.add(utils.get_domain(url))\n\n if url in self._processed_feeds:\n LOGGER.debug(\"Skipping already processed feed %s\", url)\n return\n self._processed_feeds.add(url)\n\n LOGGER.debug(\"++WAIT: %s: get feed\", url)\n feed, previous, updated = await feeds.get_feed(self, url)\n LOGGER.debug(\"++DONE: %s: get feed\", url)\n\n if updated:\n LOGGER.info(\"Feed %s has been updated\", url)\n\n if not feed:\n return\n\n LOGGER.debug(\"--- starting process_feed %s %s\", url, send_mentions)\n\n pending = []\n\n try:\n for link in feed.links:\n href = link['href']\n if not href:\n continue\n\n # RFC5005 archive links\n if self.args.archive and link.get('rel') in ('prev-archive',\n 'next-archive',\n 'prev-page',\n 'next-page'):\n LOGGER.debug(\"Found archive link %s\", link)\n pending.append(\n (\"process feed \" + href, self.process_feed(href, send_mentions)))\n\n # WebSub notification\n if updated and link.get('rel') == 'hub' and not feed.is_archive:\n LOGGER.debug(\"Found WebSub hub %s\", link)\n pending.append(\n (\"update websub \" + href, feed.update_websub(self, href)))\n except (AttributeError, KeyError):\n LOGGER.debug(\"Feed %s has no links\", url)\n\n # Schedule the entries\n items = set(feed.entry_links)\n if previous:\n items |= set(previous.entry_links)\n for entry in items:\n pending.append((\"process entry \" + entry,\n self.process_entry(entry, send_mentions=send_mentions)))\n\n LOGGER.debug(\"--- finish process_feed %s %s\", url, send_mentions)\n\n if pending:\n LOGGER.debug(\"+++WAIT: process_feed(%s): %d subtasks\",\n url, len(pending))\n LOGGER.debug(\"%s\", [name for (name, _) in pending])\n await asyncio.wait([task for (_, task) in pending])\n LOGGER.debug(\"+++DONE: process_feed(%s): %d subtasks\",\n url, len(pending))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsend a webmention from an entry to a URL.", "response": "async def send_webmention(self, entry, url):\n \"\"\" send a webmention from an entry to a URL \"\"\"\n\n if (entry.url, url) in self._processed_mentions:\n LOGGER.debug(\n \"Skipping already processed mention %s -> %s\", entry.url, url)\n self._processed_mentions.add((entry.url, url))\n\n LOGGER.debug(\"++WAIT: webmentions.get_target %s\", url)\n target = await webmentions.get_target(self, url)\n LOGGER.debug(\"++DONE: webmentions.get_target %s\", url)\n\n if target:\n LOGGER.debug(\"++WAIT: Sending webmention %s -> %s\", entry.url, url)\n await target.send(self, entry)\n LOGGER.debug(\"++DONE: Sending webmention %s -> %s\", entry.url, url)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nupdate the locale message files for the given application.", "response": "def makemessages(application, locale):\n \"\"\"\n Updates the locale message files\n \"\"\"\n from django.core.management import call_command\n\n if not locale:\n locale = 'en'\n with work_in(application):\n call_command('makemessages', locale=(locale,))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef cms_check(migrate_cmd=False):\n from django.core.management import call_command\n try:\n import cms # NOQA # nopyflakes\n _create_db(migrate_cmd)\n call_command('cms', 'check')\n except ImportError:\n print('cms_check available only if django CMS is installed')", "response": "Runs the django CMS check command"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngenerating migrations for the specified application.", "response": "def makemigrations(application, merge=False, dry_run=False, empty=False, extra_applications=None):\n \"\"\"\n Generate migrations\n \"\"\"\n from django.core.management import call_command\n\n apps = [application]\n if extra_applications:\n if isinstance(extra_applications, text_type):\n apps += [extra_applications]\n elif isinstance(extra_applications, list):\n apps += extra_applications\n\n for app in apps:\n call_command('makemigrations', *(app,), merge=merge, dry_run=dry_run, empty=empty)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngenerating the list of authors for the current user", "response": "def generate_authors():\n \"\"\"\n Updates the authors list\n \"\"\"\n print('Generating AUTHORS')\n\n # Get our list of authors\n print('Collecting author names')\n r = subprocess.Popen(['git', 'log', '--use-mailmap', '--format=%aN'],\n stdout=subprocess.PIPE)\n seen_authors = []\n authors = []\n for authfile in ('AUTHORS', 'AUTHORS.rst'):\n if os.path.exists(authfile):\n break\n with open(authfile, 'r') as f:\n for line in f.readlines():\n if line.startswith(\"*\"):\n author = force_text(line).strip(\"* \\n\")\n if author.lower() not in seen_authors:\n seen_authors.append(author.lower())\n authors.append(author)\n for author in r.stdout.readlines():\n author = force_text(author).strip()\n if author.lower() not in seen_authors:\n seen_authors.append(author.lower())\n authors.append(author)\n\n # Sort our list of Authors by their case insensitive name\n authors = sorted(authors, key=lambda x: x.lower())\n\n # Write our authors to the AUTHORS file\n print('Authors (%s):\\n\\n\\n* %s' % (len(authors), '\\n* '.join(authors)))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ninitialing parse setup. Call at start of `self.parse()`.", "response": "def setup_parse(self, inputstring, document):\n \"\"\"Initial parse setup. Call at start of `self.parse()`.\"\"\"\n self.inputstring = inputstring\n self.document = document"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef check_version(expected_version):\n\n with open(os.path.join(root_folder, 'VERSION'), 'r') as version_file:\n version = version_file.read().strip()\n\n if version != expected_version:\n raise EnvironmentError(\"Version mismatch during release, expected={}, found={}\"\n .format(expected_version, version))", "response": "Check that the version of the package in setuptools matches the expected version."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nuploads a given component to pypi", "response": "def upload(dist_dir):\n \"\"\"Upload a given component to pypi\n The pypi username and password must either be specified in a ~/.pypirc file\n or in environment variables PYPI_USER and PYPI_PASS\n \"\"\"\n\n if 'PYPI_USER' in os.environ and 'PYPI_PASS' in os.environ:\n pypi_user = os.environ['PYPI_USER']\n pypi_pass = os.environ['PYPI_PASS']\n else:\n pypi_user = None\n pypi_pass = None\n print(\"No PYPI user information in environment\")\n\n packages = glob.glob(dist_dir)\n\n # Invoke upload this way since subprocess call of twine cli has cross platform issues\n twine_upload(packages, 'pypi', False, None, pypi_user, pypi_pass, None, None, '~/.pypirc', False, None, None, None)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef simple_cmd():\n parser = argparse.ArgumentParser(\n prog=\"Simple command-line function toolkit.\",\n description=\"\"\"Input function name and args and kwargs.\n python xxx.py main -a 1 2 3 -k a=1,b=2,c=3\"\"\",\n )\n parser.add_argument(\"-f\", \"--func_name\", default=\"main\")\n parser.add_argument(\"-a\", \"--args\", dest=\"args\", nargs=\"*\")\n parser.add_argument(\"-k\", \"--kwargs\", dest=\"kwargs\")\n parser.add_argument(\n \"-i\",\n \"-s\",\n \"--info\",\n \"--show\",\n \"--status\",\n dest=\"show\",\n action=\"store_true\",\n help=\"show the args, kwargs and function's source code.\",\n )\n params = parser.parse_args()\n func_name = params.func_name\n func = globals().get(func_name)\n if not (callable(func)):\n Config.utils_logger.warning(\"invalid func_name: %s\" % func_name)\n return\n args = params.args or []\n kwargs = params.kwargs or {}\n if kwargs:\n items = [re.split(\"[:=]\", i) for i in re.split(\"[,;]+\", kwargs)]\n kwargs = dict(items)\n if params.show:\n from inspect import getsource\n\n Config.utils_logger.info(\"args: %s; kwargs: %s\" % (args, kwargs))\n Config.utils_logger.info(getsource(func))\n return\n func(*args, **kwargs)", "response": "Simple command - line toolkit."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef print_mem(unit=\"MB\"):\n try:\n import psutil\n\n B = float(psutil.Process(os.getpid()).memory_info().vms)\n KB = B / 1024\n MB = KB / 1024\n GB = MB / 1024\n result = vars()[unit]\n print_info(\"memory usage: %.2f(%s)\" % (result, unit))\n return result\n except ImportError:\n print_info(\"pip install psutil first.\")", "response": "Show the proc - mem - cost with psutil use this only for lazinesssss."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ntranslate curl - string into dict of request.", "response": "def curlparse(string, encoding=\"utf-8\"):\n \"\"\"Translate curl-string into dict of request.\n :param string: standard curl-string, like `r'''curl ...'''`.\n :param encoding: encoding for post-data encoding.\n\n Basic Usage::\n\n >>> from torequests.utils import curlparse\n >>> curl_string = '''curl 'https://p.3.cn?skuIds=1&nonsense=1&nonce=0' -H 'Pragma: no-cache' -H 'DNT: 1' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: zh-CN,zh;q=0.9' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'Cache-Control: no-cache' -H 'Referer: https://p.3.cn?skuIds=1&nonsense=1&nonce=0' -H 'Cookie: ASPSESSIONIDSQRRSADB=MLHDPOPCAMBDGPFGBEEJKLAF' -H 'Connection: keep-alive' --compressed'''\n >>> request_args = curlparse(curl_string)\n >>> request_args\n {'url': 'https://p.3.cn?skuIds=1&nonsense=1&nonce=0', 'headers': {'Pragma': 'no-cache', 'Dnt': '1', 'Accept-Encoding': 'gzip, deflate', 'Accept-Language': 'zh-CN,zh;q=0.9', 'Upgrade-Insecure-Requests': '1', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8', 'Cache-Control': 'no-cache', 'Referer': 'https://p.3.cn?skuIds=1&nonsense=1&nonce=0', 'Cookie': 'ASPSESSIONIDSQRRSADB=MLHDPOPCAMBDGPFGBEEJKLAF', 'Connection': 'keep-alive'}, 'method': 'get'}\n >>> import requests\n >>> requests.request(**request_args)\n \n \"\"\"\n assert \"\\n\" not in string, 'curl-string should not contain \\\\n, try r\"...\".'\n if string.startswith(\"http\"):\n return {\"url\": string, \"method\": \"get\"}\n args, unknown = _Curl.parser.parse_known_args(shlex.split(string.strip()))\n requests_args = {}\n headers = {}\n requests_args[\"url\"] = args.url\n for header in args.header:\n key, value = header.split(\":\", 1)\n headers[key.title()] = value.strip()\n if args.user_agent:\n headers[\"User-Agent\"] = args.user_agent\n if headers:\n requests_args[\"headers\"] = headers\n if args.user:\n requests_args[\"auth\"] = tuple(u for u in args.user.split(\":\", 1) + [\"\"])[:2]\n # if args.proxy:\n # pass\n data = args.data or args.data_binary or args.form\n if data:\n if data.startswith(\"$\"):\n data = data[1:]\n args.method = \"post\"\n if \"application/x-www-form-urlencoded\" in headers.get(\"Content-Type\", \"\"):\n data = dict(\n [\n (i.split(\"=\")[0], unquote_plus(i.split(\"=\")[1]))\n for i in data.split(\"&\")\n ]\n )\n requests_args[\"data\"] = data\n else:\n data = data.encode(encoding)\n requests_args[\"data\"] = data\n requests_args[\"method\"] = args.method.lower()\n if args.connect_timeout:\n requests_args[\"timeout\"] = args.connect_timeout\n return requests_args"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef slice_into_pieces(seq, n):\n length = len(seq)\n if length % n == 0:\n size = length // n\n else:\n size = length // n + 1\n for it in slice_by_size(seq, size):\n yield it", "response": "Slice a sequence into n pieces return a generation of n pieces"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nslice a sequence into chunks with size.", "response": "def slice_by_size(seq, size):\n \"\"\"Slice a sequence into chunks, return as a generation of chunks with `size`.\"\"\"\n filling = null\n for it in zip(*(itertools_chain(seq, [filling] * size),) * size):\n if filling in it:\n it = tuple(i for i in it if i is not filling)\n if it:\n yield it"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntranslate timestamp into human - readable format.", "response": "def ttime(timestamp=None, tzone=None, fail=\"\", fmt=\"%Y-%m-%d %H:%M:%S\"):\n \"\"\"Translate timestamp into human-readable: %Y-%m-%d %H:%M:%S.\n\n :param timestamp: the timestamp float, or `time.time()` by default.\n :param tzone: time compensation, int(-time.timezone / 3600) by default,\n (can be set with Config.TIMEZONE).\n :param fail: while raising an exception, return it.\n :param fmt: %Y-%m-%d %H:%M:%S, %z not work.\n :rtype: str\n\n >>> ttime()\n 2018-03-15 01:24:35\n >>> ttime(1486572818.421858323)\n 2017-02-09 00:53:38\n \"\"\"\n tzone = Config.TIMEZONE if tzone is None else tzone\n fix_tz = tzone * 3600\n if timestamp is None:\n timestamp = time.time()\n else:\n timestamp = float(timestamp)\n if 1e12 <= timestamp < 1e13:\n # Compatible timestamp with 13-digit milliseconds\n timestamp = timestamp / 1000\n try:\n timestamp = time.time() if timestamp is None else timestamp\n return time.strftime(fmt, time.gmtime(timestamp + fix_tz))\n except:\n import traceback\n\n traceback.print_exc()\n return fail"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef ptime(timestr=None, tzone=None, fail=0, fmt=\"%Y-%m-%d %H:%M:%S\"):\n tzone = Config.TIMEZONE if tzone is None else tzone\n fix_tz = -(tzone * 3600 + time.timezone)\n #: str(timestr) for datetime.datetime object\n timestr = str(timestr or ttime())\n try:\n return int(time.mktime(time.strptime(timestr, fmt)) + fix_tz)\n except:\n return fail", "response": "Translate a string like 2018 - 03 - 15 01 - 27 - 56 into a timestamp."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsplit seconds into a list of time units divisors and units.", "response": "def split_seconds(seconds):\n \"\"\"Split seconds into [day, hour, minute, second, ms]\n\n `divisor: 1, 24, 60, 60, 1000`\n\n `units: day, hour, minute, second, ms`\n\n >>> split_seconds(6666666)\n [77, 3, 51, 6, 0]\n \"\"\"\n ms = seconds * 1000\n divisors = (1, 24, 60, 60, 1000)\n quotient, result = ms, []\n for divisor in divisors[::-1]:\n quotient, remainder = divmod(quotient, divisor)\n result.append(quotient) if divisor == 1 else result.append(remainder)\n return result[::-1]"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ntranslates seconds into human - readable form.", "response": "def timeago(seconds=0, accuracy=4, format=0, lang=\"en\"):\n \"\"\"Translate seconds into human-readable.\n\n :param seconds: seconds (float/int).\n :param accuracy: 4 by default (units[:accuracy]), determine the length of elements.\n :param format: index of [led, literal, dict].\n :param lang: en or cn.\n :param units: day, hour, minute, second, ms.\n\n >>> timeago(93245732.0032424, 5)\n '1079 days, 05:35:32,003'\n >>> timeago(93245732.0032424, 4, 1)\n '1079 days 5 hours 35 minutes 32 seconds'\n >>> timeago(-389, 4, 1)\n '-6 minutes 29 seconds 0 ms'\n \"\"\"\n assert format in [0, 1, 2], ValueError(\"format arg should be one of 0, 1, 2\")\n negative = \"-\" if seconds < 0 else \"\"\n seconds = abs(seconds)\n if lang == \"en\":\n units = (\"day\", \"hour\", \"minute\", \"second\", \"ms\")\n elif lang == \"cn\":\n units = (u\"\u5929\", u\"\u5c0f\u65f6\", u\"\u5206\u949f\", u\"\u79d2\", u\"\u6beb\u79d2\")\n times = split_seconds(seconds)\n if format == 2:\n return dict(zip(units, times))\n\n day, hour, minute, second, ms = times\n\n if format == 0:\n day_str = (\n \"%d %s%s, \" % (day, units[0], \"s\" if day > 1 and lang == \"en\" else \"\")\n if day\n else \"\"\n )\n mid_str = \":\".join((\"%02d\" % i for i in (hour, minute, second)))\n if accuracy > 4:\n mid_str += \",%03d\" % ms\n return negative + day_str + mid_str\n elif format == 1:\n # find longest valid fields index (non-zero in front)\n valid_index = 0\n for x, i in enumerate(times):\n if i > 0:\n valid_index = x\n break\n else:\n valid_index = x\n result_str = [\n \"%d %s%s\"\n % (num, unit, \"s\" if lang == \"en\" and num > 1 and unit != \"ms\" else \"\")\n for num, unit in zip(times, units)\n ][valid_index:][:accuracy]\n result_str = \" \".join(result_str)\n return negative + result_str"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreverse conversion for parse_qs", "response": "def unparse_qs(qs, sort=False, reverse=False):\n \"\"\"Reverse conversion for parse_qs\"\"\"\n result = []\n items = qs.items()\n if sort:\n items = sorted(items, key=lambda x: x[0], reverse=reverse)\n for keys, values in items:\n query_name = quote(keys)\n for value in values:\n result.append(query_name + \"=\" + quote(value))\n return \"&\".join(result)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef unparse_qsl(qsl, sort=False, reverse=False):\n result = []\n items = qsl\n if sort:\n items = sorted(items, key=lambda x: x[0], reverse=reverse)\n for keys, values in items:\n query_name = quote(keys)\n result.append(query_name + \"=\" + quote(values))\n return \"&\".join(result)", "response": "Reverse conversion for parse_qsl"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef kill_after(seconds, timeout=2):\n pid = os.getpid()\n kill = os.kill\n run_after_async(seconds, kill, pid, signal.SIGTERM)\n run_after_async(seconds + timeout, kill, pid, 9)", "response": "Kill self after seconds"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef try_import(module_name, names=None, default=ImportErrorModule, warn=True):\n try:\n module = importlib.import_module(module_name)\n except ImportError:\n if warn:\n if warn is True:\n Config.utils_logger.warning(\n \"Module `%s` not found. Install it to remove this warning\"\n % module_name\n )\n else:\n warn(module_name, names, default)\n module = (\n ImportErrorModule(module_name) if default is ImportErrorModule else default\n )\n if not names:\n return module\n if not isinstance(names, (tuple, set, list)):\n names = [names]\n result = []\n for name in names:\n if hasattr(module, name):\n result.append(module.__getattribute__(name))\n else:\n result.append(\n ImportErrorModule(\"%s.%s\" % (module_name, name))\n if default is ImportErrorModule\n else default\n )\n return result[0] if len(result) == 1 else result", "response": "Try import module_name except ImportError and return default."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef ensure_request(request):\n if isinstance(request, dict):\n result = request\n elif isinstance(request, (unicode, str)):\n request = request.strip()\n if request.startswith(\"http\"):\n result = {\"method\": \"get\", \"url\": request}\n elif request.startswith(\"curl \"):\n result = curlparse(request)\n else:\n raise ValueError(\"request should be dict or str.\")\n result[\"method\"] = result.setdefault(\"method\", \"get\").lower()\n return result", "response": "Used for requests. request with ** ensure_request ( request )\n "} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nset the dict key as key. title ; keys should be str. Always be used to headers.", "response": "def ensure_dict_key_title(dict_obj):\n \"\"\"Set the dict key as key.title(); keys should be str.\n Always be used to headers.\n\n >>> from torequests.utils import ensure_dict_key_title\n >>> ensure_dict_key_title({'hello-world':1, 'HELLOWORLD':2})\n {'Hello-World': 1, 'Helloworld': 2}\n \"\"\"\n if not all((isinstance(i, unicode) for i in dict_obj.keys())):\n return dict_obj\n return {key.title(): value for key, value in dict_obj.items()}"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngiving a seq of number return the median of the number.", "response": "def guess_interval(nums, accuracy=0):\n \"\"\"Given a seq of number, return the median, only calculate interval >= accuracy.\n\n ::\n\n from torequests.utils import guess_interval\n import random\n\n seq = [random.randint(1, 100) for i in range(20)]\n print(guess_interval(seq, 5))\n # sorted_seq: [2, 10, 12, 19, 19, 29, 30, 32, 38, 40, 41, 54, 62, 69, 75, 79, 82, 88, 97, 99]\n # diffs: [8, 7, 10, 6, 13, 8, 7, 6, 6, 9]\n # median: 8\n \"\"\"\n if not nums:\n return 0\n nums = sorted([int(i) for i in nums])\n if len(nums) == 1:\n return nums[0]\n diffs = [nums[i + 1] - nums[i] for i in range(len(nums) - 1)]\n diffs = [item for item in diffs if item >= accuracy]\n sorted_diff = sorted(diffs)\n result = sorted_diff[len(diffs) // 2]\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nruns a function in background will not block main thread s exit.", "response": "def bg(func):\n \"\"\"Run a function in background, will not block main thread's exit.(thread.daemon=True)\n ::\n\n from torequests.utils import bg, print_info\n import time\n\n def test1(n):\n time.sleep(n)\n print_info(n, 'done')\n\n @bg\n def test2(n):\n time.sleep(n)\n print_info(n, 'done')\n\n test3 = bg(test1)\n\n test2(1)\n test3(1)\n print_info('not be blocked')\n time.sleep(2)\n\n # [2018-06-12 23:46:19](L81): not be blocked\n # [2018-06-12 23:46:20](L81): 1 done\n # [2018-06-12 23:46:20](L81): 1 done\n \"\"\"\n\n @wraps(func)\n def wrapper(*args, **kwargs):\n t = Thread(target=func, args=args, kwargs=kwargs)\n t.daemon = True\n t.start()\n return t\n\n return wrapper"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef countdown(\n seconds=None,\n block=True,\n interval=1,\n daemon=True,\n tick_callback=None,\n finish_callback=None,\n):\n \"\"\"Run a countdown function to wait something, similar to threading.Timer,\n but will show the detail tick by tick_callback.\n ::\n\n from torequests.utils import countdown\n\n countdown(3)\n # 3 2 1 \n # countdown finished [3 seconds]: 2018-06-13 00:12:55 => 2018-06-13 00:12:58.\n countdown('2018-06-13 00:13:29')\n # 10 9 8 7 6 5 4 3 2 1 \n # countdown finished [10 seconds]: 2018-06-13 00:13:18 => 2018-06-13 00:13:28.\n\"\"\"\n\n def default_tick_callback(s, seconds, *args):\n flush_print(s, sep=\"\", end=\" \")\n\n def default_finish_callback(seconds, start_time):\n flush_print()\n\n def cd(seconds, interval):\n for s in range(seconds, 0, -interval):\n tick_callback(s, seconds, interval)\n time.sleep(interval)\n if callable(finish_callback):\n finish_callback(seconds, start_time)\n\n start_time = time.time()\n tick_callback = tick_callback or default_tick_callback\n finish_callback = (\n default_finish_callback if finish_callback is None else finish_callback\n )\n\n if unicode(seconds).isdigit():\n seconds = int(seconds)\n elif isinstance(seconds, (unicode, str)):\n seconds = int(ptime(seconds) - time.time())\n t = Thread(target=cd, args=(seconds, interval))\n t.daemon = daemon\n t.start()\n if block:\n t.join()", "response": "Run a countdown function to wait something."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef flush_print(*args, **kwargs):\n # PY2 raise SyntaxError for : def flush_print(*args, sep='', end=''):\n sep, end, flush = (\n kwargs.pop(\"sep\", \" \"),\n kwargs.pop(\"end\", \"\\n\"),\n kwargs.pop(\"flush\", 1),\n )\n string = sep.join((unicode(i) for i in args))\n sys.stdout.write(\"%s%s\" % (string, end))\n if flush:\n sys.stdout.flush()", "response": "Print a sequence of strings to stdout."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nusing tPool to request for curl string.", "response": "def curlrequests(curl_string, **kwargs):\n \"\"\"Use tPool to request for curl string.\n If kwargs contains the req which hasattr request method, like req=requests.\n\n :param curl_string: standard curl string.\n :type curl_string: str\n :param kwargs: valid kwargs for tPool.\n :type curl_string: dict\n\n Basic Usage::\n\n from torequests.utils import curlrequests\n\n\n r = curlrequests('''curl 'http://p.3.cn/' -H 'Connection: keep-alive' -H 'Cache-Control: max-age=0' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36' -H 'DNT: 1' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: zh-CN,zh;q=0.9,en;q=0.8' -H 'If-None-Match: \"55dd9090-264\"' -H 'If-Modified-Since: Wed, 26 Aug 2015 10:10:24 GMT' --compressed''', retry=1)\n print(r.text)\n \"\"\"\n req = kwargs.pop('req', tPool())\n kwargs.update(curlparse(curl_string))\n return req.request(**kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nregistering one object which can be matched by regex.", "response": "def register(self, patterns, obj=None, instances=None, **reg_kwargs):\n \"\"\"Register one object which can be matched/searched by regex.\n\n :param patterns: a list/tuple/set of regex-pattern.\n :param obj: return it while search/match success.\n :param instances: instance list will search/match the patterns.\n :param reg_kwargs: kwargs for re.compile.\n \"\"\"\n assert obj, \"bool(obj) should be True.\"\n patterns = patterns if isinstance(patterns, (list, tuple, set)) else [patterns]\n instances = instances or []\n instances = (\n instances if isinstance(instances, (list, tuple, set)) else [instances]\n )\n for pattern in patterns:\n pattern_compiled = re.compile(pattern, **reg_kwargs)\n self.container.append((pattern_compiled, obj, instances))\n if self.ensure_mapping:\n # check all instances to avoid one-to-many instances.\n self._check_instances()\n else:\n # no need to check all instances.\n for instance in instances:\n assert self.search(instance) == [obj] or self.match(instance) == [\n obj\n ], (\n \"instance %s should fit at least one pattern %s\"\n % (instance, pattern)\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning match or search result.", "response": "def find(self, string, default=None):\n \"\"\"Return match or search result.\n\n :rtype: list\"\"\"\n return self.match(string) or self.search(string) or default"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef search(self, string, default=None):\n default = default if default else []\n result = [item[1] for item in self.container if item[0].search(string)]\n if self.ensure_mapping:\n assert len(result) < 2, \"%s matches more than one pattern: %s\" % (\n string,\n result,\n )\n return result if result else default", "response": "Use re. search to find the result of search"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngiving suggestion from all instances.", "response": "def fuzzy(self, key, limit=5):\n \"\"\"Give suggestion from all instances.\"\"\"\n instances = [i[2] for i in self.container if i[2]]\n if not instances:\n return\n instances = sum(instances, [])\n from fuzzywuzzy import process\n\n maybe = process.extract(key, instances, limit=limit)\n return maybe"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef show_all(self, as_string=True):\n result = []\n for item in self.container:\n pattern = str(item[0])[10:] if PY3 else item[0].pattern\n instances = item[2] or []\n value = (\n '%s \"%s\"' % (item[1].__name__, (item[1].__doc__ or \"\"))\n if callable(item[1])\n else str(item[1])\n )\n value = \"%s %s\" % (type(item[1]), value)\n result.append(\" => \".join((pattern, \",\".join(instances), value)))\n return \"\\n\".join(result) if as_string else result", "response": "returns a string with all the available items"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncalling self. log_func and return expect_string.", "response": "def x(self):\n \"\"\"Call self.log_func(self) and return expect_string.\"\"\"\n self._log_after_del = False\n passed_string = self.string\n if self.log_func:\n self.log_func(self)\n else:\n print_info(\n \"Timer [%(passed)s]: %(name)s, start at %(start)s.\"\n % (\n dict(\n name=self.name, start=ttime(self.start_at), passed=passed_string\n )\n )\n )\n return passed_string"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef tick(self):\n string = self.passed\n if self.rounding:\n string = round(string)\n if self.readable:\n string = self.readable(string)\n return string", "response": "Return the time cost string as expect."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef default_callback(self, text):\n text = text.replace(\"\\r\\n\", \"\\n\")\n text = \"%s\\n\" % text\n flush_print(text, sep=\"\", end=\"\")\n return text", "response": "Default clean the \\\\ n in text."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef watch(self, limit=None, timeout=None):\n start_time = time.time()\n count = 0\n while not timeout or time.time() - start_time < timeout:\n new = self.read()\n if new != self.temp:\n count += 1\n self.callback(new)\n if count == limit:\n break\n self.temp = new\n time.sleep(self.interval)", "response": "Block method to watch the clipboard changing."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef watch_async(self, limit=None, timeout=None):\n return self.watch(limit=limit, timeout=timeout)", "response": "Non - block method to watch the clipboard changing."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfind one entry in the registry.", "response": "def find_one(cls, pattern, string, flags=0):\n \"\"\"JS-like match object. Use index number to get groups, if not match or no group, will return ''.\n\n Basic Usage::\n\n >>> from torequests.utils import find_one\n >>> string = \"abcd\"\n >>> find_one(\"a.*\", string)\n \n >>> find_one(\"a.*\", string)[0]\n 'abcd'\n >>> find_one(\"a.*\", string)[1]\n ''\n >>> find_one(\"a(.)\", string)[0]\n 'ab'\n >>> find_one(\"a(.)\", string)[1]\n 'b'\n >>> find_one(\"a(.)\", string)[2] or \"default\"\n 'default'\n >>> import re\n >>> item = find_one(\"a(B)(C)\", string, flags=re.I | re.S)\n >>> item\n \n >>> item[0]\n 'abc'\n >>> item[1]\n 'b'\n >>> item[2]\n 'c'\n >>> item[3]\n ''\n >>> # import re\n >>> # re.findone = find_one\n >>> register_re_findone()\n >>> re.findone('a(b)', 'abcd')[1] or 'default'\n 'b'\n\n \"\"\"\n item = re.search(pattern, string, flags=flags)\n return cls(item)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_issue(title, body, repo, token):\n '''create a Github issue, given a title, body, repo, and token.\n\n Parameters\n ==========\n title: the issue title\n body: the issue body\n repo: the full name of the repo\n token: the user's personal Github token\n\n '''\n owner, name = repo.split('/')\n url = 'https://api.github.com/repos/%s/%s/issues' % (owner, name)\n \n data = {'title': title, 'body': body }\n\n headers = { \"Authorization\": \"token %s\" % token,\n \"Accept\": \"application/vnd.github.symmetra-preview+json\" }\n\n response = requests.post(url, data=json.dumps(data), headers=headers)\n\n if response.status_code in [201, 202]:\n url = response.json()['html_url']\n bot.info(url)\n return url\n\n elif response.status_code == 404:\n bot.error('Cannot create issue. Does your token have scope repo?')\n sys.exit(1)\n\n else:\n bot.error('Cannot create issue %s' %title)\n bot.error(response.content)\n sys.exit(1)", "response": "create a Github issue given a title body repo and token"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\nasync def get_target(config, url):\n\n previous = config.cache.get(\n 'target', url, schema_version=SCHEMA_VERSION) if config.cache else None\n\n headers = previous.caching if previous else None\n\n request = await utils.retry_get(config, url, headers=headers)\n if not request or not request.success:\n return previous\n\n if request.cached:\n return previous\n\n current = Target(request)\n\n if config.cache:\n config.cache.set('target', url, current)\n\n return current", "response": "Given a URL get the webmention endpoint"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsends a webmention to this target from the specified entry.", "response": "async def send(self, config, entry):\n \"\"\" Send a webmention to this target from the specified entry \"\"\"\n if self.endpoint:\n LOGGER.debug(\"%s -> %s\", entry.url, self.url)\n try:\n await self.endpoint.send(config, entry.url, self.url)\n except Exception as err: # pylint:disable=broad-except\n LOGGER.warning(\"Ping %s: got %s: %s\",\n self.url, err.__class__.__name__, err)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a new Helper object based on the given name and quiet.", "response": "def get_helper(name=None, quiet=True, **kwargs):\n '''\n get the correct helper depending on the environment variable\n HELPME_CLIENT\n\n quiet: if True, suppress most output about the client (e.g. speak)\n\n '''\n # Second priority, from environment\n from helpme.defaults import HELPME_CLIENT\n\n # First priority, from command line\n if name is not None:\n HELPME_CLIENT = name\n\n # If no obvious credential provided, we can use HELPME_CLIENT\n if HELPME_CLIENT == 'github': from .github import Helper;\n elif HELPME_CLIENT == 'uservoice': from .uservoice import Helper\n elif HELPME_CLIENT == 'discourse': from .discourse import Helper\n else: from .github import Helper\n\n Helper.name = HELPME_CLIENT\n Helper.quiet = quiet\n\n # Initialize the database\n return Helper()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nmock the BaBLEInterface class with some controllers inside.", "response": "def mock_bable(monkeypatch):\n \"\"\" Mock the BaBLEInterface class with some controllers inside. \"\"\"\n mocked_bable = MockBaBLE()\n mocked_bable.set_controllers([\n Controller(0, '11:22:33:44:55:66', '#0'),\n Controller(1, '22:33:44:55:66:11', '#1', settings={'powered': True, 'low_energy': True}),\n Controller(2, '33:44:55:66:11:22', '#2', settings={'powered': True})\n ])\n\n monkeypatch.setattr(bable_interface, 'BaBLEInterface', lambda: mocked_bable)\n\n return mocked_bable"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nexecute the actual Stratum program.", "response": "def handle(self):\n \"\"\"\n Executes the actual Stratum program.\n \"\"\"\n self.output = PyStratumStyle(self.input, self.output)\n\n command = self.get_application().find('constants')\n ret = command.execute(self.input, self.output)\n if ret:\n return ret\n\n command = self.get_application().find('loader')\n ret = command.execute(self.input, self.output)\n if ret:\n return ret\n\n command = self.get_application().find('wrapper')\n ret = command.execute(self.input, self.output)\n\n self.output.writeln('')\n\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nloading a python module from its absolute filesystem path", "response": "def load_from_file(module_path):\n \"\"\"\n Load a python module from its absolute filesystem path\n\n Borrowed from django-cms\n \"\"\"\n from imp import load_module, PY_SOURCE\n\n imported = None\n if module_path:\n with open(module_path, 'r') as openfile:\n imported = load_module('mod', openfile, module_path, ('imported', 'r', PY_SOURCE))\n return imported"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _reset_django(settings):\n if settings._wrapped != empty:\n clear_url_caches()\n from django.apps import apps\n apps.clear_cache()\n settings._wrapped = empty\n clear_url_caches()", "response": "Hackish way to reset the django instance settings and AppConfig\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nensuring that the arguments in args_list are unicode and ensure that they are unique.", "response": "def ensure_unicoded_and_unique(args_list, application):\n \"\"\"\n Iterate over args_list, make it unicode if needed and ensure that there\n are no duplicates.\n Returns list of unicoded arguments in the same order.\n \"\"\"\n unicoded_args = []\n for argument in args_list:\n argument = (six.u(argument)\n if not isinstance(argument, six.text_type) else argument)\n if argument not in unicoded_args or argument == application:\n unicoded_args.append(argument)\n return unicoded_args"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef rmtree(path):\n if islink(path):\n unlink(path)\n elif os.path.isdir(path):\n for sub in os.listdir(path):\n sub = os.path.join(path, sub)\n if os.path.isfile(sub):\n if not os.access(sub, os.W_OK):\n # Is the error an access error?\n os.chmod(sub, stat.S_IWUSR)\n os.unlink(sub)\n else:\n rmtree(sub)\n os.rmdir(path)\n else:\n raise RuntimeError(\"rmtree called on non-link/folder\")", "response": "This function removes the contents of the specified folder."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning True if the path is a link.", "response": "def islink(path):\n \"\"\"os.path.islink() but return True for junction points on Windows.\n \"\"\"\n if platform.system() == \"Windows\":\n if sys.version_info[:2] < (3, 5):\n try:\n # pylint: disable=undefined-variable\n attrs = ctypes.windll.kernel32.GetFileAttributesW(unicode(path)) # noqa\n assert attrs != -1\n return bool(attrs & FILE_ATTRIBUTE_REPARSE_POINT)\n except (AttributeError, AssertionError):\n return False\n try:\n st = os.lstat(path) # pylint: disable=invalid-name\n except (OSError, AttributeError):\n return False\n # pylint: disable=no-member\n return stat.S_ISLNK(st.st_mode) or st.st_file_attributes & stat.FILE_ATTRIBUTE_REPARSE_POINT\n return os.path.islink(path)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef write_two_phases(filename, data, io):\n write_flag = True\n if os.path.exists(filename):\n with open(filename, 'r') as file:\n old_data = file.read()\n if data == old_data:\n write_flag = False\n\n if write_flag:\n tmp_filename = filename + '.tmp'\n with open(tmp_filename, 'w+') as file:\n file.write(data)\n os.replace(tmp_filename, filename)\n io.text('Wrote: {0}'.format(filename))\n else:\n io.text('File {0} is up to date'.format(filename))", "response": "Writes a file in two phases to the filesystem."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget an entry from the cache and return it", "response": "async def get_entry(config, url):\n \"\"\" Given an entry URL, return the entry\n\n Arguments:\n\n config -- the configuration\n url -- the URL of the entry\n\n Returns: 3-tuple of (current, previous, updated) \"\"\"\n\n previous = config.cache.get(\n 'entry', url,\n schema_version=SCHEMA_VERSION) if config.cache else None\n\n headers = previous.caching if previous else None\n\n request = await utils.retry_get(config, url, headers=headers)\n if not request or not request.success:\n LOGGER.error(\"Could not get entry %s: %d\", url,\n request.status if request else -1)\n return None, previous, False\n\n # cache hit\n if request.cached:\n return previous, previous, False\n\n current = Entry(request)\n\n # Content updated\n if config.cache:\n config.cache.set('entry', url, current)\n\n return current, previous, (not previous\n or previous.digest != current.digest\n or previous.status != current.status)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _check_rel(attrs, rel_whitelist, rel_blacklist):\n\n rels = attrs.get('rel', [None])\n\n if rel_blacklist:\n # Never return True for a link whose rel appears in the blacklist\n for rel in rels:\n if rel in rel_blacklist:\n return False\n\n if rel_whitelist:\n # If there is a whitelist for rels, only return true for a rel that\n # appears in it\n for rel in rels:\n if rel in rel_whitelist:\n return True\n # If there is a whitelist and we don't match, then reject\n return False\n\n return True", "response": "Check a link s relations against the whitelist or blacklist."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nchecks that a link is not on the same domain as the source URL.", "response": "def _domain_differs(self, href):\n \"\"\" Check that a link is not on the same domain as the source URL \"\"\"\n target = utils.get_domain(href)\n if not target:\n return False\n\n origin = utils.get_domain(self.url)\n return target != origin"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngiving an Entry object return all of the outgoing links.", "response": "def get_targets(self, config):\n \"\"\" Given an Entry object, return all of the outgoing links. \"\"\"\n\n return {urllib.parse.urljoin(self.url, attrs['href'])\n for attrs in self._targets\n if self._check_rel(attrs, config.rel_whitelist, config.rel_blacklist)\n and self._domain_differs(attrs['href'])}"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef run_command(self, config_file, sources):\n config = configparser.ConfigParser()\n config.read(config_file)\n\n rdbms = config.get('database', 'rdbms').lower()\n\n loader = self.create_routine_loader(rdbms)\n status = loader.main(config_file, sources)\n\n return status", "response": "Run the command line."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create_routine_loader(self, rdbms):\n # Note: We load modules and classes dynamically such that on the end user's system only the required modules\n # and other dependencies for the targeted RDBMS must be installed (and required modules and other\n # dependencies for the other RDBMSs are not required).\n\n if rdbms == 'mysql':\n module = locate('pystratum_mysql.MySqlRoutineLoader')\n return module.MySqlRoutineLoader(self.output)\n\n if rdbms == 'mssql':\n module = locate('pystratum_mssql.MsSqlRoutineLoader')\n return module.MsSqlRoutineLoader(self.output)\n\n if rdbms == 'pgsql':\n module = locate('pystratum_pgsql.PgSqlRoutineLoader')\n return module.PgSqlRoutineLoader(self.output)\n\n raise Exception(\"Unknown RDBMS '{0!s}'.\".format(rdbms))", "response": "Returns a Routine Loader object for loading stored routines into a RDBMS instance."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\nasync def create_vm(self, *, preset_name, image, flavor, security_groups=None,\n userdata=None, key_name=None, availability_zone=None,\n subnet=None):\n \"\"\"\n Dummy create_vm func.\n \"\"\"\n info = {\n 'id': next(self._id_it),\n 'name': preset_name,\n 'ip': ['127.0.0.1'],\n 'created': 0,\n 'state': VmState.RUNNING,\n 'flavor': flavor,\n 'image': image,\n 'metadata': {'test-meta': 'abctest'},\n 'timed_shutdown_at': 1522753481,\n 'tags': ['a-tag', 'b-tag', 'c-tag']\n }\n logging.debug('Prepare vm: %s', info)\n vm = Vm(self, **info)\n self._vms[vm.id] = vm\n logging.debug('Create: %s', vm)\n return None", "response": "Create a new VM."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\nasync def list_vms(self, preset_name):\n return list(vm for vm in self._vms.values() if vm.name == preset_name)", "response": "List all VMs in the specified preset."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef suggest_filename(file_path, exists=None):\n import os.path\n import re\n if not isinstance(exists, bool):\n exists = os.path.exists(file_path)\n\n if exists:\n file_path, file_extension = os.path.splitext(file_path)\n # print(file_path)\n m = re.search(r\"_\\d+$\", file_path)\n if m is None:\n # cislo = 2\n new_cislo_str = \"_2\"\n else:\n cislostr = (m.group())\n cislo = int(cislostr[1:]) + 1\n # it is normal number\n file_path = file_path[:-len(cislostr)]\n new_cislo_str = \"_\" + str(cislo)\n\n file_path = file_path + new_cislo_str + file_extension # .zfill(2)\n # trorcha rekurze\n file_path = suggest_filename(file_path)\n\n return file_path", "response": "Try if exist path and append number to its end."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nread object from file.", "response": "def obj_from_file(filename='annotation.yaml', filetype='auto'):\n ''' Read object from file '''\n\n if filetype == 'auto':\n _, ext = os.path.splitext(filename)\n filetype = ext[1:]\n\n if filetype in ('yaml', 'yml'):\n from ruamel.yaml import YAML\n yaml = YAML(typ=\"unsafe\")\n with open(filename, encoding=\"utf-8\") as f:\n obj = yaml.load(f)\n if obj is None:\n obj = {}\n # import yaml\n # with open(filename, encoding=\"utf-8\") as f:\n # intext = f.read()\n # obj = yaml.load(intext)\n elif filetype in ('pickle', 'pkl', 'pklz', 'picklezip'):\n fcontent = read_pkl_and_pklz(filename)\n # import pickle\n if sys.version_info[0] < 3:\n import cPickle as pickle\n else:\n import _pickle as pickle\n # import sPickle as pickle\n if sys.version_info.major == 2:\n obj = pickle.loads(fcontent)\n else:\n obj = pickle.loads(fcontent, encoding=\"latin1\")\n else:\n logger.error('Unknown filetype ' + filetype)\n return obj"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nwrites an object to a file.", "response": "def obj_to_file(obj, filename, filetype='auto', ndarray_to_list=False, squeeze=True):\n '''Writes annotation in file.\n\n :param filetype:\n auto\n yaml\n pkl, pickle\n pklz, picklezip\n :param ndarray_to_list: convert ndarrays in obj to lists\n :param squeeze: squeeze ndarray\n\n '''\n # import json\n # with open(filename, mode='w') as f:\n # json.dump(annotation,f)\n if ndarray_to_list:\n obj = ndarray_to_list_in_structure(obj, squeeze=squeeze)\n\n # write to yaml\n d = os.path.dirname(os.path.abspath(filename))\n if not os.path.exists(d):\n os.makedirs(d)\n\n if filetype == 'auto':\n _, ext = os.path.splitext(filename)\n filetype = ext[1:]\n\n if filetype in ('yaml', 'yml'):\n # import yaml\n from ruamel.yaml import YAML\n yaml = YAML(typ=\"unsafe\")\n with open(filename, 'wt', encoding=\"utf-8\") as f:\n yaml.dump(obj, f)\n # if sys.version_info.major == 2:\n # with open(filename, 'wb') as f:\n # yaml.dump(obj, f, encoding=\"utf-8\")\n # else:\n # with open(filename, \"w\", encoding=\"utf-8\") as f:\n # yaml.dump(obj, f)\n elif filetype in ('pickle', 'pkl'):\n f = open(filename, 'wb')\n logger.info(\"filename \" + filename)\n # if sys.version_info[0] < 3: import cPickle as pickle\n # else: import _pickle as pickle\n import pickle\n pickle.dump(obj, f, -1)\n f.close\n elif filetype in ('streamingpicklezip', 'spklz'):\n # this is not working :-(\n import gzip\n import sPickle as pickle\n f = gzip.open(filename, 'wb', compresslevel=1)\n # f = open(filename, 'wb')\n pickle.s_dump(obj, f)\n f.close\n elif filetype in ('picklezip', 'pklz'):\n import gzip\n if sys.version_info[0] < 3: import cPickle as pickle\n else: import _pickle as pickle\n f = gzip.open(filename, 'wb', compresslevel=1)\n # f = open(filename, 'wb')\n pickle.dump(obj, f)\n f.close\n elif filetype in('mat'):\n\n import scipy.io as sio\n sio.savemat(filename, obj)\n else:\n logger.error('Unknown filetype ' + filetype)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef resize_to_shape(data, shape, zoom=None, mode='nearest', order=0):\n # @TODO remove old code in except part\n\n try:\n # rint 'pred vyjimkou'\n # aise Exception ('test without skimage')\n # rint 'za vyjimkou'\n import skimage\n import skimage.transform\n# Now we need reshape seeds and segmentation to original size\n\n segm_orig_scale = skimage.transform.resize(\n data, shape, order=0,\n preserve_range=True,\n mode=\"constant\",\n )\n\n segmentation = segm_orig_scale\n logger.debug('resize to orig with skimage')\n except:\n import scipy\n import scipy.ndimage\n dtype = data.dtype\n if zoom is None:\n zoom = shape / np.asarray(data.shape).astype(np.double)\n\n segm_orig_scale = scipy.ndimage.zoom(\n data,\n 1.0 / zoom,\n mode=mode,\n order=order\n ).astype(dtype)\n logger.debug('resize to orig with scipy.ndimage')\n\n# @TODO odstranit hack pro o\u0159\u00edznut\u00ed na stejnou velikost\n# v podstat\u011b je to vy\u0159e\u0161eno, ale nechalo by se to d\u011blat elegantn\u011bji v zoom\n# tam je bohu\u017eel patrn\u011b bug\n # rint 'd3d ', self.data3d.shape\n # rint 's orig scale shape ', segm_orig_scale.shape\n shp = [\n np.min([segm_orig_scale.shape[0], shape[0]]),\n np.min([segm_orig_scale.shape[1], shape[1]]),\n np.min([segm_orig_scale.shape[2], shape[2]]),\n ]\n # elf.data3d = self.data3d[0:shp[0], 0:shp[1], 0:shp[2]]\n # mport ipdb; ipdb.set_trace() # BREAKPOINT\n\n segmentation = np.zeros(shape, dtype=dtype)\n segmentation[\n 0:shp[0],\n 0:shp[1],\n 0:shp[2]] = segm_orig_scale[0:shp[0], 0:shp[1], 0:shp[2]]\n\n del segm_orig_scale\n return segmentation", "response": "Function resize input data to specific shape."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfunctioning can resize data3d or segmentation to specified voxelsize mm", "response": "def resize_to_mm(data3d, voxelsize_mm, new_voxelsize_mm, mode='nearest'):\n \"\"\"\n Function can resize data3d or segmentation to specifed voxelsize_mm\n :new_voxelsize_mm: requested voxelsize. List of 3 numbers, also\n can be a string 'orig', 'orgi*2' and 'orgi*4'.\n\n :voxelsize_mm: size of voxel\n :mode: default is 'nearest'\n \"\"\"\n import scipy\n import scipy.ndimage\n\n if np.all(list(new_voxelsize_mm) == 'orig'):\n new_voxelsize_mm = np.array(voxelsize_mm)\n elif np.all(list(new_voxelsize_mm) == 'orig*2'):\n new_voxelsize_mm = np.array(voxelsize_mm) * 2\n elif np.all(list(new_voxelsize_mm) == 'orig*4'):\n new_voxelsize_mm = np.array(voxelsize_mm) * 4\n # vx_size = np.array(metadata['voxelsize_mm']) * 4\n\n zoom = voxelsize_mm / (1.0 * np.array(new_voxelsize_mm))\n data3d_res = scipy.ndimage.zoom(\n data3d,\n zoom,\n mode=mode,\n order=1\n ).astype(data3d.dtype)\n return data3d_res"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef suits_with_dtype(mn, mx, dtype):\n type_info = np.iinfo(dtype)\n if mx <= type_info.max and mn >= type_info.min:\n return True\n else:\n return False", "response": "Check whether the given range of values can be stored into defined data type."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef use_economic_dtype(data3d, slope=1, inter=0, dtype=None):\n\n if dtype is None:\n dtype = data3d.dtype\n if issubclass(dtype.type, np.integer):\n mn = data3d.min() * slope + inter\n mx = data3d.max() * slope + inter\n if suits_with_dtype(mn, mx, dtype=np.uint8):\n dtype = np.uint8\n elif suits_with_dtype(mn, mx, dtype=np.int8):\n dtype = np.int8\n elif suits_with_dtype(mn, mx, dtype=np.uint16):\n dtype = np.uint16\n elif suits_with_dtype(mn, mx, dtype=np.int16):\n dtype = np.int16\n elif suits_with_dtype(mn, mx, dtype=np.uint32):\n dtype = np.uint32\n elif suits_with_dtype(mn, mx, dtype=np.int32):\n dtype = np.int32\n\n # new_data3d = ((np.float(slope) * data3d) + np.float(inter)).astype(dtype)\n if slope == 1 and inter == 0:\n # this can prevent out of memmory\n new_data3d = data3d.astype(dtype)\n else:\n new_data3d = ((slope * data3d) + inter).astype(dtype)\n return new_data3d", "response": "Use more economic integer - like dtype if it is possible."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_sitk_image_from_ndarray(data3d):\n\n import SimpleITK as sitk\n rescale_intercept = None\n if sitk.Version.MajorVersion() > 0:\n if data3d.dtype == np.int8:\n rescale_intercept = -2**7\n data3d = (data3d - rescale_intercept).astype(np.uint8)\n elif data3d.dtype == np.int16:\n # simpleitk is not able to store this. It uses only 11 bites\n # rescale_intercept = -2**15\n rescale_intercept = -2**10\n data3d = (data3d - rescale_intercept).astype(np.uint16)\n elif data3d.dtype == np.int32:\n rescale_intercept = -2**31\n data3d = (data3d - rescale_intercept).astype(np.uint16)\n\n dim = sitk.GetImageFromArray(data3d)\n if sitk.Version.MajorVersion() > 0:\n if rescale_intercept is not None:\n # rescale slope (0028|1053), rescale intercept (0028|1052)\n dim.SetMetaData(\"0028|1052\", str(rescale_intercept))\n dim.SetMetaData(\"0028|1053\", \"1\")\n\n return dim", "response": "This function returns the SimpleItk Image object from the data array."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_slope_and_intercept_from_pdcm(dcmdata):\n\n if hasattr(dcmdata, \"RescaleSlope\") and hasattr(dcmdata, \"RescaleIntercept\"):\n rescale_slope = dcmdata.RescaleSlope\n rescale_intercept = dcmdata.RescaleIntercept\n slope, inter = get_slope_and_intercept_from_strings(rescale_slope, rescale_intercept)\n else:\n slope = 1\n inter = 0\n\n return slope, inter", "response": "Get slope and intercept from pydicom file object."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsplit an integer into evenly distributed list of size n", "response": "def split_evenly(n, chunks):\n \"\"\"Split an integer into evenly distributed list\n\n >>> split_evenly(7, 3)\n [3, 2, 2]\n\n >>> split_evenly(12, 3)\n [4, 4, 4]\n\n >>> split_evenly(35, 10)\n [4, 4, 4, 4, 4, 3, 3, 3, 3, 3]\n\n >>> split_evenly(1, 2)\n Traceback (most recent call last):\n ...\n ChunkingError: Number of chunks is greater than number\n\n \"\"\"\n if n < chunks:\n raise ChunkingError(\"Number of chunks is greater than number\")\n if n % chunks == 0:\n # Either we can evenly split or only 1 chunk left\n return [n / chunks] * chunks\n # otherwise the current chunk should be a bit larger\n max_size = n / chunks + 1\n return [max_size] + split_evenly(n - max_size, chunks - 1)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef check_env(self, envar, value):\n '''ensure that variable envar is set to some value, \n otherwise exit on error.\n \n Parameters\n ==========\n envar: the environment variable name\n value: the setting that shouldn't be None\n '''\n if value is None:\n bot.error('You must export %s to use Discourse' % envar)\n print('https://vsoch.github.io/helpme/helper-discourse')\n sys.exit(1)", "response": "ensure that variable envar is set to some value"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngenerate a keypair on behalf of the user", "response": "def _generate_keys(self):\n '''the discourse API requires the interactions to be signed, so we \n generate a keypair on behalf of the user\n '''\n from helpme.defaults import HELPME_CLIENT_SECRETS\n keypair_dir = os.path.join(os.path.dirname(HELPME_CLIENT_SECRETS),\n 'discourse')\n\n # Have we generated a keypair file before?\n self.keypair_file = os.path.join(keypair_dir, 'private.pem')\n\n # We likely won't have generated it on first use!\n if not hasattr(self, 'key'):\n self.key = generate_keypair(self.keypair_file) \n\n # If we generated the keypair file, we will have already loaded the key\n if not hasattr(self, 'public_key'):\n self.public_key = load_keypair(self.keypair_file)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _submit(self):\n '''submit the question to the board. When we get here we should have \n (under self.data)\n \n {'record_environment': [('DISPLAY', ':0')],\n 'user_prompt_board': 'http://127.0.0.1',\n 'user_prompt_issue': 'I want to know why dinosaurs are so great!',\n 'user_prompt_title': 'Why are dinosaurs so great?'}\n\n self.token should be propogated with the personal access token\n ''' \n body = self.data['user_prompt_issue']\n title = self.data['user_prompt_title']\n board = self.data['user_prompt_board']\n username = self.data['user_prompt_username']\n category = self.data['user_prompt_category']\n\n # Step 1: Token\n if self.token == None:\n self.token = self.request_token(board)\n self._get_and_update_setting('HELPME_DISCOURSE_TOKEN', self.token)\n\n # Step 1: Environment\n\n envars = self.data.get('record_environment') \n body = body + envars_to_markdown(envars)\n\n # Step 2: Asciinema\n\n asciinema = self.data.get('record_asciinema')\n if asciinema not in [None, '']:\n url = upload_asciinema(asciinema)\n\n # If the upload is successful, add a link to it.\n\n if url is not None:\n body += \"\\n[View Asciinema Recording](%s)\" % url\n \n # Add other metadata about client\n\n body += \"\\n\\ngenerated by [HelpMe](https://vsoch.github.io/helpme/)\"\n body += \"\\nHelpMe Discourse Id: %s\" %(self.run_id)\n\n # Submit the issue\n\n post = self.create_post(title, body, board, category, username)\n return post", "response": "Submit the question to the board."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nretrieves an object which id_field matches id_val.", "response": "def get(self, cls, id_field, id_val):\n \"\"\"\n Retrieve an object which `id_field` matches `id_val`. If it exists in\n the cache, it will be fetched from Redis. If not, it will be fetched\n via the `fetch` method and cached in Redis (unless the cache flag got\n invalidated in the meantime).\n \"\"\"\n cache_key, flag_key = self.get_keys(cls, id_field, id_val)\n\n result = self.get_cached_or_set_flag(keys=(cache_key, flag_key))\n\n # in Lua, arrays cannot hold nil values, so e.g. if [1, nil] is returned,\n # we'll only get [1] here. That's why we need to append None ourselves.\n if len(result) == 1:\n result.append(None)\n\n previous_flag, cached_data = result\n\n # if cached data was found, deserialize and return it\n if cached_data is not None:\n deserialized = self.deserialize(cls, cached_data)\n\n # verify that the cached object matches our expectations\n # if not, return from the persistant storage instead.\n if self.verify(cls, id_field, id_val, deserialized):\n return deserialized\n else:\n # invalidate the cache if it didn't pass verification\n self.invalidate(cls, id_field, id_val)\n\n obj = self.fetch(cls, id_field, id_val)\n\n # If the flag wasn't previously set, then we set it and we're responsible\n # for putting the item in the cache. Do this unless the cache got\n # invalidated and the flag was removed.\n if not previous_flag:\n obj_serialized = self.serialize(obj)\n self.cache(keys=(cache_key, flag_key), args=(obj_serialized,))\n\n return obj"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nverifies that the object we retrieved from cache matches the requested ID.", "response": "def verify(self, cls, id_field, id_val, obj_from_cache):\n \"\"\"\n Verify that the object we retrieved from cache matches the requested\n `id_field`/`id_val`.\n \"\"\"\n return getattr(obj_from_cache, id_field) == id_val"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ninvalidating the cache for a given Mongo object.", "response": "def invalidate(self, cls, id_field, id_val):\n \"\"\"\n Invalidate the cache for a given Mongo object by deleting the cached\n data and the cache flag.\n \"\"\"\n cache_key, flag_key = self.get_keys(cls, id_field, id_val)\n\n pipeline = self.redis.pipeline()\n pipeline.delete(cache_key)\n pipeline.delete(flag_key)\n pipeline.execute()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nupdate the state of the NukiLock object.", "response": "def update(self, aggressive=False):\n \"\"\"\n Update the state of the NukiLock\n :param aggressive: Whether to aggressively poll the Bridge. If set to\n True, this will actively query the Lock, thus using more battery.\n :type aggressive: bool\n \"\"\"\n if aggressive:\n data = self._bridge.lock_state(self.nuki_id)\n if not data['success']:\n logger.warning(\n 'Failed to update the state of lock {}'.format(self.nuki_id)\n )\n self._json.update({k: v for k, v in data.items() if k != 'success'})\n else:\n data = [l for l in self._bridge.locks if l.nuki_id == self.nuki_id]\n assert data, (\n 'Failed to update data for lock. '\n 'Nuki ID {} volatized.'.format(self.nuki_id))\n self._json.update(data[0]._json)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a tuple containing the item properties.", "response": "def get_dict_properties(item, fields, mixed_case_fields=[], formatters={}):\n \"\"\"Return a tuple containing the item properties.\n :param item: a single dict resource\n :param fields: tuple of strings with the desired field names\n :param mixed_case_fields: tuple of field names to preserve case\n :param formatters: dictionary mapping field names to callables\n to format the values\n \"\"\"\n row = []\n\n for field in fields:\n if field in mixed_case_fields:\n field_name = field.replace(' ', '_')\n else:\n field_name = field.lower().replace(' ', '_')\n data = item[field_name] if field_name in item else ''\n if field in formatters:\n row.append(formatters[field](data))\n else:\n row.append(data)\n return tuple(row)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef log_method(log, level=logging.DEBUG):\n\n def decorator(func):\n func_name = func.__name__\n\n @six.wraps(func)\n def wrapper(self, *args, **kwargs):\n if log.isEnabledFor(level):\n pretty_args = []\n if args:\n pretty_args.extend(str(a) for a in args)\n if kwargs:\n pretty_args.extend(\n \"%s=%s\" % (k, v) for k, v in six.iteritems(kwargs))\n log.log(level, \"%s(%s)\", func_name, \", \".join(pretty_args))\n return func(self, *args, **kwargs)\n\n return wrapper\n\n return decorator", "response": "A decorator that logs a method and its arguments when entered."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef check_local() -> None:\n to_check = ['./replay', './replay/toDo', './replay/archive']\n\n for i in to_check:\n if not os.path.exists(i):\n os.makedirs(i)", "response": "Checks that the required directories exist."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create_file(project: str, environment: str, feature: str, state: str) -> None:\n check_local()\n save_path = './replay/toDo/'\n filename = '{0}.txt'.format(str(uuid.uuid1()))\n complete_name = os.path.join(save_path, filename)\n\n with open(complete_name, 'w') as filename:\n filename.write('rc update-ld-api -p {0} -e {1} -f {2} -s {3}'.format(\n project,\n environment,\n feature,\n state\n ))", "response": "Create file to replay."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nexecuting all commands in replay and move them to the replay archive directory.", "response": "def execute_replay() -> None:\n \"\"\"\n Execute all commands.\n\n For every command that is found in replay/toDo, execute each of them\n and move the file to the replay/archive directory.\n \"\"\"\n files = glob.glob('./replay/toDo/*')\n sorted_files = sorted(files, key=os.path.getctime)\n\n if not sorted_files: # list is not empty\n LOG.debug('Found %s, beginning execution.', sorted_files)\n for command_file in sorted_files:\n with open(command_file, 'r') as command:\n cmd = command.read()\n LOG.debug('executing command: %s', cmd)\n resp = run([cmd, '-v', 'DEBUG'], shell=True, check=True)\n LOG.debug(resp)\n LOG.debug('moving %s to archive', command.name)\n move_command = 'mv {0} ./replay/archive/'.format(command.name)\n run(move_command, shell=True, check=True)\n LOG.info('LaunchDarkly is now up to date.')\n else:\n LOG.warning('No files found, nothing to replay.')"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nasking the user for a prompt and only return when one of the requested options is provided.", "response": "def choice_prompt(prompt, choices=None, choice=None):\n '''Ask the user for a prompt, and only return when one of the requested\n options is provided.\n\n Parameters\n ==========\n prompt: the prompt to ask the user\n choices: a list of choices that are valid, defaults to [Y/N/y/n]\n \n '''\n if not choices:\n choices = [\"y\", \"n\", \"Y\", \"N\"]\n\n print(prompt)\n get_input = getattr(__builtins__, 'raw_input', input)\n pretty_choices = '/'.join(choices)\n message = 'Please enter your choice [%s] : ' %(pretty_choices)\n while choice not in choices:\n choice = get_input(message).strip()\n\n # If the option isn't valid, this is shown next\n message = \"Please enter a valid option in [%s]\" %(pretty_choices) \n return choice"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nask the user for a text entry that matches a regular expression.", "response": "def regexp_prompt(prompt, regexp='.', answer=''):\n '''Ask the user for a text entry that matches a regular expression\n\n Parameters\n ==========\n prompt: the prompt to ask the user\n regexp: the regular expression to match. defaults to anything.\n \n '''\n get_input = getattr(__builtins__, 'raw_input', input)\n while not re.search(regexp, answer):\n answer = get_input(prompt + ': ').strip()\n # If the option isn't valid, this is shown next\n message = \"Your entry must match the regular expression %s\" %regexp \n\n return answer"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef which(software, strip_newline=True):\n '''get_install will return the path to where an executable is installed.\n '''\n if software is None:\n software = \"singularity\"\n cmd = ['which', software ]\n try:\n result = run_command(cmd)\n if strip_newline is True:\n result['message'] = result['message'].strip('\\n')\n return result\n\n except: # FileNotFoundError\n return None", "response": "get_install will return the path to where an executable is installed."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef guess_encoding(request):\n ctype = request.headers.get('content-type')\n if not ctype:\n # we don't have a content-type, somehow, so...\n LOGGER.warning(\"%s: no content-type; headers are %s\",\n request.url, request.headers)\n return 'utf-8'\n\n # explicit declaration\n match = re.search(r'charset=([^ ;]*)(;| |$)', ctype)\n if match:\n return match[1]\n\n # html default\n if ctype.startswith('text/html'):\n return 'iso-8859-1'\n\n # everything else's default\n return 'utf-8'", "response": "Try to guess the encoding of a request without going through the slow chardet process"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _make_headers(config, kwargs):\n\n headers = kwargs.get('headers')\n headers = headers.copy() if headers is not None else {}\n headers['User-Agent'] = config.args.user_agent\n\n kwargs = kwargs.copy()\n kwargs['headers'] = headers\n return kwargs", "response": "Replace the kwargs with one where the headers include our user - agent"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\nasync def retry_get(config, url, *args, **kwargs):\n return await _retry_do(config.session.get, url, *args,\n **_make_headers(config, kwargs))", "response": "Aiohttp wrapper for GET"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\nasync def retry_post(config, url, *args, **kwargs):\n return await _retry_do(config.session.post, url, *args,\n **_make_headers(config, kwargs))", "response": "Aiohttp wrapper for POST"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nrun the entire helper procedure", "response": "def run(self, positionals=None):\n '''run the entire helper procedure, including:\n\n - start: initialize the helper, collection preferences\n - record: record any relevant features for the environment / session\n - interact: interact with the user for additional informatoin\n - submit: submit the completed request\n \n Each of the above functions for a helper can determine global\n collection preferences from the system helpme.cfg in the module\n root. After performing global actions, each function then calls\n a class specific function of the same name (e.g., start calls\n _start) that is implemented by the helper class to do custom\n operations for the helper. \n '''\n # Step 0: Each run session is given a fun name\n self.run_id = RobotNamer().generate()\n\n # Step 1: get config steps\n steps = self.config._sections[self.name]\n\n # Step 2: Start the helper (announce and run start, which is init code)\n self.start(positionals)\n\n # Step 3: Iterate through flow, check each step for known record/prompt,\n # and collect outputs appropriately\n\n for step, content in steps.items():\n self.collect(step, content)\n \n # Step 4: When data collected, pass data structures to submit\n self.submit()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef start(self, positionals=None):\n '''start the helper flow. We check helper system configurations to\n determine components that should be collected for the submission.\n This is where the client can also pass on any extra (positional)\n arguments in a list from the user.\n '''\n bot.info('[helpme|%s]' %(self.name))\n self.speak()\n self._start(positionals)", "response": "start the helper flow"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncollect the required metadata from the user", "response": "def collect(self, step, content):\n '''given a name of a configuration key and the provided content, collect\n the required metadata from the user.\n \n Parameters\n ==========\n step: the key in the configuration. Can be one of:\n user_message_\n runtime_arg_\n record_asciinema\n record_environment\n user_prompt_\n content: the default value or boolean to indicate doing the step.\n '''\n\n # Option 1: The step is just a message to print to the user\n if step.startswith('user_message'):\n print(content) \n \n # Option 2: The step is to collect a user prompt (if not at runtime)\n elif step.startswith('user_prompt'):\n self.collect_argument(step, content)\n\n # Option 3: The step is to record an asciinema!\n elif step == 'record_asciinema':\n self.record_asciinema()\n\n # Option 4: Record the user environment\n elif step == \"record_environment\":\n self.record_environment()\n\n bot.debug(self.data)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncollecting the runtime argument if the key is not found in the configuration", "response": "def collect_argument(self, step, message):\n '''given a key in the configuration, collect the runtime argument if\n provided. Otherwise, prompt the user for the value.\n\n Parameters\n ==========\n step: the name of the step, should be 'runtime_arg_'\n message: the content of the step, the message to show the user if the\n argument is not found under args.\n\n '''\n if step not in self.data:\n self.data[step] = regexp_prompt(message)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef record_environment(self):\n '''collect a limited set of environment variables based on the list\n under record_envirionment in the configuration file.\n '''\n\n # whitelist is a newline separated list under record_environment\n\n envars = self._get_setting(name='whitelist', \n section='record_environment',\n user=False)\n\n if envars is not None:\n \n # User uppercase\n \n envars = [x.upper() for x in envars.split('\\n')]\n\n # Make transparent for the user\n\n bot.custom(prefix=\"Environment \",\n message='|'.join(envars),\n color=\"CYAN\")\n\n # Iterate through and collect based on name\n\n keep = [(k,v) for k,v in os.environ.items() if k.upper() in envars]\n\n # Ask the user for permission\n\n if confirm_prompt('Is this list ok to share?'):\n self.data['record_environment'] = keep", "response": "collect a limited set of environment variables based on the list\n under record_envirionment in the configuration file."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nrecord an asciinema from the user session saving the file to a temporary file and showing the user so if he needs to do it again.", "response": "def record_asciinema(self):\n '''record an asciinema from the user session, saving the file to\n a temporary file and showing the user so if he/she needs to do it\n again, the file can be provided. The flow of events below makes\n the following checks:\n\n 1. The user confirms it is ok to record\n 2. The record_asciinema setting is present and True in the config\n 3. An asciinema file path has not been provided by the user\n\n '''\n\n # If the user already provided a file, we don't need to ask again\n\n if 'record_asciinema' not in self.data:\n\n if confirm_prompt(\"Would you like to send a terminal recording?\"):\n\n try:\n record = self.config.getboolean(self.name, 'record_asciinema')\n filename = record_asciinema()\n self.data['record_asciinema'] = filename\n\n message = '''If you need to run helpme again you can give\n the path to this file with --asciinema %s''' % filename\n\n bot.custom(prefix=\"Asciinema \", message=message, color=\"CYAN\")\n\n except NoOptionError:\n\n bot.warning('Cannot record asciinema, skipping.')"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef speak(self):\n '''\n a function for the helper to announce him or herself, depending\n on the level specified. If you want your client to have additional\n announced things here, then implement the class `_speak` for your\n client.\n\n '''\n if self.quiet is False:\n bot.info('[helper|%s]' %(self.name))\n self._speak()", "response": "speak the user s log for this object"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nlogs the number of constants generated.", "response": "def __log_number_of_constants(self):\n \"\"\"\n Logs the number of constants generated.\n \"\"\"\n n_id = len(self._labels)\n n_widths = len(self._constants) - n_id\n\n self._io.writeln('')\n self._io.text('Number of constants based on column widths: {0}'.format(n_widths))\n self._io.text('Number of constants based on database IDs : {0}'.format(n_id))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreading the parameters from the configuration file.", "response": "def _read_configuration_file(self, config_filename):\n \"\"\"\n Reads parameters from the configuration file.\n\n :param str config_filename: The name of the configuration file.\n \"\"\"\n config = configparser.ConfigParser()\n config.read(config_filename)\n\n self._constants_filename = config.get('constants', 'columns')\n self._prefix = config.get('constants', 'prefix')\n self._class_name = config.get('constants', 'class')"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nwriting the constant class to the file.", "response": "def __write_constant_class(self):\n \"\"\"\n Inserts new and replaces old (if any) constant declaration statements in the class that acts like a namespace\n for constants.\n \"\"\"\n helper = ConstantClass(self._class_name, self._io)\n\n content = helper.source_with_constants(self._constants)\n\n Util.write_two_phases(helper.file_name(), content, self._io)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncheck that the environment variable is set and not empty.", "response": "def _check_env_var(envvar: str) -> bool:\n \"\"\"Check Environment Variable to verify that it is set and not empty.\n\n :param envvar: Environment Variable to Check.\n\n :returns: True if Environment Variable is set and not empty.\n\n :raises: KeyError if Environment Variable is not set or is empty.\n\n .. versionadded:: 0.0.12\n \"\"\"\n if os.getenv(envvar) is None:\n raise KeyError(\n \"Required ENVVAR: {0} is not set\".format(envvar))\n if not os.getenv(envvar): # test if env var is empty\n raise KeyError(\n \"Required ENVVAR: {0} is empty\".format(envvar))\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nvalidates State Argument Validate State", "response": "def valid_state(state: str) -> bool:\n \"\"\"Validate State Argument\n\n Checks that either 'on' or 'off' was entered as an argument to the\n CLI and make it lower case.\n\n :param state: state to validate.\n\n :returns: True if state is valid.\n\n .. versionchanged:: 0.0.12\n This moethod was renamed from validateState to valid_state to conform\n to PEP-8. Also removed \"magic\" text for state and instead reference the\n _VALID_STATES constant.\n \"\"\"\n lower_case_state = state.lower()\n\n if lower_case_state in _VALID_STATES:\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nvalidate that required env vars exist.", "response": "def valid_env_vars() -> bool:\n \"\"\"Validate that required env vars exist.\n\n :returns: True if required env vars exist.\n\n .. versionadded:: 0.0.12\n \"\"\"\n for envvar in _REQUIRED_ENV_VARS:\n try:\n _check_env_var(envvar)\n except KeyError as ex:\n LOG.error(ex)\n sys.exit(1)\n return True"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef configure_panel(self):\n webroot = os.path.dirname(__file__)\n\n self.template_path = os.path.join(webroot, 'templates')\n aiohttp_jinja2.setup(\n self, loader=jinja2.FileSystemLoader(self.template_path),\n filters={'sorted': sorted, 'int': int}\n )\n\n self['static_root_url'] = '/static'\n self.router.add_view('/', Panel)\n self.router.add_static(\n '/static/', path=os.path.join(webroot, 'static'), name='static'\n )", "response": "Configure templates and routing for the current locale."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\nasync def start(self):\n logging.info('Starting server, listening on %s.', self.port)\n runner = web.AppRunner(self)\n await runner.setup()\n site = web.TCPSite(runner, '', self.port)\n await site.start()", "response": "Initialize and start the server."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\nasync def get(self):\n shepherd = self.request.app.vmshepherd\n data = {'presets': {}, 'config': shepherd.config}\n presets = await shepherd.preset_manager.list_presets()\n runtime = shepherd.runtime_manager\n for name in presets:\n preset = shepherd.preset_manager.get_preset(name)\n data['presets'][name] = {\n 'preset': preset,\n 'vms': preset.vms,\n 'runtime': await runtime.get_preset_data(name),\n 'vmshepherd_id': shepherd.instance_id,\n 'now': time.time()\n }\n return data", "response": "Get all available presets and data"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nserialize a dictionary into a string like object.", "response": "def serialize(dictionary):\n \"\"\"\n Turn dictionary into argument like string.\n\n \"\"\"\n\n data = []\n for key, value in dictionary.items():\n data.append('{0}=\"{1}\"'.format(key, value))\n return ', '.join(data)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef getAmountOfHostsConnected(self, lanInterfaceId=1, timeout=1):\n namespace = Lan.getServiceType(\"getAmountOfHostsConnected\") + str(lanInterfaceId)\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetHostNumberOfEntries\", timeout=timeout)\n\n return int(results[\"NewHostNumberOfEntries\"])", "response": "Execute NewHostNumberOfEntries action to get the amount of hosts connected to the LAN interface."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nexecuting GetGenericHostEntry action to get detailed information s of a connected host.", "response": "def getHostDetailsByIndex(self, index, lanInterfaceId=1, timeout=1):\n \"\"\"Execute GetGenericHostEntry action to get detailed information's of a connected host.\n\n :param index: the index of the host\n :param int lanInterfaceId: the id of the LAN interface\n :param float timeout: the timeout to wait for the action to be executed\n :return: the detailed information's of a connected host.\n :rtype: HostDetails\n\n .. seealso:: :meth:`~simpletr64.actions.Lan.getAmountOfHostsConnected`\n \"\"\"\n namespace = Lan.getServiceType(\"getHostDetailsByIndex\") + str(lanInterfaceId)\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetGenericHostEntry\", timeout=timeout, NewIndex=index)\n\n return HostDetails(results)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef getHostDetailsByMACAddress(self, macAddress, lanInterfaceId=1, timeout=1):\n namespace = Lan.getServiceType(\"getHostDetailsByMACAddress\") + str(lanInterfaceId)\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetSpecificHostEntry\", timeout=timeout, NewMACAddress=macAddress)\n\n return HostDetails(results, macAddress=macAddress)", "response": "Execute GetSpecificHostEntry action to get the details for a host specified by its MAC address."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nexecutes GetInfo action to get information about the Ethernet interface.", "response": "def getEthernetInfo(self, lanInterfaceId=1, timeout=1):\n \"\"\"Execute GetInfo action to get information's about the Ethernet interface.\n\n :param int lanInterfaceId: the id of the LAN interface\n :param float timeout: the timeout to wait for the action to be executed\n :return: information's about the Ethernet interface.\n :rtype: EthernetInfo\n \"\"\"\n namespace = Lan.getServiceType(\"getEthernetInfo\") + str(lanInterfaceId)\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetInfo\", timeout=timeout)\n\n return EthernetInfo(results)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nexecute GetStatistics action to get statistics of the Ethernet interface.", "response": "def getEthernetStatistic(self, lanInterfaceId=1, timeout=1):\n \"\"\"Execute GetStatistics action to get statistics of the Ethernet interface.\n\n :param int lanInterfaceId: the id of the LAN interface\n :param float timeout: the timeout to wait for the action to be executed\n :return: statisticss of the Ethernet interface.\n :rtype: EthernetStatistic\n \"\"\"\n namespace = Lan.getServiceType(\"getEthernetStatistic\") + str(lanInterfaceId)\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetStatistics\", timeout=timeout)\n\n return EthernetStatistic(results)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef setEnable(self, status, lanInterfaceId=1, timeout=1):\n namespace = Lan.getServiceType(\"setEnable\") + str(lanInterfaceId)\n uri = self.getControlURL(namespace)\n\n if status:\n setStatus = 1\n else:\n setStatus = 0\n\n self.execute(uri, namespace, \"SetEnable\", timeout=timeout, NewEnable=setStatus)", "response": "Set enable status for a LAN interface"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsubmit a uservoice ticket.", "response": "def _submit(self):\n '''submit a uservoice ticket. When we get here we should have:\n \n {'user_prompt_issue': 'I want to do the thing.', \n 'record_asciinema': '/tmp/helpme.93o__nt5.json',\n 'record_environment': ((1,1),(2,2)...(N,N))}\n\n Required Client Variables\n self.api_key\n self.api_secret\n self.subdomain\n self.email\n\n '''\n\n # Step 0: Authenticate with uservoice API\n self.authenticate()\n \n title = \"HelpMe UserVoice Ticket: %s\" %(self.run_id)\n body = self.data['user_prompt_issue']\n\n # Step 1: Environment\n\n envars = self.data.get('record_environment')\n if envars not in [None, '', []]:\n body += '\\n\\nEnvironment:\\n'\n for envar in envars:\n body += ' - %s: %s\\n' %(envar[0], envar[1])\n\n # Step 2: Asciinema\n\n asciinema = self.data.get('record_asciinema')\n if asciinema not in [None, '']:\n url = upload_asciinema(asciinema)\n \n # If the upload is successful, add a link to it.\n\n if url is not None:\n body += \"\\n\\nAsciinema Recording: %s\" %url\n \n # Add other metadata about client\n\n body += \"\\ngenerated by HelpMe: https://vsoch.github.io/helpme/\"\n\n # Submit the ticket!\n\n self.post_ticket(title, body)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef authenticate(self):\n '''authenticate with uservoice by creating a client.'''\n\n if not hasattr(self, 'client'):\n self.client = uservoice.Client(self.subdomain,\n self.api_key,\n self.api_secret)", "response": "authenticate with uservoice by creating a client."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef post_ticket(self, title, body):\n '''post_ticket will post a ticket to the uservoice helpdesk\n\n Parameters\n ==========\n title: the title (subject) of the issue\n body: the message to send\n\n '''\n\n # Populate the ticket\n ticket = {'subject': title,\n 'message': body }\n\n response = self.client.post(\"/api/v1/tickets.json\", {\n 'email': self.email,\n 'ticket': ticket })['ticket']\n bot.info(response['url'])", "response": "post a ticket to the uservoice helpdesk\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef read(datapath, qt_app=None, dataplus_format=True, gui=False, start=0, stop=None, step=1, convert_to_gray=True,\n series_number=None, dicom_expected=None, **kwargs):\n \"\"\"Simple read function. Internally calls DataReader.Get3DData()\"\"\"\n dr = DataReader()\n return dr.Get3DData(datapath=datapath, qt_app=qt_app, dataplus_format=dataplus_format, gui=gui, start=start,\n stop=stop, step=step, convert_to_gray=convert_to_gray, series_number=series_number,\n use_economic_dtype=True, dicom_expected=dicom_expected , **kwargs)", "response": "Simple 3D read function. Internally calls DataReader. Get3DData"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfunctions which returns metadata dict.", "response": "def _metadata(image, datapath):\n \"\"\"Function which returns metadata dict.\n\n :param image: image to get spacing from\n :param datapath: path to data\n :return: {'series_number': '', 'datadir': '', 'voxelsize_mm': ''}\n \"\"\"\n metadata = {'series_number': 0, 'datadir': datapath}\n spacing = image.GetSpacing()\n metadata['voxelsize_mm'] = [\n spacing[2],\n spacing[0],\n spacing[1],\n ]\n return metadata"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting 3D data from a given directory.", "response": "def Get3DData(self, datapath, qt_app=None, dataplus_format=True, gui=False, start=0, stop=None, step=1,\n convert_to_gray=True, series_number=None, use_economic_dtype=True, dicom_expected=None, **kwargs):\n \"\"\"Returns 3D data and its metadata.\n\n # NOTE(:param qt_app:) If it is set to None (as default) all dialogs for series selection are performed in\n terminal. If qt_app is set to QtGui.QApplication() dialogs are in Qt.\n\n :param datapath: directory with input data\n :param qt_app: Dialog destination. If None (default) -> terminal, if 'QtGui.QApplication()' -> Qt\n :param dataplus_format: New data format. Metadata and data are returned in one structure.\n :param gui: True if 'QtGui.QApplication()' instead of terminal should be used\n :param int start: used for DicomReader, defines where 3D data reading should start\n :param int stop: used for DicomReader, defines where 3D data reading should stop\n :param int step: used for DicomReader, defines step for 3D data reading\n :param bool convert_to_gray: if True -> RGB is converted to gray\n :param int series_number: used in DicomReader, essential in metadata\n :param use_economic_dtype: if True, casts 3D data array to less space consuming dtype\n :param dicom_expected: set true if it is known that data is in dicom format. Set False to suppress\n dicom warnings.\n :return: tuple (data3d, metadata)\n \"\"\"\n self.orig_datapath = datapath\n datapath = os.path.expanduser(datapath)\n\n if series_number is not None and type(series_number) != int:\n series_number = int(series_number)\n\n if not os.path.exists(datapath):\n logger.error(\"Path '\" + datapath + \"' does not exist\")\n return\n if qt_app is None and gui is True:\n from PyQt4.QtGui import QApplication\n qt_app = QApplication(sys.argv)\n\n if type(datapath) is not str:\n datapath = str(datapath)\n datapath = os.path.normpath(datapath)\n\n self.start = start\n self.stop = stop\n self.step = step\n self.convert_to_gray = convert_to_gray\n self.series_number = series_number\n self.kwargs = kwargs\n self.qt_app = qt_app\n self.gui = gui\n\n if os.path.isfile(datapath):\n logger.debug('file read recognized')\n data3d, metadata = self.__ReadFromFile(datapath)\n\n elif os.path.exists(datapath):\n logger.debug('directory read recognized')\n data3d, metadata = self.__ReadFromDirectory(datapath=datapath, dicom_expected=dicom_expected)\n # datapath, start, stop, step, gui=gui, **kwargs)\n else:\n logger.error('Data path {} not found'.format(datapath))\n\n if convert_to_gray:\n if len(data3d.shape) > 3:\n # TODO: implement better rgb2gray\n data3d = data3d[:, :, :, 0]\n if use_economic_dtype:\n data3d = self.__use_economic_dtype(data3d)\n\n if dataplus_format:\n logger.debug('dataplus format')\n # metadata = {'voxelsize_mm': [1, 1, 1]}\n datap = metadata\n datap['data3d'] = data3d\n logger.debug('datap keys () : ' + str(datap.keys()))\n return datap\n else:\n return data3d, metadata"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef __ReadFromFile(self, datapath):\n def _create_meta(_datapath):\n \"\"\"Just simply returns some dict. This functions exists in order to keep DRY\"\"\"\n meta = {\n 'series_number': 0,\n 'datadir': _datapath\n }\n return meta\n\n path, ext = os.path.splitext(datapath)\n ext = ext[1:]\n if ext in ('pklz', 'pkl'):\n logger.debug('pklz format detected')\n from . import misc\n data = misc.obj_from_file(datapath, filetype='pkl')\n data3d = data.pop('data3d')\n # metadata must have series_number\n metadata = _create_meta(datapath)\n metadata.update(data)\n\n elif ext in ['hdf5']:\n from . import hdf5_io\n datap = hdf5_io.load_dict_from_hdf5(datapath)\n # datap = self.read_hdf5(datapath)\n data3d = datap.pop('data3d')\n\n # back compatibility\n if 'metadata' in datap.keys():\n datap = datap['metadata']\n # metadata must have series_number\n metadata = _create_meta(datapath)\n metadata.update(datap)\n\n elif ext in ['idx']:\n from . import idxformat\n idxreader = idxformat.IDXReader()\n data3d, metadata = idxreader.read(datapath)\n elif ext in ['dcm', 'DCM', 'dicom']:\n data3d, metadata = self._read_with_sitk(datapath)\n metadata = self._fix_sitk_bug(datapath, metadata)\n elif ext in [\"bz2\"]:\n new_datapath = tgz.untar(datapath)\n data3d, metadata = self.__ReadFromDirectory(new_datapath)\n else:\n logger.debug('file format \"' + str(ext) + '\"')\n # reading raw file\n data3d, metadata = self._read_with_sitk(datapath)\n return data3d, metadata", "response": "Reads a single file and returns 3D data and its metadata."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _read_with_sitk(datapath):\n try:\n import SimpleITK as Sitk\n except ImportError as e:\n logger.error(\"Unable to import SimpleITK. On Windows try version 1.0.1\")\n image = Sitk.ReadImage(datapath)\n data3d = dcmtools.get_pixel_array_from_sitk(image)\n # data3d, original_dtype = dcmreaddata.get_pixel_array_from_dcmobj(image)\n metadata = _metadata(image, datapath)\n return data3d, metadata", "response": "Reads file using SimpleITK. Returns array of pixels image and metadata."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _fix_sitk_bug(path, metadata):\n ds = dicom.read_file(path)\n try:\n metadata[\"voxelsize_mm\"][0] = ds.SpacingBetweenSlices\n except Exception as e:\n logger.warning(\"Read dicom 'SpacingBetweenSlices' failed: \", e)\n return metadata", "response": "This function is a fix for simple ITK for Z axis in 3D images."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_pid(pid=None):\n '''get_pid will return a pid of interest. First we use given variable,\n then environmental variable PID, and then PID of running process\n '''\n if pid == None:\n if os.environ.get(\"PID\",None) != None:\n pid = int(os.environ.get(\"PID\"))\n # Then use current running script as process\n else:\n pid = os.getpid()\n print(\"pid is %s\" %pid)\n return pid", "response": "get_pid will return a pid of interest. First we use given variable and environmental variable PID then current running script as process\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nload the stored routine into the instance of MySQL.", "response": "def load_stored_routine(self):\n \"\"\"\n Loads the stored routine into the instance of MySQL.\n\n Returns the metadata of the stored routine if the stored routine is loaded successfully. Otherwise returns\n False.\n\n :rtype: dict[str,str]|bool\n \"\"\"\n try:\n self._routine_name = os.path.splitext(os.path.basename(self._source_filename))[0]\n\n if os.path.exists(self._source_filename):\n if os.path.isfile(self._source_filename):\n self._m_time = int(os.path.getmtime(self._source_filename))\n else:\n raise LoaderException(\"Unable to get mtime of file '{}'\".format(self._source_filename))\n else:\n raise LoaderException(\"Source file '{}' does not exist\".format(self._source_filename))\n\n if self._pystratum_old_metadata:\n self._pystratum_metadata = self._pystratum_old_metadata\n\n load = self._must_reload()\n if load:\n self.__read_source_file()\n\n self.__get_placeholders()\n\n self._get_designation_type()\n\n self._get_name()\n\n self.__substitute_replace_pairs()\n\n self._load_routine_file()\n\n if self._designation_type == 'bulk_insert':\n self._get_bulk_insert_table_columns_info()\n\n self._get_routine_parameters_info()\n\n self.__get_doc_block_parts_wrapper()\n\n self.__save_shadow_copy()\n\n self._update_metadata()\n\n return self._pystratum_metadata\n\n except Exception as exception:\n self._log_exception(exception)\n return False"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef __read_source_file(self):\n with open(self._source_filename, 'r', encoding=self._routine_file_encoding) as file:\n self._routine_source_code = file.read()\n\n self._routine_source_code_lines = self._routine_source_code.split(\"\\n\")", "response": "Reads the source file and sets the attributes self. _routine_source_code and self. _routine_source_lines."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsave a copy of the stored routine source with pure SQL.", "response": "def __save_shadow_copy(self):\n \"\"\"\n Saves a copy of the stored routine source with pure SQL (if shadow directory is set).\n \"\"\"\n if not self.shadow_directory:\n return\n\n destination_filename = os.path.join(self.shadow_directory, self._routine_name) + '.sql'\n\n if os.path.realpath(destination_filename) == os.path.realpath(self._source_filename):\n raise LoaderException(\"Shadow copy will override routine source '{}'\".format(self._source_filename))\n\n # Remove the (read only) shadow file if it exists.\n if os.path.exists(destination_filename):\n os.remove(destination_filename)\n\n # Write the shadow file.\n with open(destination_filename, 'wt', encoding=self._routine_file_encoding) as handle:\n handle.write(self._routine_source_code)\n\n # Make the file read only.\n mode = os.stat(self._source_filename)[stat.ST_MODE]\n os.chmod(destination_filename, mode & ~stat.S_IWUSR & ~stat.S_IWGRP & ~stat.S_IWOTH)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef __substitute_replace_pairs(self):\n self._set_magic_constants()\n\n routine_source = []\n i = 0\n for line in self._routine_source_code_lines:\n self._replace['__LINE__'] = \"'%d'\" % (i + 1)\n for search, replace in self._replace.items():\n tmp = re.findall(search, line, re.IGNORECASE)\n if tmp:\n line = line.replace(tmp[0], replace)\n routine_source.append(line)\n i += 1\n\n self._routine_source_code = \"\\n\".join(routine_source)", "response": "Substitutes all replace pairs in the stored routine source code."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _log_exception(self, exception):\n self._io.error(str(exception).strip().split(os.linesep))", "response": "Logs an exception.\n\n :param Exception exception: The exception.\n\n :rtype: None"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef __get_placeholders(self):\n ret = True\n\n pattern = re.compile('(@[A-Za-z0-9_.]+(%(max-)?type)?@)')\n matches = pattern.findall(self._routine_source_code)\n\n placeholders = []\n\n if len(matches) != 0:\n for tmp in matches:\n placeholder = tmp[0]\n if placeholder.lower() not in self._replace_pairs:\n raise LoaderException(\"Unknown placeholder '{0}' in file {1}\".\n format(placeholder, self._source_filename))\n if placeholder not in placeholders:\n placeholders.append(placeholder)\n\n for placeholder in placeholders:\n if placeholder not in self._replace:\n self._replace[placeholder] = self._replace_pairs[placeholder.lower()]\n\n return ret", "response": "Extracts the placeholders from the stored routine source."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_designation_type(self):\n positions = self._get_specification_positions()\n if positions[0] != -1 and positions[1] != -1:\n pattern = re.compile(r'^\\s*--\\s+type\\s*:\\s*(\\w+)\\s*(.+)?\\s*', re.IGNORECASE)\n for line_number in range(positions[0], positions[1] + 1):\n matches = pattern.findall(self._routine_source_code_lines[line_number])\n if matches:\n self._designation_type = matches[0][0].lower()\n tmp = str(matches[0][1])\n if self._designation_type == 'bulk_insert':\n n = re.compile(r'([a-zA-Z0-9_]+)\\s+([a-zA-Z0-9_,]+)', re.IGNORECASE)\n info = n.findall(tmp)\n if not info:\n raise LoaderException('Expected: -- type: bulk_insert in file {0}'.\n format(self._source_filename))\n self._table_name = info[0][0]\n self._columns = str(info[0][1]).split(',')\n\n elif self._designation_type == 'rows_with_key' or self._designation_type == 'rows_with_index':\n self._columns = str(matches[0][1]).split(',')\n else:\n if matches[0][1]:\n raise LoaderException('Expected: -- type: {}'.format(self._designation_type))\n\n if not self._designation_type:\n raise LoaderException(\"Unable to find the designation type of the stored routine in file {0}\".\n format(self._source_filename))", "response": "Extracts the designation type of the stored routine."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_specification_positions(self):\n start = -1\n for (i, line) in enumerate(self._routine_source_code_lines):\n if self._is_start_of_stored_routine(line):\n start = i\n\n end = -1\n for (i, line) in enumerate(self._routine_source_code_lines):\n if self._is_start_of_stored_routine_body(line):\n end = i - 1\n\n return start, end", "response": "Returns a tuple with the start and end line numbers of the stored routine specification."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef __get_doc_block_lines(self):\n line1 = None\n line2 = None\n\n i = 0\n for line in self._routine_source_code_lines:\n if re.match(r'\\s*/\\*\\*', line):\n line1 = i\n\n if re.match(r'\\s*\\*/', line):\n line2 = i\n\n if self._is_start_of_stored_routine(line):\n break\n\n i += 1\n\n return line1, line2", "response": "Returns the start and end line of the DOcBlock of the stored routine code."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef __get_doc_block_parts_source(self):\n line1, line2 = self.__get_doc_block_lines()\n\n if line1 is not None and line2 is not None and line1 <= line2:\n doc_block = self._routine_source_code_lines[line1:line2 - line1 + 1]\n else:\n doc_block = list()\n\n reflection = DocBlockReflection(doc_block)\n\n self._doc_block_parts_source['description'] = reflection.get_description()\n\n self._doc_block_parts_source['parameters'] = list()\n for tag in reflection.get_tags('param'):\n parts = re.match(r'^(@param)\\s+(\\w+)\\s*(.+)?', tag, re.DOTALL)\n if parts:\n self._doc_block_parts_source['parameters'].append({'name': parts.group(2),\n 'description': parts.group(3)})", "response": "Extracts the DocBlock from the stored routine source."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef __get_doc_block_parts_wrapper(self):\n self.__get_doc_block_parts_source()\n\n helper = self._get_data_type_helper()\n\n parameters = list()\n for parameter_info in self._parameters:\n parameters.append(\n {'parameter_name': parameter_info['name'],\n 'python_type': helper.column_type_to_python_type(parameter_info),\n 'data_type_descriptor': parameter_info['data_type_descriptor'],\n 'description': self.__get_parameter_doc_description(parameter_info['name'])})\n\n self._doc_block_parts_wrapper['description'] = self._doc_block_parts_source['description']\n self._doc_block_parts_wrapper['parameters'] = parameters", "response": "Generates the DocBlock parts to be used by the wrapper generator."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nupdating the metadata of the stored routine.", "response": "def _update_metadata(self):\n \"\"\"\n Updates the metadata of the stored routine.\n \"\"\"\n self._pystratum_metadata['routine_name'] = self._routine_name\n self._pystratum_metadata['designation'] = self._designation_type\n self._pystratum_metadata['table_name'] = self._table_name\n self._pystratum_metadata['parameters'] = self._parameters\n self._pystratum_metadata['columns'] = self._columns\n self._pystratum_metadata['fields'] = self._fields\n self._pystratum_metadata['column_types'] = self._columns_types\n self._pystratum_metadata['timestamp'] = self._m_time\n self._pystratum_metadata['replace'] = self._replace\n self._pystratum_metadata['pydoc'] = self._doc_block_parts_wrapper"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsets the magic constants to replace list.", "response": "def _set_magic_constants(self):\n \"\"\"\n Adds magic constants to replace list.\n \"\"\"\n real_path = os.path.realpath(self._source_filename)\n\n self._replace['__FILE__'] = \"'%s'\" % real_path\n self._replace['__ROUTINE__'] = \"'%s'\" % self._routine_name\n self._replace['__DIR__'] = \"'%s'\" % os.path.dirname(real_path)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nremoves magic constants from current replace list.", "response": "def _unset_magic_constants(self):\n \"\"\"\n Removes magic constants from current replace list.\n \"\"\"\n if '__FILE__' in self._replace:\n del self._replace['__FILE__']\n\n if '__ROUTINE__' in self._replace:\n del self._replace['__ROUTINE__']\n\n if '__DIR__' in self._replace:\n del self._replace['__DIR__']\n\n if '__LINE__' in self._replace:\n del self._replace['__LINE__']"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _print_sql_with_error(self, sql, error_line):\n if os.linesep in sql:\n lines = sql.split(os.linesep)\n digits = math.ceil(math.log(len(lines) + 1, 10))\n i = 1\n for line in lines:\n if i == error_line:\n self._io.text('{0:{width}} {1}'.format(i, line, width=digits, ))\n else:\n self._io.text('{0:{width}} {1}'.format(i, line, width=digits, ))\n i += 1\n else:\n self._io.text(sql)", "response": "Writes a SQL statement with an error to the output."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a list of items according to filter parameters.", "response": "def list_filter(lst, startswith=None, notstartswith=None,\n contain=None, notcontain=None):\n \"\"\" Keep in list items according to filter parameters.\n\n :param lst: item list\n :param startswith: keep items starting with\n :param notstartswith: remove items starting with\n :return:\n \"\"\"\n keeped = []\n for item in lst:\n keep = False\n if startswith is not None:\n if item.startswith(startswith):\n keep = True\n if notstartswith is not None:\n if not item.startswith(notstartswith):\n keep = True\n if contain is not None:\n if contain in item:\n keep = True\n if notcontain is not None:\n if not notcontain in item:\n keep = True\n\n if keep:\n keeped.append(item)\n return keeped"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsplitting dict into two subdicts based on keys.", "response": "def split_dict(dct, keys):\n \"\"\"\n Split dict into two subdicts based on keys.\n\n :param dct:\n :param keys:\n :return: dict_in, dict_out\n \"\"\"\n if type(dct) == collections.OrderedDict:\n dict_in = collections.OrderedDict()\n dict_out = collections.OrderedDict()\n else:\n dict_in = {}\n dict_out = {}\n\n for key, value in dct.items:\n if key in keys:\n dict_in[key] = value\n else:\n dict_out[key] = value\n return dict_in, dict_out"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nflattens dict with defined key join symbol.", "response": "def flatten_dict_join_keys(dct, join_symbol=\" \"):\n \"\"\" Flatten dict with defined key join symbol.\n\n :param dct: dict to flatten\n :param join_symbol: default value is \" \"\n :return:\n \"\"\"\n return dict( flatten_dict(dct, join=lambda a,b:a+join_symbol+b) )"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef list_contains(list_of_strings, substring, return_true_false_array=False):\n key_tf = [keyi.find(substring) != -1 for keyi in list_of_strings]\n if return_true_false_array:\n return key_tf\n keys_to_remove = list_of_strings[key_tf]\n return keys_to_remove", "response": "Get strings in list which contains substring."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef df_drop_duplicates(df, ignore_key_pattern=\"time\"):\n\n keys_to_remove = list_contains(df.keys(), ignore_key_pattern)\n #key_tf = [key.find(noinfo_key_pattern) != -1 for key in df.keys()]\n # keys_to_remove\n # remove duplicates\n ks = copy.copy(list(df.keys()))\n for key in keys_to_remove:\n ks.remove(key)\n\n df = df.drop_duplicates(ks)\n return df", "response": "Drop duplicates from dataframe ignore columns with keys containing defined pattern."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef ndarray_to_list_in_structure(item, squeeze=True):\n tp = type(item)\n\n if tp == np.ndarray:\n if squeeze:\n item = item.squeeze()\n item = item.tolist()\n elif tp == list:\n for i in range(len(item)):\n item[i] = ndarray_to_list_in_structure(item[i])\n elif tp == dict:\n for lab in item:\n item[lab] = ndarray_to_list_in_structure(item[lab])\n\n return item", "response": "Change ndarray in structure of lists and dicts into lists."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfinding first suitable key in dict.", "response": "def dict_find_key(dd, value):\n \"\"\" Find first suitable key in dict.\n\n :param dd:\n :param value:\n :return:\n \"\"\"\n key = next(key for key, val in dd.items() if val == value)\n return key"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sort_list_of_dicts(lst_of_dct, keys, reverse=False, **sort_args):\n\n if type(keys) != list:\n keys = [keys]\n # dcmdir = lst_of_dct[:]\n # lst_of_dct.sort(key=lambda x: [x[key] for key in keys], reverse=reverse, **sort_args)\n lst_of_dct.sort(key=lambda x: [((False, x[key]) if key in x else (True, 0)) for key in keys], reverse=reverse, **sort_args)\n return lst_of_dct", "response": "Sort list of dicts by one or multiple keys."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef ordered_dict_to_dict(config):\n\n if type(config) == collections.OrderedDict:\n config = dict(config)\n if type(config) == list:\n for i in range(0, len(config)):\n config[i] = ordered_dict_to_dict(config[i])\n elif type(config) == dict:\n for key in config:\n config[key] = ordered_dict_to_dict(config[key])\n\n return config", "response": "Convert ordered dict to dict"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef run_command(self, config_file):\n config = configparser.ConfigParser()\n config.read(config_file)\n\n rdbms = config.get('database', 'rdbms').lower()\n\n wrapper = self.create_routine_wrapper_generator(rdbms)\n wrapper.main(config_file)", "response": "Runs the neccesary command."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create_routine_wrapper_generator(self, rdbms):\n # Note: We load modules and classes dynamically such that on the end user's system only the required modules\n # and other dependencies for the targeted RDBMS must be installed (and required modules and other\n # dependencies for the other RDBMSs are not required).\n\n if rdbms == 'mysql':\n module = locate('pystratum_mysql.MySqlRoutineWrapperGenerator')\n return module.MySqlRoutineWrapperGenerator(self.output)\n\n if rdbms == 'mssql':\n module = locate('pystratum_mssql.MsSqlRoutineWrapperGenerator')\n return module.MsSqlRoutineWrapperGenerator(self.output)\n\n if rdbms == 'pgsql':\n module = locate('pystratum_pgsql.PgSqlRoutineWrapperGenerator')\n return module.PgSqlRoutineWrapperGenerator(self.output)\n\n raise Exception(\"Unknown RDBMS '{0!s}'.\".format(rdbms))", "response": "Returns a routine wrapper generator for the given RDBMS."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreading an option for a configuration file.", "response": "def _get_option(config, supplement, section, option, fallback=None):\n \"\"\"\n Reads an option for a configuration file.\n\n :param configparser.ConfigParser config: The main config file.\n :param configparser.ConfigParser supplement: The supplement config file.\n :param str section: The name of the section op the option.\n :param str option: The name of the option.\n :param str|None fallback: The fallback value of the option if it is not set in either configuration files.\n\n :rtype: str\n\n :raise KeyError:\n \"\"\"\n if supplement:\n return_value = supplement.get(section, option, fallback=config.get(section, option, fallback=fallback))\n else:\n return_value = config.get(section, option, fallback=fallback)\n\n if fallback is None and return_value is None:\n raise KeyError(\"Option '{0!s}' is not found in section '{1!s}'.\".format(option, section))\n\n return return_value"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreading the configuration file and returns the configparser and config_supplement object.", "response": "def _read_configuration(config_filename):\n \"\"\"\n Checks the supplement file.\n\n :param str config_filename: The name of the configuration file.\n\n :rtype: (configparser.ConfigParser,configparser.ConfigParser)\n \"\"\"\n config = ConfigParser()\n config.read(config_filename)\n\n if 'supplement' in config['database']:\n path = os.path.dirname(config_filename) + '/' + config.get('database', 'supplement')\n config_supplement = ConfigParser()\n config_supplement.read(path)\n else:\n config_supplement = None\n\n return config, config_supplement"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef record_asciinema():\n '''a wrapper around generation of an asciinema.api.Api and a custom \n recorder to pull out the input arguments to the Record from argparse.\n The function generates a filename in advance and a return code\n so we can check the final status. \n '''\n\n import asciinema.config as aconfig\n from asciinema.api import Api\n\n # Load the API class\n\n cfg = aconfig.load()\n api = Api(cfg.api_url, os.environ.get(\"USER\"), cfg.install_id)\n\n # Create dummy class to pass in as args\n recorder = HelpMeRecord(api)\n code = recorder.execute()\n \n if code == 0 and os.path.exists(recorder.filename):\n return recorder.filename\n print('Problem generating %s, return code %s' %(recorder.filename, code))", "response": "a wrapper around generating an asciinema. api. Api and a custom helpmeRecord class to pull out the input arguments to the Record from argparse."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef sendWakeOnLan(self, macAddress, lanInterfaceId=1, timeout=1):\n namespace = Fritz.getServiceType(\"sendWakeOnLan\") + str(lanInterfaceId)\n uri = self.getControlURL(namespace)\n\n self.execute(uri, namespace, \"X_AVM-DE_WakeOnLANByMACAddress\", timeout=timeout,\n NewMACAddress=macAddress)", "response": "Send a wake up package to a device specified by its MAC address."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndoes a software update of the Fritz Box if available.", "response": "def doUpdate(self, timeout=1):\n \"\"\"Do a software update of the Fritz Box if available.\n\n :param float timeout: the timeout to wait for the action to be executed\n :return: a list of if an update was available and the update state (bool, str)\n :rtype: tuple(bool, str)\n \"\"\"\n namespace = Fritz.getServiceType(\"doUpdate\")\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"X_AVM-DE_DoUpdate\", timeout=timeout)\n\n return results[\"NewUpgradeAvailable\"], results[\"NewX_AVM-DE_UpdateState\"]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef isOptimizedForIPTV(self, wifiInterfaceId=1, timeout=1):\n namespace = Fritz.getServiceType(\"isOptimizedForIPTV\") + str(wifiInterfaceId)\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"X_AVM-DE_GetIPTVOptimized\", timeout=timeout)\n\n return bool(int(results[\"NewX_AVM-DE_IPTVoptimize\"]))", "response": "Execute X_AVM - DE - GetIPTVOptimized action to get the optimized status of the Wifi interface."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef setOptimizedForIPTV(self, status, wifiInterfaceId=1, timeout=1):\n namespace = Fritz.getServiceType(\"setOptimizedForIPTV\") + str(wifiInterfaceId)\n uri = self.getControlURL(namespace)\n\n if status:\n setStatus = 1\n else:\n setStatus = 0\n\n arguments = {\"timeout\": timeout, \"NewX_AVM-DE_IPTVoptimize\": setStatus}\n\n self.execute(uri, namespace, \"X_AVM-DE_SetIPTVOptimized\", **arguments)", "response": "Set if the Wifi interface is optimized for IP TV"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets the list of phone calls made by the user.", "response": "def getCallList(self, timeout=1):\n \"\"\"Get the list of phone calls made\n\n Example of a phone call result:\n\n ::\n\n [{'Count': None, 'Name': None, 'CalledNumber': '030868709971', 'Numbertype': 'sip', 'Duration': '0:01',\n 'Caller': '015155255399', 'Called': 'SIP: 030868729971', 'Date': '02.01.14 13:14',\n 'Device': 'Anrufbeantworter','Path': None, 'Port': '40', 'Type': '1', 'Id': '15'}]\n\n Types:\n\n * 1 - answered\n * 2 - missed\n * 3 - outgoing\n\n :param float timeout: the timeout to wait for the action to be executed\n :return: the list of made phone calls\n :rtype: list[dict[str: str]]\n \"\"\"\n namespace = Fritz.getServiceType(\"getCallList\")\n uri = self.getControlURL(namespace)\n\n results = self.execute(uri, namespace, \"GetCallList\")\n\n # setup proxies\n proxies = {}\n if self.httpProxy:\n proxies = {\"https\": self.httpProxy}\n\n if self.httpsProxy:\n proxies = {\"http\": self.httpsProxy}\n\n # get the content\n request = requests.get(results[\"NewCallListURL\"], proxies=proxies, timeout=float(timeout))\n\n if request.status_code != 200:\n errorStr = DeviceTR64._extractErrorString(request)\n raise ValueError('Could not get CPE definitions \"' + results[\"NewCallListURL\"] + '\" : ' +\n str(request.status_code) + ' - ' + request.reason + \" -- \" + errorStr)\n\n # parse xml\n try:\n root = ET.fromstring(request.text.encode('utf-8'))\n except Exception as e:\n raise ValueError(\"Could not parse call list '\" + results[\"NewCallListURL\"] + \"': \" + str(e))\n\n calls = []\n\n for child in root.getchildren():\n if child.tag.lower() == \"call\":\n\n callParameters = {}\n\n for callChild in child.getchildren():\n callParameters[callChild.tag] = callChild.text\n\n calls.append(callParameters)\n\n return calls"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef add_tcp_callback(port, callback, threaded_callback=False):\n _rpio.add_tcp_callback(port, callback, threaded_callback)", "response": "Adds a unix socket server callback which will be invoked when values\n arrive from a connected socket client."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef add_interrupt_callback(gpio_id, callback, edge='both', \\\n pull_up_down=PUD_OFF, threaded_callback=False, \\\n debounce_timeout_ms=None):\n \"\"\"\n Add a callback to be executed when the value on 'gpio_id' changes to\n the edge specified via the 'edge' parameter (default='both').\n\n `pull_up_down` can be set to `RPIO.PUD_UP`, `RPIO.PUD_DOWN`, and\n `RPIO.PUD_OFF`.\n\n If `threaded_callback` is True, the callback will be started\n inside a Thread.\n\n If debounce_timeout_ms is set, new interrupts will not be forwarded\n until after the specified amount of milliseconds.\n \"\"\"\n _rpio.add_interrupt_callback(gpio_id, callback, edge, pull_up_down, \\\n threaded_callback, debounce_timeout_ms)", "response": "Add a callback to be executed when the value of gpio_id changes to the edge specified via the edge parameter."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nblocks loop to listen for GPIO interrupts and distribute them to the associated callbacks.", "response": "def wait_for_interrupts(threaded=False, epoll_timeout=1):\n \"\"\"\n Blocking loop to listen for GPIO interrupts and distribute them to\n associated callbacks. epoll_timeout is an easy way to shutdown the\n blocking function. Per default the timeout is set to 1 second; if\n `_is_waiting_for_interrupts` is set to False the loop will exit.\n\n If an exception occurs while waiting for interrupts, the interrupt\n gpio interfaces will be cleaned up (/sys/class/gpio unexports). In\n this case all interrupts will be reset and you'd need to add the\n callbacks again before using `wait_for_interrupts(..)` again.\n\n If the argument `threaded` is True, wait_for_interrupts will be\n started in a daemon Thread. To quit it, call\n `RPIO.stop_waiting_for_interrupts()`.\n \"\"\"\n if threaded:\n t = Thread(target=_rpio.wait_for_interrupts, args=(epoll_timeout,))\n t.daemon = True\n t.start()\n else:\n _rpio.wait_for_interrupts(epoll_timeout)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nbuilds source with Python 2. 7 and 3. 2 and tests.", "response": "def build_gpio():\n \"\"\" Builds source with Python 2.7 and 3.2, and tests import \"\"\"\n with cd(\"/tmp/source/c_gpio\"):\n test = \"import _GPIO; print(_GPIO.VERSION_GPIO)\"\n run(\"make gpio2.7 && cp build/_GPIO.so .\")\n run('sudo python2.7 -c \"%s\"' % test)\n run(\"cp _GPIO.so ../RPIO/\")\n run(\"cp _GPIO.so ../RPIO/_GPIO27.so\")\n run(\"make gpio3.2 && cp build/_GPIO.so .\")\n run('sudo python3.2 -c \"%s\"' % test)\n run(\"mv _GPIO.so ../RPIO/_GPIO32.so\")"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef build_pwm():\n with cd(\"/tmp/source/c_pwm\"):\n test = \"import _PWM; print(_PWM.VERSION)\"\n run(\"make py2.7\")\n run('sudo python2.7 -c \"%s\"' % test)\n run(\"cp _PWM.so ../RPIO/PWM/\")\n run(\"mv _PWM.so ../RPIO/PWM/_PWM27.so\")\n run(\"make py3.2\")\n run('python3.2 -c \"%s\"' % test)\n run(\"mv _PWM.so ../RPIO/PWM/_PWM32.so\")", "response": "Builds source with Python 2. 7 and 3. 2 and tests."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef upload_to_pypi():\n # One more safety input and then we are ready to go :)\n x = prompt(\"Are you sure to upload the current version to pypi?\")\n if not x or not x.lower() in [\"y\", \"yes\"]:\n return\n\n local(\"rm -rf dist\")\n local(\"python setup.py sdist\")\n version = _get_cur_version()\n fn = \"RPIO-%s.tar.gz\" % version\n put(\"dist/%s\" % fn, \"/tmp/\")\n with cd(\"/tmp\"):\n run(\"tar -xf /tmp/%s\" % fn)\n with cd(\"/tmp/RPIO-%s\" % version):\n run(\"python2.6 setup.py bdist_egg upload\")\n run(\"python2.7 setup.py bdist_egg upload\")\n run(\"python3.2 setup.py bdist_egg upload\")\n local(\"python setup.py sdist upload\")", "response": "Upload sdist and bdist_eggs to pypi"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef add_tcp_callback(self, port, callback, threaded_callback=False):\n if not callback:\n raise AttributeError(\"No callback\")\n\n serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n serversocket.bind((_TCP_SOCKET_HOST, port))\n serversocket.listen(1)\n serversocket.setblocking(0)\n self._epoll.register(serversocket.fileno(), select.EPOLLIN)\n\n # Prepare the callback (wrap in Thread if needed)\n cb = callback if not threaded_callback else \\\n partial(_threaded_callback, callback)\n\n self._tcp_server_sockets[serversocket.fileno()] = (serversocket, cb)\n debug(\"Socket server started at port %s and callback added.\" % port)", "response": "Adds a unix socket server callback which will be invoked when values\n arrive from a connected socket client."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nadding a callback to be executed when the value of a gpio_id changes to the specified edge.", "response": "def add_interrupt_callback(self, gpio_id, callback, edge='both',\n pull_up_down=_GPIO.PUD_OFF, threaded_callback=False,\n debounce_timeout_ms=None):\n \"\"\"\n Add a callback to be executed when the value on 'gpio_id' changes to\n the edge specified via the 'edge' parameter (default='both').\n\n `pull_up_down` can be set to `RPIO.PUD_UP`, `RPIO.PUD_DOWN`, and\n `RPIO.PUD_OFF`.\n\n If `threaded_callback` is True, the callback will be started\n inside a Thread.\n \"\"\"\n gpio_id = _GPIO.channel_to_gpio(gpio_id)\n debug(\"Adding callback for GPIO %s\" % gpio_id)\n if not edge in [\"falling\", \"rising\", \"both\", \"none\"]:\n raise AttributeError(\"'%s' is not a valid edge.\" % edge)\n\n if not pull_up_down in [_GPIO.PUD_UP, _GPIO.PUD_DOWN, _GPIO.PUD_OFF]:\n raise AttributeError(\"'%s' is not a valid pull_up_down.\" % edge)\n\n # Make sure the gpio_id is valid\n if not gpio_id in set(chain(RPIO.GPIO_LIST_R1, RPIO.GPIO_LIST_R2, \\\n RPIO.GPIO_LIST_R3)):\n raise AttributeError(\"GPIO %s is not a valid gpio-id.\" % gpio_id)\n\n # Require INPUT pin setup; and set the correct PULL_UPDN\n if RPIO.gpio_function(int(gpio_id)) == RPIO.IN:\n RPIO.set_pullupdn(gpio_id, pull_up_down)\n else:\n debug(\"- changing gpio function from %s to INPUT\" % \\\n (GPIO_FUNCTIONS[RPIO.gpio_function(int(gpio_id))]))\n RPIO.setup(gpio_id, RPIO.IN, pull_up_down)\n\n # Prepare the callback (wrap in Thread if needed)\n cb = callback if not threaded_callback else \\\n partial(_threaded_callback, callback)\n\n # Prepare the /sys/class path of this gpio\n path_gpio = \"%sgpio%s/\" % (_SYS_GPIO_ROOT, gpio_id)\n\n # If initial callback for this GPIO then set everything up. Else make\n # sure the edge detection is the same.\n if gpio_id in self._map_gpioid_to_callbacks:\n with open(path_gpio + \"edge\", \"r\") as f:\n e = f.read().strip()\n if e != edge:\n raise AttributeError((\"Cannot add callback for gpio %s:\"\n \" edge detection '%s' not compatible with existing\"\n \" edge detection '%s'.\") % (gpio_id, edge, e))\n\n # Check whether edge is the same, else throw Exception\n debug(\"- kernel interface already setup for GPIO %s\" % gpio_id)\n self._map_gpioid_to_callbacks[gpio_id].append(cb)\n\n else:\n # If kernel interface already exists unexport first for clean setup\n if os.path.exists(path_gpio):\n if self._show_warnings:\n warn(\"Kernel interface for GPIO %s already exists.\" % \\\n gpio_id)\n debug(\"- unexporting kernel interface for GPIO %s\" % gpio_id)\n with open(_SYS_GPIO_ROOT + \"unexport\", \"w\") as f:\n f.write(\"%s\" % gpio_id)\n time.sleep(0.1)\n\n # Export kernel interface /sys/class/gpio/gpioN\n with open(_SYS_GPIO_ROOT + \"export\", \"w\") as f:\n f.write(\"%s\" % gpio_id)\n self._gpio_kernel_interfaces_created.append(gpio_id)\n debug(\"- kernel interface exported for GPIO %s\" % gpio_id)\n\n # Configure gpio as input\n with open(path_gpio + \"direction\", \"w\") as f:\n f.write(\"in\")\n\n # Configure gpio edge detection\n with open(path_gpio + \"edge\", \"w\") as f:\n f.write(edge)\n\n debug((\"- kernel interface configured for GPIO %s \"\n \"(edge='%s', pullupdn=%s)\") % (gpio_id, edge, \\\n _PULL_UPDN[pull_up_down]))\n\n # Open the gpio value stream and read the initial value\n f = open(path_gpio + \"value\", 'r')\n val_initial = f.read().strip()\n debug(\"- inital gpio value: %s\" % val_initial)\n f.seek(0)\n\n # Add callback info to the mapping dictionaries\n self._map_fileno_to_file[f.fileno()] = f\n self._map_fileno_to_gpioid[f.fileno()] = gpio_id\n self._map_fileno_to_options[f.fileno()] = {\n \"debounce_timeout_s\": debounce_timeout_ms / 1000.0 if \\\n debounce_timeout_ms else 0,\n \"interrupt_last\": 0,\n \"edge\": edge\n }\n self._map_gpioid_to_fileno[gpio_id] = f.fileno()\n self._map_gpioid_to_callbacks[gpio_id] = [cb]\n\n # Add to epoll\n self._epoll.register(f.fileno(), select.EPOLLPRI | select.EPOLLERR)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef del_interrupt_callback(self, gpio_id):\n debug(\"- removing interrupts on gpio %s\" % gpio_id)\n gpio_id = _GPIO.channel_to_gpio(gpio_id)\n fileno = self._map_gpioid_to_fileno[gpio_id]\n\n # 1. Remove from epoll\n self._epoll.unregister(fileno)\n\n # 2. Cache the file\n f = self._map_fileno_to_file[fileno]\n\n # 3. Remove from maps\n del self._map_fileno_to_file[fileno]\n del self._map_fileno_to_gpioid[fileno]\n del self._map_fileno_to_options[fileno]\n del self._map_gpioid_to_fileno[gpio_id]\n del self._map_gpioid_to_callbacks[gpio_id]\n\n # 4. Close file last in case of IOError\n f.close()", "response": "Delete all interrupt callbacks from a certain gpio"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _handle_interrupt(self, fileno, val):\n val = int(val)\n\n # Filter invalid edge values (sometimes 1 comes in when edge=falling)\n edge = self._map_fileno_to_options[fileno][\"edge\"]\n if (edge == 'rising' and val == 0) or (edge == 'falling' and val == 1):\n return\n\n # If user activated debounce for this callback, check timing now\n debounce = self._map_fileno_to_options[fileno][\"debounce_timeout_s\"]\n if debounce:\n t = time.time()\n t_last = self._map_fileno_to_options[fileno][\"interrupt_last\"]\n if t - t_last < debounce:\n debug(\"- don't start interrupt callback due to debouncing\")\n return\n self._map_fileno_to_options[fileno][\"interrupt_last\"] = t\n\n # Start the callback(s) now\n gpio_id = self._map_fileno_to_gpioid[fileno]\n if gpio_id in self._map_gpioid_to_callbacks:\n for cb in self._map_gpioid_to_callbacks[gpio_id]:\n cb(gpio_id, val)", "response": "Handles interrupt events from the specified fileno."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef wait_for_interrupts(self, epoll_timeout=1):\n self._is_waiting_for_interrupts = True\n while self._is_waiting_for_interrupts:\n events = self._epoll.poll(epoll_timeout)\n for fileno, event in events:\n debug(\"- epoll event on fd %s: %s\" % (fileno, event))\n if fileno in self._tcp_server_sockets:\n # New client connection to socket server\n serversocket, cb = self._tcp_server_sockets[fileno]\n connection, address = serversocket.accept()\n connection.setblocking(0)\n f = connection.fileno()\n self._epoll.register(f, select.EPOLLIN)\n self._tcp_client_sockets[f] = (connection, cb)\n\n elif event & select.EPOLLIN:\n # Input from TCP socket\n socket, cb = self._tcp_client_sockets[fileno]\n content = socket.recv(1024)\n if not content or not content.strip():\n # No content means quitting\n self.close_tcp_client(fileno)\n else:\n sock, cb = self._tcp_client_sockets[fileno]\n cb(self._tcp_client_sockets[fileno][0], \\\n content.strip())\n\n elif event & select.EPOLLHUP:\n # TCP Socket Hangup\n self.close_tcp_client(fileno)\n\n elif event & select.EPOLLPRI:\n # GPIO interrupts\n f = self._map_fileno_to_file[fileno]\n # read() is workaround for not getting new values\n # with read(1)\n val = f.read().strip()\n f.seek(0)\n self._handle_interrupt(fileno, val)", "response": "Wait for interrupts and distribute them to the interrupt related callbacks."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nclean up the list of created GPIO interfaces that this script created.", "response": "def cleanup_interfaces(self):\n \"\"\"\n Removes all /sys/class/gpio/gpioN interfaces that this script created,\n and deletes callback bindings. Should be used after using interrupts.\n \"\"\"\n debug(\"Cleaning up interfaces...\")\n for gpio_id in self._gpio_kernel_interfaces_created:\n # Close the value-file and remove interrupt bindings\n self.del_interrupt_callback(gpio_id)\n\n # Remove the kernel GPIO interface\n debug(\"- unexporting GPIO %s\" % gpio_id)\n with open(_SYS_GPIO_ROOT + \"unexport\", \"w\") as f:\n f.write(\"%s\" % gpio_id)\n\n # Reset list of created interfaces\n self._gpio_kernel_interfaces_created = []"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nclose all TCP connections and then the socket servers.", "response": "def cleanup_tcpsockets(self):\n \"\"\"\n Closes all TCP connections and then the socket servers\n \"\"\"\n for fileno in self._tcp_client_sockets.keys():\n self.close_tcp_client(fileno)\n for fileno, items in self._tcp_server_sockets.items():\n socket, cb = items\n debug(\"- _cleanup server socket connection (fd %s)\" % fileno)\n self._epoll.unregister(fileno)\n socket.close()\n self._tcp_server_sockets = {}"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nadding a pulse for a specific GPIO to a dma channel subcycle.", "response": "def add_channel_pulse(dma_channel, gpio, start, width):\n \"\"\"\n Add a pulse for a specific GPIO to a dma channel subcycle. `start` and\n `width` are multiples of the pulse-width increment granularity.\n \"\"\"\n return _PWM.add_channel_pulse(dma_channel, gpio, start, width)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set_servo(self, gpio, pulse_width_us):\n # Make sure we can set the exact pulse_width_us\n _pulse_incr_us = _PWM.get_pulse_incr_us()\n if pulse_width_us % _pulse_incr_us:\n # No clean division possible\n raise AttributeError((\"Pulse width increment granularity %sus \"\n \"cannot divide a pulse-time of %sus\") % (_pulse_incr_us,\n pulse_width_us))\n\n # Initialize channel if not already done, else check subcycle time\n if _PWM.is_channel_initialized(self._dma_channel):\n _subcycle_us = _PWM.get_channel_subcycle_time_us(self._dma_channel)\n if _subcycle_us != self._subcycle_time_us:\n raise AttributeError((\"Error: DMA channel %s is setup with a \"\n \"subcycle_time of %sus (instead of %sus)\") % \\\n (self._dma_channel, _subcycle_us, \n self._subcycle_time_us))\n else:\n init_channel(self._dma_channel, self._subcycle_time_us)\n\n # Add pulse for this GPIO\n add_channel_pulse(self._dma_channel, gpio, 0, \\\n int(pulse_width_us / _pulse_incr_us))", "response": "Sets a pulse - width on a gpio to repeat every subcycle - time."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef unique(_list):\n ret = []\n\n for item in _list:\n if item not in ret:\n ret.append(item)\n\n return ret", "response": "Returns a list of unique items in the order of the items in the list"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nperforming initial query cleanup", "response": "def preprocess_query(query):\n \"\"\"\n Perform initial query cleanup\n\n :type query str\n :rtype str\n \"\"\"\n # 1. remove aliases\n # FROM `dimension_wikis` `dw`\n # INNER JOIN `fact_wam_scores` `fwN`\n query = re.sub(r'(\\s(FROM|JOIN)\\s`[^`]+`)\\s`[^`]+`', r'\\1', query, flags=re.IGNORECASE)\n\n # 2. `database`.`table` notation -> database.table\n query = re.sub(r'`([^`]+)`\\.`([^`]+)`', r'\\1.\\2', query)\n\n # 2. database.table notation -> table\n # query = re.sub(r'([a-z_0-9]+)\\.([a-z_0-9]+)', r'\\2', query, flags=re.IGNORECASE)\n\n return query"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_query_tokens(query):\n query = preprocess_query(query)\n parsed = sqlparse.parse(query)\n\n # handle empty queries (#12)\n if not parsed:\n return []\n\n tokens = TokenList(parsed[0].tokens).flatten()\n # print([(token.value, token.ttype) for token in tokens])\n\n return [token for token in tokens if token.ttype is not Whitespace]", "response": "returns a list of SQL Token objects from a query string"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_query_columns(query):\n columns = []\n last_keyword = None\n last_token = None\n\n # print(preprocess_query(query))\n\n # these keywords should not change the state of a parser\n # and not \"reset\" previously found SELECT keyword\n keywords_ignored = ['AS', 'AND', 'OR', 'IN', 'IS', 'NOT', 'NOT NULL', 'LIKE', 'CASE', 'WHEN']\n\n # these function should be ignored\n # and not \"reset\" previously found SELECT keyword\n functions_ignored = ['COUNT', 'MIN', 'MAX', 'FROM_UNIXTIME', 'DATE_FORMAT', 'CAST', 'CONVERT']\n\n for token in get_query_tokens(query):\n if token.is_keyword and token.value.upper() not in keywords_ignored:\n # keep the name of the last keyword, e.g. SELECT, FROM, WHERE, (ORDER) BY\n last_keyword = token.value.upper()\n # print('keyword', last_keyword)\n elif token.ttype is Name:\n # analyze the name tokens, column names and where condition values\n if last_keyword in ['SELECT', 'WHERE', 'BY', 'ON'] \\\n and last_token.value.upper() not in ['AS']:\n # print(last_keyword, last_token, token.value)\n\n if token.value.upper() not in functions_ignored:\n if str(last_token) == '.':\n # print('DOT', last_token, columns[-1])\n\n # we have table.column notation example\n # append column name to the last entry of columns\n # as it is a table name in fact\n table_name = columns[-1]\n columns[-1] = '{}.{}'.format(table_name, token)\n else:\n columns.append(str(token.value))\n elif last_keyword in ['INTO'] and last_token.ttype is Punctuation:\n # INSERT INTO `foo` (col1, `col2`) VALUES (..)\n # print(last_keyword, token, last_token)\n columns.append(str(token.value).strip('`'))\n elif token.ttype is Wildcard:\n # handle * wildcard in SELECT part, but ignore count(*)\n # print(last_keyword, last_token, token.value)\n if last_keyword == 'SELECT' and last_token.value != '(':\n\n if str(last_token) == '.':\n # handle SELECT foo.*\n table_name = columns[-1]\n columns[-1] = '{}.{}'.format(table_name, str(token))\n else:\n columns.append(str(token.value))\n\n last_token = token\n\n return unique(columns)", "response": "returns a list of column names for the given query"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_query_tables(query):\n tables = []\n last_keyword = None\n last_token = None\n\n table_syntax_keywords = [\n # SELECT queries\n 'FROM', 'WHERE', 'JOIN', 'INNER JOIN', 'LEFT JOIN', 'RIGHT JOIN', 'ON',\n # INSERT queries\n 'INTO', 'VALUES',\n # UPDATE queries\n 'UPDATE', 'SET',\n # Hive queries\n 'TABLE', # INSERT TABLE\n ]\n\n # print(query, get_query_tokens(query))\n\n for token in get_query_tokens(query):\n # print([token, token.ttype, last_token, last_keyword])\n if token.is_keyword and token.value.upper() in table_syntax_keywords:\n # keep the name of the last keyword, the next one can be a table name\n last_keyword = token.value.upper()\n # print('keyword', last_keyword)\n elif str(token) == '(':\n # reset the last_keyword for INSERT `foo` VALUES(id, bar) ...\n last_keyword = None\n elif token.is_keyword and str(token) in ['FORCE', 'ORDER']:\n # reset the last_keyword for \"SELECT x FORCE INDEX\" queries and \"SELECT x ORDER BY\"\n last_keyword = None\n elif token.is_keyword and str(token) == 'SELECT' and last_keyword in ['INTO', 'TABLE']:\n # reset the last_keyword for \"INSERT INTO SELECT\" and \"INSERT TABLE SELECT\" queries\n last_keyword = None\n elif token.ttype is Name or token.is_keyword:\n # print([last_keyword, last_token, token.value])\n # analyze the name tokens, column names and where condition values\n if last_keyword in ['FROM', 'JOIN', 'INNER JOIN', 'LEFT JOIN', 'RIGHT JOIN',\n 'INTO', 'UPDATE', 'TABLE'] \\\n and last_token not in ['AS'] \\\n and token.value not in ['AS', 'SELECT']:\n\n if last_token == '.':\n # we have database.table notation example\n # append table name to the last entry of tables\n # as it is a database name in fact\n database_name = tables[-1]\n tables[-1] = '{}.{}'.format(database_name, token)\n last_keyword = None\n elif last_token not in [',', last_keyword]:\n # it's not a list of tables, e.g. SELECT * FROM foo, bar\n # hence, it can be the case of alias without AS, e.g. SELECT * FROM foo bar\n pass\n else:\n table_name = str(token.value.strip('`'))\n tables.append(table_name)\n\n last_token = token.value.upper()\n\n return unique(tables)", "response": "returns a list of tables from a query string"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_query_limit_and_offset(query):\n limit = None\n offset = None\n last_keyword = None\n last_token = None\n\n # print(query)\n for token in get_query_tokens(query):\n # print([token, token.ttype, last_keyword])\n\n if token.is_keyword and token.value.upper() in ['LIMIT', 'OFFSET']:\n last_keyword = token.value.upper()\n elif token.ttype is Number.Integer:\n # print([token, last_keyword, last_token_was_integer])\n if last_keyword == 'LIMIT':\n # LIMIT \n limit = int(token.value)\n last_keyword = None\n elif last_keyword == 'OFFSET':\n # OFFSET \n offset = int(token.value)\n last_keyword = None\n elif last_token and last_token.ttype is Punctuation:\n # LIMIT ,\n offset = limit\n limit = int(token.value)\n\n last_token = token\n\n if limit is None:\n return None\n\n return limit, offset or 0", "response": "returns limit and offset from a query string"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef normalize_likes(sql):\n sql = sql.replace('%', '')\n\n # LIKE '%bot'\n sql = re.sub(r\"LIKE '[^\\']+'\", 'LIKE X', sql)\n\n # or all_groups LIKE X or all_groups LIKE X\n matches = re.finditer(r'(or|and) [^\\s]+ LIKE X', sql, flags=re.IGNORECASE)\n matches = [match.group(0) for match in matches] if matches else None\n\n if matches:\n for match in set(matches):\n sql = re.sub(r'(\\s?' + re.escape(match) + ')+', ' ' + match + ' ...', sql)\n\n return sql", "response": "Normalize and wrap LIKE statements\n "} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngeneralizes a SQL query into a single segment of the SQL query.", "response": "def generalize_sql(sql):\n \"\"\"\n Removes most variables from an SQL query and replaces them with X or N for numbers.\n\n Based on Mediawiki's DatabaseBase::generalizeSQL\n\n :type sql str|None\n :rtype: str\n \"\"\"\n if sql is None:\n return None\n\n # multiple spaces\n sql = re.sub(r'\\s{2,}', ' ', sql)\n\n # MW comments\n # e.g. /* CategoryDataService::getMostVisited N.N.N.N */\n sql = remove_comments_from_sql(sql)\n\n # handle LIKE statements\n sql = normalize_likes(sql)\n\n sql = re.sub(r\"\\\\\\\\\", '', sql)\n sql = re.sub(r\"\\\\'\", '', sql)\n sql = re.sub(r'\\\\\"', '', sql)\n sql = re.sub(r\"'[^\\']*'\", 'X', sql)\n sql = re.sub(r'\"[^\\\"]*\"', 'X', sql)\n\n # All newlines, tabs, etc replaced by single space\n sql = re.sub(r'\\s+', ' ', sql)\n\n # All numbers => N\n sql = re.sub(r'-?[0-9]+', 'N', sql)\n\n # WHERE foo IN ('880987','882618','708228','522330')\n sql = re.sub(r' (IN|VALUES)\\s*\\([^,]+,[^)]+\\)', ' \\\\1 (XYZ)', sql, flags=re.IGNORECASE)\n\n return sql.strip()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef deblur_system_call(params, input_fp):\n logger = logging.getLogger(__name__)\n logger.debug('[%s] deblur system call params %s, input_fp %s' %\n (mp.current_process().name, params, input_fp))\n\n # construct command\n script_name = \"deblur\"\n script_subprogram = \"workflow\"\n command = [script_name,\n script_subprogram,\n '--seqs-fp', input_fp,\n '--is-worker-thread',\n '--keep-tmp-files']\n command.extend(params)\n\n logger.debug('[%s] running command %s' % (mp.current_process().name,\n command))\n return _system_call(command)", "response": "Build deblur command for subprocess."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef run_functor(functor, *args, **kwargs):\n try:\n # This is where you do your actual work\n return functor(*args, **kwargs)\n except Exception:\n # Put all exception text into an exception and raise that\n raise Exception(\"\".join(traceback.format_exception(*sys.exc_info())))", "response": "Given a functor run it and return its result."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef parallel_deblur(inputs, params,\n pos_ref_db_fp, neg_ref_dp_fp, jobs_to_start=1):\n \"\"\"Dispatch execution over a pool of processors\n\n This code was adopted from the American Gut project:\n https://github.com/biocore/American-Gut/blob/master/americangut/parallel.py\n\n Parameters\n ----------\n inputs : iterable of str\n File paths to input per-sample sequence files\n params : list of str\n list of CLI parameters supplied to the deblur workflow\n (argv - first 2 are 'deblur','workflow' and are ignored)\n pos_ref_db_fp : list of str\n the indexed positive (16s) sortmerna database\n (created in the main thread)\n neg_ref_db_fp : list of str\n the indexed negative (artifacts) sortmerna database\n (created in the main thread)\n jobs_to_start : int, optional\n The number of processors on the local system to use\n\n Returns\n -------\n all_result_paths : list\n list of expected output files\n \"\"\"\n logger = logging.getLogger(__name__)\n logger.info('parallel deblur started for %d inputs' % len(inputs))\n\n # remove the irrelevant parameters\n remove_param_list = ['-O', '--jobs-to-start', '--seqs-fp',\n '--pos-ref-db-fp', '--neg-ref-db-fp']\n skipnext = False\n newparams = []\n for carg in params[2:]:\n if skipnext:\n skipnext = False\n continue\n if carg in remove_param_list:\n skipnext = True\n continue\n newparams.append(carg)\n\n # add the ref_db_fp (since it may be not present in the\n # original command parameters)\n if pos_ref_db_fp:\n new_pos_ref_db_fp = ','.join(pos_ref_db_fp)\n newparams.append('--pos-ref-db-fp')\n newparams.append(new_pos_ref_db_fp)\n if neg_ref_dp_fp:\n new_neg_ref_db_fp = ','.join(neg_ref_dp_fp)\n newparams.append('--neg-ref-db-fp')\n newparams.append(new_neg_ref_db_fp)\n\n logger.debug('ready for functor %s' % newparams)\n functor = partial(run_functor, deblur_system_call, newparams)\n logger.debug('ready for pool %d jobs' % jobs_to_start)\n pool = mp.Pool(processes=jobs_to_start)\n logger.debug('almost running...')\n for stdout, stderr, es in pool.map(functor, inputs):\n if es != 0:\n raise RuntimeError(\"stdout: %s\\nstderr: %s\\nexit: %d\" % (stdout,\n stderr,\n es))", "response": "Dispatch execution over a pool of processors"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nyielding (id, sequence) from an input file Parameters ---------- input_fp : filepath A filepath, which can be any valid fasta or fastq file within the limitations of scikit-bio's IO registry. Notes ----- The use of this method is a stopgap to replicate the existing `parse_fasta` functionality while at the same time allowing for fastq support. Raises ------ skbio.io.FormatIdentificationWarning If the format of the input file cannot be determined. Returns ------- (str, str) The ID and sequence.", "response": "def sequence_generator(input_fp):\n \"\"\"Yield (id, sequence) from an input file\n\n Parameters\n ----------\n input_fp : filepath\n A filepath, which can be any valid fasta or fastq file within the\n limitations of scikit-bio's IO registry.\n\n Notes\n -----\n The use of this method is a stopgap to replicate the existing `parse_fasta`\n functionality while at the same time allowing for fastq support.\n\n Raises\n ------\n skbio.io.FormatIdentificationWarning\n If the format of the input file cannot be determined.\n\n Returns\n -------\n (str, str)\n The ID and sequence.\n\n \"\"\"\n logger = logging.getLogger(__name__)\n kw = {}\n if sniff_fasta(input_fp)[0]:\n format = 'fasta'\n elif sniff_fastq(input_fp)[0]:\n format = 'fastq'\n\n kw['variant'] = _get_fastq_variant(input_fp)\n else:\n # usually happens when the fasta file is empty\n # so need to return no sequences (and warn)\n msg = \"input file %s does not appear to be FASTA or FASTQ\" % input_fp\n logger.warn(msg)\n warnings.warn(msg, UserWarning)\n return\n\n # some of the test code is using file paths, some is using StringIO.\n if isinstance(input_fp, io.TextIOBase):\n input_fp.seek(0)\n\n for record in skbio.read(input_fp, format=format, **kw):\n yield (record.metadata['id'], str(record))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef trim_seqs(input_seqs, trim_len, left_trim_len):\n # counters for the number of trimmed and total sequences\n logger = logging.getLogger(__name__)\n\n okseqs = 0\n totseqs = 0\n\n if trim_len < -1:\n raise ValueError(\"Invalid trim_len: %d\" % trim_len)\n\n for label, seq in input_seqs:\n totseqs += 1\n\n if trim_len == -1:\n okseqs += 1\n yield label, seq\n elif len(seq) >= trim_len:\n okseqs += 1\n yield label, seq[left_trim_len:trim_len]\n\n if okseqs < 0.01*totseqs:\n logger = logging.getLogger(__name__)\n errmsg = 'Vast majority of sequences (%d / %d) are shorter ' \\\n 'than the trim length (%d). ' \\\n 'Are you using the correct -t trim length?' \\\n % (totseqs-okseqs, totseqs, trim_len)\n logger.warn(errmsg)\n warnings.warn(errmsg, UserWarning)\n else:\n logger.debug('trimmed to length %d (%d / %d remaining)'\n % (trim_len, okseqs, totseqs))", "response": "Trim FASTA sequences to specified length."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nbuild a SortMeRNA index for all reference databases.", "response": "def build_index_sortmerna(ref_fp, working_dir):\n \"\"\"Build a SortMeRNA index for all reference databases.\n\n Parameters\n ----------\n ref_fp: tuple\n filepaths to FASTA reference databases\n working_dir: string\n working directory path where to store the indexed database\n\n Returns\n -------\n all_db: tuple\n filepaths to SortMeRNA indexed reference databases\n \"\"\"\n logger = logging.getLogger(__name__)\n logger.info('build_index_sortmerna files %s to'\n ' dir %s' % (ref_fp, working_dir))\n all_db = []\n for db in ref_fp:\n fasta_dir, fasta_filename = split(db)\n index_basename = splitext(fasta_filename)[0]\n db_output = join(working_dir, index_basename)\n logger.debug('processing file %s into location %s' % (db, db_output))\n params = ['indexdb_rna', '--ref', '%s,%s' %\n (db, db_output), '--tmpdir', working_dir]\n sout, serr, res = _system_call(params)\n if not res == 0:\n logger.error('Problem running indexdb_rna on file %s to dir %s. '\n 'database not indexed' % (db, db_output))\n logger.debug('stdout: %s' % sout)\n logger.debug('stderr: %s' % serr)\n logger.critical('execution halted')\n raise RuntimeError('Cannot index database file %s' % db)\n logger.debug('file %s indexed' % db)\n all_db.append(db_output)\n return all_db"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfilters samples from a biom table that have less than minreads reads", "response": "def filter_minreads_samples_from_table(table, minreads=1, inplace=True):\n \"\"\"Filter samples from biom table that have less than\n minreads reads total\n\n Paraneters\n ----------\n table : biom.Table\n the biom table to filter\n minreads : int (optional)\n the minimal number of reads in a sample in order to keep it\n inplace : bool (optional)\n if True, filter the biom table in place, if false create a new copy\n\n Returns\n -------\n table : biom.Table\n the filtered biom table\n \"\"\"\n logger = logging.getLogger(__name__)\n logger.debug('filter_minreads_started. minreads=%d' % minreads)\n samp_sum = table.sum(axis='sample')\n samp_ids = table.ids(axis='sample')\n bad_samples = samp_ids[samp_sum < minreads]\n if len(bad_samples) > 0:\n logger.warn('removed %d samples with reads per sample<%d'\n % (len(bad_samples), minreads))\n table = table.filter(bad_samples, axis='sample',\n inplace=inplace, invert=True)\n else:\n logger.debug('all samples contain > %d reads' % minreads)\n return table"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef fasta_from_biom(table, fasta_file_name):\n '''Save sequences from a biom table to a fasta file\n\n Parameters\n ----------\n table : biom.Table\n The biom table containing the sequences\n fasta_file_name : str\n Name of the fasta output file\n '''\n logger = logging.getLogger(__name__)\n logger.debug('saving biom table sequences to fasta file %s' % fasta_file_name)\n\n with open(fasta_file_name, 'w') as f:\n for cseq in table.ids(axis='observation'):\n f.write('>%s\\n%s\\n' % (cseq, cseq))\n logger.info('saved biom table sequences to fasta file %s' % fasta_file_name)", "response": "Save sequences from a biom table to a fasta file"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef remove_artifacts_from_biom_table(table_filename,\n fasta_filename,\n ref_fp,\n biom_table_dir,\n ref_db_fp,\n threads=1,\n verbose=False,\n sim_thresh=None,\n coverage_thresh=None):\n \"\"\"Remove artifacts from a biom table using SortMeRNA\n\n Parameters\n ----------\n table : str\n name of the biom table file\n fasta_filename : str\n the fasta file containing all the sequences of the biom table\n\n Returns\n -------\n tmp_files : list of str\n The temp files created during the artifact removal step\n \"\"\"\n logger = logging.getLogger(__name__)\n logger.info('getting 16s sequences from the biom table')\n\n # remove artifacts from the fasta file. output is in clean_fp fasta file\n clean_fp, num_seqs_left, tmp_files = remove_artifacts_seqs(fasta_filename, ref_fp,\n working_dir=biom_table_dir,\n ref_db_fp=ref_db_fp,\n negate=False, threads=threads,\n verbose=verbose,\n sim_thresh=sim_thresh,\n coverage_thresh=coverage_thresh)\n if clean_fp is None:\n logger.warn(\"No clean sequences in %s\" % fasta_filename)\n return tmp_files\n\n logger.debug('removed artifacts from sequences input %s'\n ' to output %s' % (fasta_filename, clean_fp))\n\n # read the clean fasta file\n good_seqs = {s for _, s in sequence_generator(clean_fp)}\n logger.debug('loaded %d sequences from cleaned biom table'\n ' fasta file' % len(good_seqs))\n\n logger.debug('loading biom table %s' % table_filename)\n table = load_table(table_filename)\n\n # filter and save the artifact biom table\n artifact_table = table.filter(list(good_seqs),\n axis='observation', inplace=False,\n invert=True)\n # remove the samples with 0 reads\n filter_minreads_samples_from_table(artifact_table)\n output_nomatch_fp = join(biom_table_dir, 'reference-non-hit.biom')\n write_biom_table(artifact_table, output_nomatch_fp)\n logger.info('wrote artifact only filtered biom table to %s'\n % output_nomatch_fp)\n # and save the reference-non-hit fasta file\n output_nomatch_fasta_fp = join(biom_table_dir, 'reference-non-hit.seqs.fa')\n fasta_from_biom(artifact_table, output_nomatch_fasta_fp)\n\n # filter and save the only 16s biom table\n table.filter(list(good_seqs), axis='observation')\n # remove the samples with 0 reads\n filter_minreads_samples_from_table(table)\n output_fp = join(biom_table_dir, 'reference-hit.biom')\n write_biom_table(table, output_fp)\n logger.info('wrote 16s filtered biom table to %s' % output_fp)\n # and save the reference-non-hit fasta file\n output_match_fasta_fp = join(biom_table_dir, 'reference-hit.seqs.fa')\n fasta_from_biom(table, output_match_fasta_fp)\n\n # we also don't need the cleaned fasta file\n tmp_files.append(clean_fp)\n return tmp_files", "response": "Remove artifacts from a biom table using SortMeRNA\n "} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nremoves artifacts from a FASTA file using SortMeRNA.", "response": "def remove_artifacts_seqs(seqs_fp,\n ref_fp,\n working_dir,\n ref_db_fp,\n negate=False,\n threads=1,\n verbose=False,\n sim_thresh=None,\n coverage_thresh=None):\n \"\"\"Remove artifacts from FASTA file using SortMeRNA.\n\n Parameters\n ----------\n seqs_fp: string\n file path to FASTA input sequence file\n ref_fp: tuple\n file path(s) to FASTA database file\n working_dir: string\n working directory path\n ref_db_fp: tuple\n file path(s) to indexed FASTA database\n negate: boolean, optional\n if True, discard all input sequences aligning\n to reference database\n threads: integer, optional\n number of threads to use for SortMeRNA\n verbose: boolean, optional\n If true, output SortMeRNA errors\n sim_thresh: float, optional\n The minimal similarity threshold (between 0 and 1)\n for keeping the sequence\n if None, the default values used are 0.65 for negate=False,\n 0.95 for negate=True\n coverage_thresh: float, optional\n The minimal coverage threshold (between 0 and 1)\n for alignments for keeping the sequence\n if None, the default values used are 0.5 for negate=False,\n 0.95 for negate=True\n\n Returns\n -------\n output_fp : str\n Name of the artifact removed fasta file\n okseqs : int\n The number of sequences left after artifact removal\n tmp_files : list of str\n Names of the tmp files created\n \"\"\"\n logger = logging.getLogger(__name__)\n logger.info('remove_artifacts_seqs file %s' % seqs_fp)\n\n if stat(seqs_fp).st_size == 0:\n logger.warn('file %s has size 0, continuing' % seqs_fp)\n return None, 0, []\n\n if coverage_thresh is None:\n if negate:\n coverage_thresh = 0.95 * 100\n else:\n coverage_thresh = 0.5 * 100\n\n if sim_thresh is None:\n if negate:\n sim_thresh = 0.95 * 100\n else:\n sim_thresh = 0.65 * 100\n\n # the minimal average bitscore per nucleotide\n bitscore_thresh = 0.65\n\n output_fp = join(working_dir,\n \"%s.no_artifacts\" % basename(seqs_fp))\n blast_output = join(working_dir,\n '%s.sortmerna' % basename(seqs_fp))\n aligned_seq_ids = set()\n for i, db in enumerate(ref_fp):\n logger.debug('running on ref_fp %s working dir %s refdb_fp %s seqs %s'\n % (db, working_dir, ref_db_fp[i], seqs_fp))\n # run SortMeRNA\n # we use -e 100 to remove E-value based filtering by sortmerna\n # since we use bitscore/identity/coverage filtering instead\n params = ['sortmerna', '--reads', seqs_fp, '--ref', '%s,%s' %\n (db, ref_db_fp[i]),\n '--aligned', blast_output, '--blast', '3', '--best', '1',\n '--print_all_reads', '-v', '-e', '100']\n\n sout, serr, res = _system_call(params)\n if not res == 0:\n logger.error('sortmerna error on file %s' % seqs_fp)\n logger.error('stdout : %s' % sout)\n logger.error('stderr : %s' % serr)\n return output_fp, 0, []\n\n blast_output_filename = '%s.blast' % blast_output\n with open(blast_output_filename, 'r') as bfl:\n for line in bfl:\n line = line.strip().split('\\t')\n # if * means no match\n if line[1] == '*':\n continue\n # check if % identity[2] and coverage[13] are large enough\n if (float(line[2]) >= sim_thresh) and \\\n (float(line[13]) >= coverage_thresh) and \\\n (float(line[11]) >= bitscore_thresh * len(line[0])):\n aligned_seq_ids.add(line[0])\n\n if negate:\n def op(x): return x not in aligned_seq_ids\n else:\n def op(x): return x in aligned_seq_ids\n\n # if negate = False, only output sequences\n # matching to at least one of the databases\n totalseqs = 0\n okseqs = 0\n badseqs = 0\n with open(output_fp, 'w') as out_f:\n for label, seq in sequence_generator(seqs_fp):\n totalseqs += 1\n label = label.split()[0]\n if op(label):\n out_f.write(\">%s\\n%s\\n\" % (label, seq))\n okseqs += 1\n else:\n badseqs += 1\n logger.info('total sequences %d, passing sequences %d, '\n 'failing sequences %d' % (totalseqs, okseqs, badseqs))\n return output_fp, okseqs, [blast_output_filename]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef multiple_sequence_alignment(seqs_fp, threads=1):\n logger = logging.getLogger(__name__)\n logger.info('multiple_sequence_alignment seqs file %s' % seqs_fp)\n\n # for mafft we use -1 to denote all threads and not 0\n if threads == 0:\n threads = -1\n\n if stat(seqs_fp).st_size == 0:\n logger.warning('msa failed. file %s has no reads' % seqs_fp)\n return None\n msa_fp = seqs_fp + '.msa'\n params = ['mafft', '--quiet', '--preservecase', '--parttree', '--auto',\n '--thread', str(threads), seqs_fp]\n sout, serr, res = _system_call(params, stdoutfilename=msa_fp)\n if not res == 0:\n logger.info('msa failed for file %s (maybe only 1 read?)' % seqs_fp)\n logger.debug('stderr : %s' % serr)\n return None\n return msa_fp", "response": "Perform multiple sequence alignment on a FASTA file using MAFFT."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef remove_chimeras_denovo_from_seqs(seqs_fp, working_dir, threads=1):\n logger = logging.getLogger(__name__)\n logger.info('remove_chimeras_denovo_from_seqs seqs file %s'\n 'to working dir %s' % (seqs_fp, working_dir))\n\n output_fp = join(\n working_dir, \"%s.no_chimeras\" % basename(seqs_fp))\n\n # we use the parameters dn=0.000001, xn=1000, minh=10000000\n # so 1 mismatch in the A/B region will cancel it being labeled as chimera\n # and ~3 unique reads in each region will make it a chimera if\n # no mismatches\n params = ['vsearch', '--uchime_denovo', seqs_fp,\n '--nonchimeras', output_fp,\n '-dn', '0.000001', '-xn', '1000',\n '-minh', '10000000', '--mindiffs', '5',\n '--fasta_width', '0', '--threads', str(threads)]\n sout, serr, res = _system_call(params)\n if not res == 0:\n logger.error('problem with chimera removal for file %s' % seqs_fp)\n logger.debug('stdout : %s' % sout)\n logger.debug('stderr : %s' % serr)\n return output_fp", "response": "Remove chimeras de novo from a set of sequences."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsplit FASTA file on sample IDs and store them in a dict of dicts.", "response": "def split_sequence_file_on_sample_ids_to_files(seqs,\n outdir):\n \"\"\"Split FASTA file on sample IDs.\n\n Parameters\n ----------\n seqs: file handler\n file handler to demultiplexed FASTA file\n outdir: string\n dirpath to output split FASTA files\n \"\"\"\n logger = logging.getLogger(__name__)\n logger.info('split_sequence_file_on_sample_ids_to_files'\n ' for file %s into dir %s' % (seqs, outdir))\n\n outputs = {}\n\n for bits in sequence_generator(seqs):\n sample = sample_id_from_read_id(bits[0])\n\n if sample not in outputs:\n outputs[sample] = open(join(outdir, sample + '.fasta'), 'w')\n outputs[sample].write(\">%s\\n%s\\n\" % (bits[0], bits[1]))\n for sample in outputs:\n outputs[sample].close()\n logger.info('split to %d files' % len(outputs))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nwrites a BIOM table to file.", "response": "def write_biom_table(table, biom_fp):\n \"\"\"Write BIOM table to file.\n\n Parameters\n ----------\n table: biom.table\n an instance of a BIOM table\n biom_fp: string\n filepath to output BIOM table\n \"\"\"\n logger = logging.getLogger(__name__)\n logger.debug('write_biom_table to file %s' % biom_fp)\n with biom_open(biom_fp, 'w') as f:\n table.to_hdf5(h5grp=f, generated_by=\"deblur\")\n logger.debug('wrote to BIOM file %s' % biom_fp)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_files_for_table(input_dir,\n file_end='.trim.derep.no_artifacts'\n '.msa.deblur.no_chimeras'):\n \"\"\"Get a list of files to add to the output table\n\n Parameters:\n -----------\n input_dir : string\n name of the directory containing the deblurred fasta files\n file_end : string\n the ending of all the fasta files to be added to the table\n (default '.fasta.trim.derep.no_artifacts.msa.deblur.no_chimeras')\n\n Returns\n -------\n names : list of tuples of (string,string)\n list of tuples of:\n name of fasta files to be added to the biom table\n sampleid (file names without the file_end and path)\n \"\"\"\n logger = logging.getLogger(__name__)\n logger.debug('get_files_for_table input dir %s, '\n 'file-ending %s' % (input_dir, file_end))\n names = []\n for cfile in glob(join(input_dir, \"*%s\" % file_end)):\n if not isfile(cfile):\n continue\n sample_id = basename(cfile)[:-len(file_end)]\n sample_id = os.path.splitext(sample_id)[0]\n names.append((cfile, sample_id))\n\n logger.debug('found %d files' % len(names))\n return names", "response": "Get a list of files to add to the biom table"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a BIOM table out of all samples in a directory and a list of deblurred sample ids and a list of sequences in a directory.", "response": "def create_otu_table(output_fp, deblurred_list,\n outputfasta_fp=None, minreads=0):\n \"\"\"Create a biom table out of all files in a directory\n\n Parameters\n ----------\n output_fp : string\n filepath to output BIOM table\n deblurred_list : list of (str, str)\n list of file names (including path), sampleid of all deblurred\n fasta files to add to the table\n outputfasta_fp : str, optional\n name of output fasta file (of all sequences in the table) or None\n to not write\n minreads : int, optional\n minimal number of reads per bacterial sequence in order to write\n it to the biom table and fasta file or 0 to write all\n \"\"\"\n logger = logging.getLogger(__name__)\n logger.info('create_otu_table for %d samples, '\n 'into output table %s' % (len(deblurred_list), output_fp))\n\n # the regexp for finding the number of reads of a sequence\n sizeregexp = re.compile('(?<=size=)\\w+')\n seqdict = {}\n seqlist = []\n sampset = set()\n samplist = []\n # arbitrary size for the sparse results matrix so we won't run out of space\n obs = scipy.sparse.dok_matrix((int(1E9), len(deblurred_list)), dtype=np.double)\n\n # load the sequences from all samples into a sprase matrix\n sneaking_extensions = {'fasta', 'fastq', 'fna', 'fq', 'fa'}\n for (cfilename, csampleid) in deblurred_list:\n if csampleid.rsplit('.', 1)[-1] in sneaking_extensions:\n csampleid = csampleid.rsplit('.', 1)[0]\n\n # test if sample has already been processed\n if csampleid in sampset:\n warnings.warn('sample %s already in table!', UserWarning)\n logger.error('sample %s already in table!' % csampleid)\n continue\n sampset.add(csampleid)\n samplist.append(csampleid)\n csampidx = len(sampset)-1\n # read the fasta file and add to the matrix\n for chead, cseq in sequence_generator(cfilename):\n cseq = cseq.upper()\n if cseq not in seqdict:\n seqdict[cseq] = len(seqlist)\n seqlist.append(cseq)\n cseqidx = seqdict[cseq]\n cfreq = float(sizeregexp.search(chead).group(0))\n try:\n obs[cseqidx, csampidx] += cfreq\n except IndexError:\n # exception means we ran out of space - add more OTUs\n shape = obs.shape\n obs.resize((shape[0]*2, shape[1]))\n obs[cseqidx, csampidx] = cfreq\n\n logger.info('for output biom table loaded %d samples, %d unique sequences'\n % (len(samplist), len(seqlist)))\n\n # and now make the sparse matrix the real size\n obs.resize((len(seqlist), len(samplist)))\n\n # do the minimal reads per otu filtering\n if minreads > 0:\n readsperotu = obs.sum(axis=1)\n keep = np.where(readsperotu >= minreads)[0]\n logger.info('keeping %d (out of %d sequences) with >=%d reads' %\n (len(keep), len(seqlist), minreads))\n obs = obs[keep, :]\n seqlist = list(np.array(seqlist)[keep])\n logger.debug('filtering completed')\n\n # convert the matrix to a biom table\n table = Table(obs.tocsr(), seqlist, samplist,\n observation_metadata=None,\n sample_metadata=None, table_id=None,\n generated_by=\"deblur\",\n create_date=datetime.now().isoformat())\n logger.debug('converted to biom table')\n\n # remove samples with 0 reads\n filter_minreads_samples_from_table(table)\n\n # save the merged otu table\n write_biom_table(table, output_fp)\n logger.info('saved to biom file %s' % output_fp)\n\n # and save the fasta file\n if outputfasta_fp is not None:\n logger.debug('saving fasta file')\n with open(outputfasta_fp, 'w') as f:\n for cseq in seqlist:\n f.write('>%s\\n%s\\n' % (cseq, cseq))\n logger.info('saved sequence fasta file to %s' % outputfasta_fp)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nlaunches a full deblur workflow for a single post split - libraries fasta file.", "response": "def launch_workflow(seqs_fp, working_dir, mean_error, error_dist,\n indel_prob, indel_max, trim_length, left_trim_length,\n min_size, ref_fp, ref_db_fp, threads_per_sample=1,\n sim_thresh=None, coverage_thresh=None):\n \"\"\"Launch full deblur workflow for a single post split-libraries fasta file\n\n Parameters\n ----------\n seqs_fp: string\n a post split library fasta file for debluring\n working_dir: string\n working directory path\n mean_error: float\n mean error for original sequence estimate\n error_dist: list\n list of error probabilities for each hamming distance\n indel_prob: float\n insertion/deletion (indel) probability\n indel_max: integer\n maximal indel number\n trim_length: integer\n sequence trim length\n left_trim_length: integer\n trim the first n reads\n min_size: integer\n upper limit on sequence abundance (discard sequences below limit)\n ref_fp: tuple\n filepath(s) to FASTA reference database for artifact removal\n ref_db_fp: tuple\n filepath(s) to SortMeRNA indexed database for artifact removal\n threads_per_sample: integer, optional\n number of threads to use for SortMeRNA/mafft/vsearch\n (0 for max available)\n sim_thresh: float, optional\n the minimal similarity for a sequence to the database.\n if None, take the defaults (0.65 for negate=False,\n 0.95 for negate=True)\n coverage_thresh: float, optional\n the minimal coverage for alignment of a sequence to the database.\n if None, take the defaults (0.3 for negate=False, 0.95 for negate=True)\n\n Return\n ------\n output_no_chimers_fp : string\n filepath to fasta file with no chimeras of None if error encountered\n \"\"\"\n logger = logging.getLogger(__name__)\n logger.info('--------------------------------------------------------')\n logger.info('launch_workflow for file %s' % seqs_fp)\n\n # Step 1: Trim sequences to specified length\n output_trim_fp = join(working_dir, \"%s.trim\" % basename(seqs_fp))\n with open(output_trim_fp, 'w') as out_f:\n for label, seq in trim_seqs(\n input_seqs=sequence_generator(seqs_fp),\n trim_len=trim_length,\n left_trim_len=left_trim_length):\n out_f.write(\">%s\\n%s\\n\" % (label, seq))\n # Step 2: Dereplicate sequences\n output_derep_fp = join(working_dir,\n \"%s.derep\" % basename(output_trim_fp))\n dereplicate_seqs(seqs_fp=output_trim_fp,\n output_fp=output_derep_fp,\n min_size=min_size, threads=threads_per_sample)\n # Step 3: Remove artifacts\n output_artif_fp, num_seqs_left, _ = remove_artifacts_seqs(seqs_fp=output_derep_fp,\n ref_fp=ref_fp,\n working_dir=working_dir,\n ref_db_fp=ref_db_fp,\n negate=True,\n threads=threads_per_sample,\n sim_thresh=sim_thresh)\n if not output_artif_fp:\n warnings.warn('Problem removing artifacts from file %s' %\n seqs_fp, UserWarning)\n logger.warning('remove artifacts failed, aborting')\n return None\n # Step 4: Multiple sequence alignment\n if num_seqs_left > 1:\n output_msa_fp = join(working_dir,\n \"%s.msa\" % basename(output_artif_fp))\n alignment = multiple_sequence_alignment(seqs_fp=output_artif_fp,\n threads=threads_per_sample)\n if not alignment:\n warnings.warn('Problem performing multiple sequence alignment '\n 'on file %s' % seqs_fp, UserWarning)\n logger.warning('msa failed. aborting')\n return None\n elif num_seqs_left == 1:\n # only one sequence after remove artifacts (but could be many reads)\n # no need to run MSA - just use the pre-msa file as input for next step\n output_msa_fp = output_artif_fp\n else:\n err_msg = ('No sequences left after artifact removal in '\n 'file %s' % seqs_fp)\n warnings.warn(err_msg, UserWarning)\n logger.warning(err_msg)\n return None\n # Step 5: Launch deblur\n output_deblur_fp = join(working_dir,\n \"%s.deblur\" % basename(output_msa_fp))\n with open(output_deblur_fp, 'w') as f:\n seqs = deblur(sequence_generator(output_msa_fp), mean_error,\n error_dist, indel_prob, indel_max)\n if seqs is None:\n warnings.warn('multiple sequence alignment file %s contains '\n 'no sequences' % output_msa_fp, UserWarning)\n logger.warn('no sequences returned from deblur for file %s' %\n output_msa_fp)\n return None\n for s in seqs:\n # remove '-' from aligned sequences\n s.sequence = s.sequence.replace('-', '')\n f.write(s.to_fasta())\n # Step 6: Chimera removal\n output_no_chimeras_fp = remove_chimeras_denovo_from_seqs(\n output_deblur_fp, working_dir, threads=threads_per_sample)\n logger.info('finished processing file')\n return output_no_chimeras_fp"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nstart the logger for the run", "response": "def start_log(level=logging.DEBUG, filename=None):\n \"\"\"start the logger for the run\n\n Parameters\n ----------\n level : int, optional\n logging.DEBUG, logging.INFO etc. for the log level (between 0-50).\n filename : str, optional\n name of the filename to save the log to or\n None (default) to use deblur.log.TIMESTAMP\n \"\"\"\n if filename is None:\n tstr = time.ctime()\n tstr = tstr.replace(' ', '.')\n tstr = tstr.replace(':', '.')\n filename = 'deblur.log.%s' % tstr\n logging.basicConfig(filename=filename, level=level,\n format='%(levelname)s(%(thread)d)'\n '%(asctime)s:%(message)s')\n logger = logging.getLogger(__name__)\n logger.info('*************************')\n logger.info('deblurring started')"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nexecuting the command cmd and return the standard output and standard error and exist status of the command.", "response": "def _system_call(cmd, stdoutfilename=None):\n \"\"\"Execute the command `cmd`\n Parameters\n ----------\n cmd : str\n The string containing the command to be run.\n stdoutfilename : str\n Name of the file to save stdout to or None\n (default) to not save to file\n stderrfilename : str\n Name of the file to save stderr to or None\n (default) to not save to file\n\n Returns\n -------\n tuple of (str, str, int)\n The standard output, standard error and exist status of the\n executed command\n\n Notes\n -----\n This function is ported and modified from QIIME\n (http://www.qiime.org), previously named\n qiime_system_call. QIIME is a GPL project, but we obtained permission from\n the authors of this function to port it to Qiita and keep it under BSD\n license.\n \"\"\"\n logger = logging.getLogger(__name__)\n logger.debug('system call: %s' % cmd)\n if stdoutfilename:\n with open(stdoutfilename, 'w') as f:\n proc = subprocess.Popen(cmd, universal_newlines=True,\n shell=False, stdout=f,\n stderr=subprocess.PIPE)\n else:\n proc = subprocess.Popen(cmd, universal_newlines=True,\n shell=False, stdout=subprocess.PIPE,\n stderr=subprocess.PIPE)\n # Communicate pulls all stdout/stderr from the PIPEs\n # This call blocks until the command is done\n stdout, stderr = proc.communicate()\n return_value = proc.returncode\n\n return stdout, stderr, return_value"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_sequences(input_seqs):\n try:\n seqs = [Sequence(id, seq) for id, seq in input_seqs]\n except Exception:\n seqs = []\n\n if len(seqs) == 0:\n logger = logging.getLogger(__name__)\n logger.warn('No sequences found in fasta file!')\n return None\n\n # Check that all the sequence lengths (aligned and unaligned are the same)\n aligned_lengths = set(s.length for s in seqs)\n unaligned_lengths = set(s.unaligned_length for s in seqs)\n\n if len(aligned_lengths) != 1 or len(unaligned_lengths) != 1:\n raise ValueError(\n \"Not all sequence have the same length. Aligned lengths: %s, \"\n \"sequence lengths: %s\"\n % (\", \".join(map(str, aligned_lengths)),\n \", \".join(map(str, unaligned_lengths))))\n\n seqs = sorted(seqs, key=attrgetter('frequency'), reverse=True)\n return seqs", "response": "Returns a list of Sequences\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef to_fasta(self):\n prefix, suffix = re.split('(?<=size=)\\w+', self.label, maxsplit=1)\n new_count = int(round(self.frequency))\n new_label = \"%s%d%s\" % (prefix, new_count, suffix)\n return \">%s\\n%s\\n\" % (new_label, self.sequence)", "response": "Returns a string with the sequence in fasta format"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nrendering options for select2.", "response": "def render_select2_options_code(self, options, id_):\n \"\"\"Render options for select2.\"\"\"\n output = []\n for key, value in options.items():\n if isinstance(value, (dict, list)):\n value = json.dumps(value)\n output.append(\"data-{name}='{value}'\".format(\n name=key,\n value=mark_safe(value)))\n return mark_safe(' '.join(output))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nrendering html container for Select2 widget with options.", "response": "def render_js_code(self, id_, *args, **kwargs):\n \"\"\"Render html container for Select2 widget with options.\"\"\"\n if id_:\n options = self.render_select2_options_code(\n dict(self.get_options()), id_)\n return mark_safe(self.html.format(id=id_, options=options))\n return u''"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nextends base class s render method by appending javascript inline text to html output.", "response": "def render(self, name, value, attrs=None, **kwargs):\n \"\"\"\n Extend base class's `render` method by appending\n javascript inline text to html output.\n \"\"\"\n output = super(Select2Mixin, self).render(\n name, value, attrs=attrs, **kwargs)\n id_ = attrs['id']\n output += self.render_js_code(\n id_, name, value, attrs=attrs, **kwargs)\n return mark_safe(output)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngenerates the console directive.", "response": "def visit_console_html(self, node):\n \"\"\"Generate HTML for the console directive.\"\"\"\n if self.builder.name in ('djangohtml', 'json') and node['win_console_text']:\n # Put a mark on the document object signaling the fact the directive\n # has been used on it.\n self.document._console_directive_used_flag = True\n uid = node['uid']\n self.body.append('''\\\n
\n\n\n\n\n
\\n''' % {'id': uid})\n try:\n self.visit_literal_block(node)\n except nodes.SkipNode:\n pass\n self.body.append('
\\n')\n\n self.body.append('
\\n' % {'id': uid})\n win_text = node['win_console_text']\n highlight_args = {'force': True}\n if 'linenos' in node:\n linenos = node['linenos']\n else:\n linenos = win_text.count('\\n') >= self.highlightlinenothreshold - 1\n\n def warner(msg):\n self.builder.warn(msg, (self.builder.current_docname, node.line))\n\n highlighted = self.highlighter.highlight_block(\n win_text, 'doscon', warn=warner, linenos=linenos, **highlight_args\n )\n self.body.append(highlighted)\n self.body.append('
\\n')\n self.body.append('
\\n')\n raise nodes.SkipNode\n else:\n self.visit_literal_block(node)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef select2_modelform_meta(model,\n meta_fields=None,\n widgets=None,\n attrs=None,\n **kwargs):\n \"\"\"\n Return `Meta` class with Select2-enabled widgets for fields\n with choices (e.g. ForeignKey, CharField, etc) for use with\n ModelForm.\n\n Arguments:\n model - a model class to create `Meta` class for.\n meta_fields - dictionary with `Meta` class fields, for\n example, {'fields': ['id', 'name']}\n attrs - select2 widget attributes (width, for example),\n must be of type `dict`.\n **kwargs - will be merged with meta_fields.\n \"\"\"\n widgets = widgets or {}\n meta_fields = meta_fields or {}\n\n # TODO: assert attrs is of type `dict`\n\n for field in model._meta.fields:\n if isinstance(field, ForeignKey) or field.choices:\n widgets.update({field.name: Select2(select2attrs=attrs)})\n\n for field in model._meta.many_to_many:\n widgets.update({field.name: Select2Multiple(select2attrs=attrs)})\n\n meta_fields.update({\n 'model': model,\n 'widgets': widgets,\n })\n if 'exclude' not in kwargs and 'fields' not in kwargs:\n meta_fields.update({'exclude': []})\n meta_fields.update(**kwargs)\n meta = type('Meta', (object,), meta_fields)\n\n return meta", "response": "Returns a new meta class with Select2 - enabled widgets for fields\nracket with kwargs"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a modelform class for a model with select2 widgets.", "response": "def select2_modelform(\n model, attrs=None, form_class=es2_forms.FixedModelForm):\n \"\"\"\n Return ModelForm class for model with select2 widgets.\n\n Arguments:\n attrs: select2 widget attributes (width, for example) of type `dict`.\n form_class: modelform base class, `forms.ModelForm` by default.\n\n ::\n\n SomeModelForm = select2_modelform(models.SomeModelBanner)\n\n is the same like::\n\n class SomeModelForm(forms.ModelForm):\n Meta = select2_modelform_meta(models.SomeModelForm)\n \"\"\"\n classname = '%sForm' % model._meta.object_name\n meta = select2_modelform_meta(model, attrs=attrs)\n return type(classname, (form_class,), {'Meta': meta})"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nyields (token_type, str_data) tokens. The last token will be (EOF, None) where EOF is the singleton object defined in this module.", "response": "def lex(self, text):\n \"\"\" Yield (token_type, str_data) tokens.\n\n The last token will be (EOF, None) where EOF is the singleton object\n defined in this module.\n \"\"\"\n for match in self.regex.finditer(text):\n for name, _ in self.lexicon:\n m = match.group(name)\n if m is not None:\n yield (name, m)\n break\n yield (EOF, None)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses a string of SVG transform = \" data.", "response": "def parse(self, text):\n \"\"\" Parse a string of SVG transform=\"\" data.\n \"\"\"\n gen = self.lexer.lex(text)\n next_val_fn = partial(next, *(gen,))\n\n commands = []\n token = next_val_fn()\n while token[0] is not EOF:\n command, token = self.rule_svg_transform(next_val_fn, token)\n commands.append(command)\n return commands"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nparsing a string of SVG data.", "response": "def parse(self, text):\n \"\"\" Parse a string of SVG data.\n \"\"\"\n gen = self.lexer.lex(text)\n next_val_fn = partial(next, *(gen,))\n token = next_val_fn()\n return self.rule_svg_path(next_val_fn, token)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef findElementsWithId(node, elems=None):\n if elems is None:\n elems = {}\n id = node.getAttribute('id')\n if id != '':\n elems[id] = node\n if node.hasChildNodes():\n for child in node.childNodes:\n # from http://www.w3.org/TR/DOM-Level-2-Core/idl-definitions.html\n # we are only really interested in nodes of type Element (1)\n if child.nodeType == Node.ELEMENT_NODE:\n findElementsWithId(child, elems)\n return elems", "response": "Returns all elements with id attributes"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfinds all elements that are referenced by a node.", "response": "def findReferencedElements(node, ids=None):\n \"\"\"\n Returns IDs of all referenced elements\n - node is the node at which to start the search.\n - returns a map which has the id as key and\n each value is is a list of nodes\n\n Currently looks at 'xlink:href' and all attributes in 'referencingProps'\n \"\"\"\n global referencingProps\n if ids is None:\n ids = {}\n # TODO: input argument ids is clunky here (see below how it is called)\n # GZ: alternative to passing dict, use **kwargs\n\n # if this node is a style element, parse its text into CSS\n if node.nodeName == 'style' and node.namespaceURI == NS['SVG']:\n # one stretch of text, please! (we could use node.normalize(), but\n # this actually modifies the node, and we don't want to keep\n # whitespace around if there's any)\n stylesheet = \"\".join([child.nodeValue for child in node.childNodes])\n if stylesheet != '':\n cssRules = parseCssString(stylesheet)\n for rule in cssRules:\n for propname in rule['properties']:\n propval = rule['properties'][propname]\n findReferencingProperty(node, propname, propval, ids)\n return ids\n\n # else if xlink:href is set, then grab the id\n href = node.getAttributeNS(NS['XLINK'], 'href')\n if href != '' and len(href) > 1 and href[0] == '#':\n # we remove the hash mark from the beginning of the id\n id = href[1:]\n if id in ids:\n ids[id].append(node)\n else:\n ids[id] = [node]\n\n # now get all style properties and the fill, stroke, filter attributes\n styles = node.getAttribute('style').split(';')\n\n for style in styles:\n propval = style.split(':')\n if len(propval) == 2:\n prop = propval[0].strip()\n val = propval[1].strip()\n findReferencingProperty(node, prop, val, ids)\n\n for attr in referencingProps:\n val = node.getAttribute(attr).strip()\n if not val:\n continue\n findReferencingProperty(node, attr, val, ids)\n\n if node.hasChildNodes():\n for child in node.childNodes:\n if child.nodeType == Node.ELEMENT_NODE:\n findReferencedElements(child, ids)\n return ids"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nremove all unreferenced elements from the document.", "response": "def removeUnreferencedElements(doc, keepDefs):\n \"\"\"\n Removes all unreferenced elements except for , , , , and .\n Also vacuums the defs of any non-referenced renderable elements.\n\n Returns the number of unreferenced elements removed from the document.\n \"\"\"\n global _num_elements_removed\n num = 0\n\n # Remove certain unreferenced elements outside of defs\n removeTags = ['linearGradient', 'radialGradient', 'pattern']\n identifiedElements = findElementsWithId(doc.documentElement)\n referencedIDs = findReferencedElements(doc.documentElement)\n\n for id in identifiedElements:\n if id not in referencedIDs:\n goner = identifiedElements[id]\n if (goner is not None and goner.nodeName in removeTags\n and goner.parentNode is not None\n and goner.parentNode.tagName != 'defs'):\n goner.parentNode.removeChild(goner)\n num += 1\n _num_elements_removed += 1\n\n if not keepDefs:\n # Remove most unreferenced elements inside defs\n defs = doc.documentElement.getElementsByTagName('defs')\n for aDef in defs:\n elemsToRemove = removeUnusedDefs(doc, aDef)\n for elem in elemsToRemove:\n elem.parentNode.removeChild(elem)\n _num_elements_removed += 1\n num += 1\n return num"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef shortenIDs(doc, prefix, unprotectedElements=None):\n num = 0\n\n identifiedElements = findElementsWithId(doc.documentElement)\n if unprotectedElements is None:\n unprotectedElements = identifiedElements\n referencedIDs = findReferencedElements(doc.documentElement)\n\n # Make idList (list of idnames) sorted by reference count\n # descending, so the highest reference count is first.\n # First check that there's actually a defining element for the current ID name.\n # (Cyn: I've seen documents with #id references but no element with that ID!)\n idList = [(len(referencedIDs[rid]), rid) for rid in referencedIDs\n if rid in unprotectedElements]\n idList.sort(reverse=True)\n idList = [rid for count, rid in idList]\n\n # Add unreferenced IDs to end of idList in arbitrary order\n idList.extend([rid for rid in unprotectedElements if rid not in idList])\n\n curIdNum = 1\n\n for rid in idList:\n curId = intToID(curIdNum, prefix)\n # First make sure that *this* element isn't already using\n # the ID name we want to give it.\n if curId != rid:\n # Then, skip ahead if the new ID is already in identifiedElement.\n while curId in identifiedElements:\n curIdNum += 1\n curId = intToID(curIdNum, prefix)\n # Then go rename it.\n num += renameID(doc, rid, curId, identifiedElements, referencedIDs)\n curIdNum += 1\n\n return num", "response": "Shortens the IDs used in the document to have the given prefix."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the ID name for the given ID number.", "response": "def intToID(idnum, prefix):\n \"\"\"\n Returns the ID name for the given ID number, spreadsheet-style, i.e. from a to z,\n then from aa to az, ba to bz, etc., until zz.\n \"\"\"\n rid = ''\n\n while idnum > 0:\n idnum -= 1\n rid = chr((idnum % 26) + ord('a')) + rid\n idnum = int(idnum / 26)\n\n return prefix + rid"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrenaming an ID name from idFrom to idTo on the declaring element identifiedElements and referencedIDs.", "response": "def renameID(doc, idFrom, idTo, identifiedElements, referencedIDs):\n \"\"\"\n Changes the ID name from idFrom to idTo, on the declaring element\n as well as all references in the document doc.\n\n Updates identifiedElements and referencedIDs.\n Does not handle the case where idTo is already the ID name\n of another element in doc.\n\n Returns the number of bytes saved by this replacement.\n \"\"\"\n\n num = 0\n\n definingNode = identifiedElements[idFrom]\n definingNode.setAttribute(\"id\", idTo)\n del identifiedElements[idFrom]\n identifiedElements[idTo] = definingNode\n num += len(idFrom) - len(idTo)\n\n # Update references to renamed node\n referringNodes = referencedIDs.get(idFrom)\n if referringNodes is not None:\n\n # Look for the idFrom ID name in each of the referencing elements,\n # exactly like findReferencedElements would.\n # Cyn: Duplicated processing!\n\n for node in referringNodes:\n # if this node is a style element, parse its text into CSS\n if node.nodeName == 'style' and node.namespaceURI == NS['SVG']:\n # node.firstChild will be either a CDATA or a Text node now\n if node.firstChild is not None:\n # concatenate the value of all children, in case\n # there's a CDATASection node surrounded by whitespace\n # nodes\n # (node.normalize() will NOT work here, it only acts on Text nodes)\n oldValue = \"\".join([child.nodeValue for child in node.childNodes])\n # not going to reparse the whole thing\n newValue = oldValue.replace('url(#' + idFrom + ')', 'url(#' + idTo + ')')\n newValue = newValue.replace(\"url(#'\" + idFrom + \"')\", 'url(#' + idTo + ')')\n newValue = newValue.replace('url(#\"' + idFrom + '\")', 'url(#' + idTo + ')')\n # and now replace all the children with this new stylesheet.\n # again, this is in case the stylesheet was a CDATASection\n node.childNodes[:] = [node.ownerDocument.createTextNode(newValue)]\n num += len(oldValue) - len(newValue)\n\n # if xlink:href is set to #idFrom, then change the id\n href = node.getAttributeNS(NS['XLINK'], 'href')\n if href == '#' + idFrom:\n node.setAttributeNS(NS['XLINK'], 'href', '#' + idTo)\n num += len(idFrom) - len(idTo)\n\n # if the style has url(#idFrom), then change the id\n styles = node.getAttribute('style')\n if styles != '':\n newValue = styles.replace('url(#' + idFrom + ')', 'url(#' + idTo + ')')\n newValue = newValue.replace(\"url('#\" + idFrom + \"')\", 'url(#' + idTo + ')')\n newValue = newValue.replace('url(\"#' + idFrom + '\")', 'url(#' + idTo + ')')\n node.setAttribute('style', newValue)\n num += len(styles) - len(newValue)\n\n # now try the fill, stroke, filter attributes\n for attr in referencingProps:\n oldValue = node.getAttribute(attr)\n if oldValue != '':\n newValue = oldValue.replace('url(#' + idFrom + ')', 'url(#' + idTo + ')')\n newValue = newValue.replace(\"url('#\" + idFrom + \"')\", 'url(#' + idTo + ')')\n newValue = newValue.replace('url(\"#' + idFrom + '\")', 'url(#' + idTo + ')')\n node.setAttribute(attr, newValue)\n num += len(oldValue) - len(newValue)\n\n del referencedIDs[idFrom]\n referencedIDs[idTo] = referringNodes\n\n return num"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef unprotected_ids(doc, options):\n identifiedElements = findElementsWithId(doc.documentElement)\n if not (options.protect_ids_noninkscape or\n options.protect_ids_list or\n options.protect_ids_prefix):\n return identifiedElements\n if options.protect_ids_list:\n protect_ids_list = options.protect_ids_list.split(\",\")\n if options.protect_ids_prefix:\n protect_ids_prefixes = options.protect_ids_prefix.split(\",\")\n for id in list(identifiedElements):\n protected = False\n if options.protect_ids_noninkscape and not id[-1].isdigit():\n protected = True\n if options.protect_ids_list and id in protect_ids_list:\n protected = True\n if options.protect_ids_prefix:\n for prefix in protect_ids_prefixes:\n if id.startswith(prefix):\n protected = True\n if protected:\n del identifiedElements[id]\n return identifiedElements", "response": "u Returns a list of unprotected IDs within the document doc."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef removeUnreferencedIDs(referencedIDs, identifiedElements):\n global _num_ids_removed\n keepTags = ['font']\n num = 0\n for id in identifiedElements:\n node = identifiedElements[id]\n if id not in referencedIDs and node.nodeName not in keepTags:\n node.removeAttribute('id')\n _num_ids_removed += 1\n num += 1\n return num", "response": "Removes the unreferenced ID attributes from the identifiedElements."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate a group of consecutive child elements with common attributes.", "response": "def createGroupsForCommonAttributes(elem):\n \"\"\"\n Creates elements to contain runs of 3 or more\n consecutive child elements having at least one common attribute.\n\n Common attributes are not promoted to the by this function.\n This is handled by moveCommonAttributesToParentGroup.\n\n If all children have a common attribute, an extra is not created.\n\n This function acts recursively on the given element.\n \"\"\"\n num = 0\n global _num_elements_removed\n\n # TODO perhaps all of the Presentation attributes in http://www.w3.org/TR/SVG/struct.html#GElement\n # could be added here\n # Cyn: These attributes are the same as in moveAttributesToParentGroup, and must always be\n for curAttr in ['clip-rule',\n 'display-align',\n 'fill', 'fill-opacity', 'fill-rule',\n 'font', 'font-family', 'font-size', 'font-size-adjust', 'font-stretch',\n 'font-style', 'font-variant', 'font-weight',\n 'letter-spacing',\n 'pointer-events', 'shape-rendering',\n 'stroke', 'stroke-dasharray', 'stroke-dashoffset', 'stroke-linecap', 'stroke-linejoin',\n 'stroke-miterlimit', 'stroke-opacity', 'stroke-width',\n 'text-anchor', 'text-decoration', 'text-rendering', 'visibility',\n 'word-spacing', 'writing-mode']:\n # Iterate through the children in reverse order, so item(i) for\n # items we have yet to visit still returns the correct nodes.\n curChild = elem.childNodes.length - 1\n while curChild >= 0:\n childNode = elem.childNodes.item(curChild)\n\n if (\n childNode.nodeType == Node.ELEMENT_NODE and\n childNode.getAttribute(curAttr) != '' and\n childNode.nodeName in [\n # only attempt to group elements that the content model allows to be children of a \n\n # SVG 1.1 (see https://www.w3.org/TR/SVG/struct.html#GElement)\n 'animate', 'animateColor', 'animateMotion', 'animateTransform', 'set', # animation elements\n 'desc', 'metadata', 'title', # descriptive elements\n 'circle', 'ellipse', 'line', 'path', 'polygon', 'polyline', 'rect', # shape elements\n 'defs', 'g', 'svg', 'symbol', 'use', # structural elements\n 'linearGradient', 'radialGradient', # gradient elements\n 'a', 'altGlyphDef', 'clipPath', 'color-profile', 'cursor', 'filter',\n 'font', 'font-face', 'foreignObject', 'image', 'marker', 'mask',\n 'pattern', 'script', 'style', 'switch', 'text', 'view',\n\n # SVG 1.2 (see https://www.w3.org/TR/SVGTiny12/elementTable.html)\n 'animation', 'audio', 'discard', 'handler', 'listener',\n 'prefetch', 'solidColor', 'textArea', 'video'\n ]\n ):\n # We're in a possible run! Track the value and run length.\n value = childNode.getAttribute(curAttr)\n runStart, runEnd = curChild, curChild\n # Run elements includes only element tags, no whitespace/comments/etc.\n # Later, we calculate a run length which includes these.\n runElements = 1\n\n # Backtrack to get all the nodes having the same\n # attribute value, preserving any nodes in-between.\n while runStart > 0:\n nextNode = elem.childNodes.item(runStart - 1)\n if nextNode.nodeType == Node.ELEMENT_NODE:\n if nextNode.getAttribute(curAttr) != value:\n break\n else:\n runElements += 1\n runStart -= 1\n else:\n runStart -= 1\n\n if runElements >= 3:\n # Include whitespace/comment/etc. nodes in the run.\n while runEnd < elem.childNodes.length - 1:\n if elem.childNodes.item(runEnd + 1).nodeType == Node.ELEMENT_NODE:\n break\n else:\n runEnd += 1\n\n runLength = runEnd - runStart + 1\n if runLength == elem.childNodes.length: # Every child has this\n # If the current parent is a already,\n if elem.nodeName == 'g' and elem.namespaceURI == NS['SVG']:\n # do not act altogether on this attribute; all the\n # children have it in common.\n # Let moveCommonAttributesToParentGroup do it.\n curChild = -1\n continue\n # otherwise, it might be an element, and\n # even if all children have the same attribute value,\n # it's going to be worth making the since\n # doesn't support attributes like 'stroke'.\n # Fall through.\n\n # Create a element from scratch.\n # We need the Document for this.\n document = elem.ownerDocument\n group = document.createElementNS(NS['SVG'], 'g')\n # Move the run of elements to the group.\n # a) ADD the nodes to the new group.\n group.childNodes[:] = elem.childNodes[runStart:runEnd + 1]\n for child in group.childNodes:\n child.parentNode = group\n # b) REMOVE the nodes from the element.\n elem.childNodes[runStart:runEnd + 1] = []\n # Include the group in elem's children.\n elem.childNodes.insert(runStart, group)\n group.parentNode = elem\n num += 1\n curChild = runStart - 1\n _num_elements_removed -= 1\n else:\n curChild -= 1\n else:\n curChild -= 1\n\n # each child gets the same treatment, recursively\n for childNode in elem.childNodes:\n if childNode.nodeType == Node.ELEMENT_NODE:\n num += createGroupsForCommonAttributes(childNode)\n\n return num"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef removeUnusedAttributesOnParent(elem):\n num = 0\n\n childElements = []\n # recurse first into the children (depth-first)\n for child in elem.childNodes:\n if child.nodeType == Node.ELEMENT_NODE:\n childElements.append(child)\n num += removeUnusedAttributesOnParent(child)\n\n # only process the children if there are more than one element\n if len(childElements) <= 1:\n return num\n\n # get all attribute values on this parent\n attrList = elem.attributes\n unusedAttrs = {}\n for index in range(attrList.length):\n attr = attrList.item(index)\n if attr.nodeName in ['clip-rule',\n 'display-align',\n 'fill', 'fill-opacity', 'fill-rule',\n 'font', 'font-family', 'font-size', 'font-size-adjust', 'font-stretch',\n 'font-style', 'font-variant', 'font-weight',\n 'letter-spacing',\n 'pointer-events', 'shape-rendering',\n 'stroke', 'stroke-dasharray', 'stroke-dashoffset', 'stroke-linecap', 'stroke-linejoin',\n 'stroke-miterlimit', 'stroke-opacity', 'stroke-width',\n 'text-anchor', 'text-decoration', 'text-rendering', 'visibility',\n 'word-spacing', 'writing-mode']:\n unusedAttrs[attr.nodeName] = attr.nodeValue\n\n # for each child, if at least one child inherits the parent's attribute, then remove\n for childNum in range(len(childElements)):\n child = childElements[childNum]\n inheritedAttrs = []\n for name in unusedAttrs:\n val = child.getAttribute(name)\n if val == '' or val is None or val == 'inherit':\n inheritedAttrs.append(name)\n for a in inheritedAttrs:\n del unusedAttrs[a]\n\n # unusedAttrs now has all the parent attributes that are unused\n for name in unusedAttrs:\n elem.removeAttribute(name)\n num += 1\n\n return num", "response": "This function recursively calls this function on all children of the passed in and removes any unused attributes on the parent elem if none of the children inherit it."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _getStyle(node):\n if node.nodeType == Node.ELEMENT_NODE and len(node.getAttribute('style')) > 0:\n styleMap = {}\n rawStyles = node.getAttribute('style').split(';')\n for style in rawStyles:\n propval = style.split(':')\n if len(propval) == 2:\n styleMap[propval[0].strip()] = propval[1].strip()\n return styleMap\n else:\n return {}", "response": "u Returns the style attribute of a node as a dictionary."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _setStyle(node, styleMap):\n fixedStyle = ';'.join([prop + ':' + styleMap[prop] for prop in styleMap])\n if fixedStyle != '':\n node.setAttribute('style', fixedStyle)\n elif node.getAttribute('style'):\n node.removeAttribute('style')\n return node", "response": "u Set the style attribute of a node to the dictionary styleMap."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the value of style that is inherited from the parents of the passed - in node.", "response": "def styleInheritedFromParent(node, style):\n \"\"\"\n Returns the value of 'style' that is inherited from the parents of the passed-in node\n\n Warning: This method only considers presentation attributes and inline styles,\n any style sheets are ignored!\n \"\"\"\n parentNode = node.parentNode\n\n # return None if we reached the Document element\n if parentNode.nodeType == Node.DOCUMENT_NODE:\n return None\n\n # check styles first (they take precedence over presentation attributes)\n styles = _getStyle(parentNode)\n if style in styles:\n value = styles[style]\n if not value == 'inherit':\n return value\n\n # check attributes\n value = parentNode.getAttribute(style)\n if value not in ['', 'inherit']:\n return parentNode.getAttribute(style)\n\n # check the next parent recursively if we did not find a value yet\n return styleInheritedFromParent(parentNode, style)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef styleInheritedByChild(node, style, nodeIsChild=False):\n # Comment, text and CDATA nodes don't have attributes and aren't containers so they can't inherit attributes\n if node.nodeType != Node.ELEMENT_NODE:\n return False\n\n if nodeIsChild:\n # if the current child node sets a new value for 'style'\n # we can stop the search in the current branch of the DOM tree\n\n # check attributes\n if node.getAttribute(style) not in ['', 'inherit']:\n return False\n # check styles\n styles = _getStyle(node)\n if (style in styles) and not (styles[style] == 'inherit'):\n return False\n else:\n # if the passed-in node does not have any children 'style' can obviously not be inherited\n if not node.childNodes:\n return False\n\n # If we have child nodes recursively check those\n if node.childNodes:\n for child in node.childNodes:\n if styleInheritedByChild(child, style, True):\n return True\n\n # If the current element is a container element the inherited style is meaningless\n # (since we made sure it's not inherited by any of its children)\n if node.nodeName in ['a', 'defs', 'glyph', 'g', 'marker', 'mask',\n 'missing-glyph', 'pattern', 'svg', 'switch', 'symbol']:\n return False\n\n # in all other cases we have to assume the inherited value of 'style' is meaningfull and has to be kept\n # (e.g nodes without children at the end of the DOM tree, text nodes, ...)\n return True", "response": "Returns whether the passed - in node is inherited by any children of the passed - in node."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef mayContainTextNodes(node):\n # Cached result of a prior call?\n try:\n return node.mayContainTextNodes\n except AttributeError:\n pass\n\n result = True # Default value\n # Comment, text and CDATA nodes don't have attributes and aren't containers\n if node.nodeType != Node.ELEMENT_NODE:\n result = False\n # Non-SVG elements? Unknown elements!\n elif node.namespaceURI != NS['SVG']:\n result = True\n # Blacklisted elements. Those are guaranteed not to be text elements.\n elif node.nodeName in ['rect', 'circle', 'ellipse', 'line', 'polygon',\n 'polyline', 'path', 'image', 'stop']:\n result = False\n # Group elements. If we're missing any here, the default of True is used.\n elif node.nodeName in ['g', 'clipPath', 'marker', 'mask', 'pattern',\n 'linearGradient', 'radialGradient', 'symbol']:\n result = False\n for child in node.childNodes:\n if mayContainTextNodes(child):\n result = True\n # Everything else should be considered a future SVG-version text element\n # at best, or an unknown element at worst. result will stay True.\n\n # Cache this result before returning it.\n node.mayContainTextNodes = result\n return result", "response": "Returns True if the passed - in node is probably a text element or at least\n one of its descendants is probably a text element."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef removeDefaultAttributeValue(node, attribute):\n if not node.hasAttribute(attribute.name):\n return 0\n\n # differentiate between text and numeric values\n if isinstance(attribute.value, str):\n if node.getAttribute(attribute.name) == attribute.value:\n if (attribute.conditions is None) or attribute.conditions(node):\n node.removeAttribute(attribute.name)\n return 1\n else:\n nodeValue = SVGLength(node.getAttribute(attribute.name))\n if ((attribute.value is None)\n or ((nodeValue.value == attribute.value) and not (nodeValue.units == Unit.INVALID))):\n if ((attribute.units is None)\n or (nodeValue.units == attribute.units)\n or (isinstance(attribute.units, list) and nodeValue.units in attribute.units)):\n if (attribute.conditions is None) or attribute.conditions(node):\n node.removeAttribute(attribute.name)\n return 1\n\n return 0", "response": "Removes the DefaultAttribute attribute from node"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nremoving default values from the node.", "response": "def removeDefaultAttributeValues(node, options, tainted=set()):\n u\"\"\"'tainted' keeps a set of attributes defined in parent nodes.\n\n For such attributes, we don't delete attributes with default values.\"\"\"\n num = 0\n if node.nodeType != Node.ELEMENT_NODE:\n return 0\n\n # Conditionally remove all default attributes defined in 'default_attributes' (a list of 'DefaultAttribute's)\n #\n # For increased performance do not iterate the whole list for each element but run only on valid subsets\n # - 'default_attributes_universal' (attributes valid for all elements)\n # - 'default_attributes_per_element' (attributes specific to one specific element type)\n for attribute in default_attributes_universal:\n num += removeDefaultAttributeValue(node, attribute)\n if node.nodeName in default_attributes_per_element:\n for attribute in default_attributes_per_element[node.nodeName]:\n num += removeDefaultAttributeValue(node, attribute)\n\n # Summarily get rid of default properties\n attributes = [node.attributes.item(i).nodeName for i in range(node.attributes.length)]\n for attribute in attributes:\n if attribute not in tainted:\n if attribute in default_properties:\n if node.getAttribute(attribute) == default_properties[attribute]:\n node.removeAttribute(attribute)\n num += 1\n else:\n tainted = taint(tainted, attribute)\n # Properties might also occur as styles, remove them too\n styles = _getStyle(node)\n for attribute in list(styles):\n if attribute not in tainted:\n if attribute in default_properties:\n if styles[attribute] == default_properties[attribute]:\n del styles[attribute]\n num += 1\n else:\n tainted = taint(tainted, attribute)\n _setStyle(node, styles)\n\n # recurse for our child elements\n for child in node.childNodes:\n num += removeDefaultAttributeValues(child, options, tainted.copy())\n\n return num"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef convertColor(value):\n s = value\n\n if s in colors:\n s = colors[s]\n\n rgbpMatch = rgbp.match(s)\n if rgbpMatch is not None:\n r = int(float(rgbpMatch.group(1)) * 255.0 / 100.0)\n g = int(float(rgbpMatch.group(2)) * 255.0 / 100.0)\n b = int(float(rgbpMatch.group(3)) * 255.0 / 100.0)\n s = '#%02x%02x%02x' % (r, g, b)\n else:\n rgbMatch = rgb.match(s)\n if rgbMatch is not None:\n r = int(rgbMatch.group(1))\n g = int(rgbMatch.group(2))\n b = int(rgbMatch.group(3))\n s = '#%02x%02x%02x' % (r, g, b)\n\n if s[0] == '#':\n s = s.lower()\n if len(s) == 7 and s[1] == s[2] and s[3] == s[4] and s[5] == s[6]:\n s = '#' + s[1] + s[3] + s[5]\n\n return s", "response": "Converts the input color string and returns a string containing the converted color."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef cleanPath(element, options):\n global _num_bytes_saved_in_path_data\n global _num_path_segments_removed\n\n # this gets the parser object from svg_regex.py\n oldPathStr = element.getAttribute('d')\n path = svg_parser.parse(oldPathStr)\n style = _getStyle(element)\n\n # This determines whether the stroke has round or square linecaps. If it does, we do not want to collapse empty\n # segments, as they are actually rendered (as circles or squares with diameter/dimension matching the path-width).\n has_round_or_square_linecaps = (\n element.getAttribute('stroke-linecap') in ['round', 'square']\n or 'stroke-linecap' in style and style['stroke-linecap'] in ['round', 'square']\n )\n\n # This determines whether the stroke has intermediate markers. If it does, we do not want to collapse\n # straight segments running in the same direction, as markers are rendered on the intermediate nodes.\n has_intermediate_markers = (\n element.hasAttribute('marker')\n or element.hasAttribute('marker-mid')\n or 'marker' in style\n or 'marker-mid' in style\n )\n\n # The first command must be a moveto, and whether it's relative (m)\n # or absolute (M), the first set of coordinates *is* absolute. So\n # the first iteration of the loop below will get x,y and startx,starty.\n\n # convert absolute coordinates into relative ones.\n # Reuse the data structure 'path', since we're not adding or removing subcommands.\n # Also reuse the coordinate lists since we're not adding or removing any.\n x = y = 0\n for pathIndex in range(len(path)):\n cmd, data = path[pathIndex] # Changes to cmd don't get through to the data structure\n i = 0\n # adjust abs to rel\n # only the A command has some values that we don't want to adjust (radii, rotation, flags)\n if cmd == 'A':\n for i in range(i, len(data), 7):\n data[i + 5] -= x\n data[i + 6] -= y\n x += data[i + 5]\n y += data[i + 6]\n path[pathIndex] = ('a', data)\n elif cmd == 'a':\n x += sum(data[5::7])\n y += sum(data[6::7])\n elif cmd == 'H':\n for i in range(i, len(data)):\n data[i] -= x\n x += data[i]\n path[pathIndex] = ('h', data)\n elif cmd == 'h':\n x += sum(data)\n elif cmd == 'V':\n for i in range(i, len(data)):\n data[i] -= y\n y += data[i]\n path[pathIndex] = ('v', data)\n elif cmd == 'v':\n y += sum(data)\n elif cmd == 'M':\n startx, starty = data[0], data[1]\n # If this is a path starter, don't convert its first\n # coordinate to relative; that would just make it (0, 0)\n if pathIndex != 0:\n data[0] -= x\n data[1] -= y\n\n x, y = startx, starty\n i = 2\n for i in range(i, len(data), 2):\n data[i] -= x\n data[i + 1] -= y\n x += data[i]\n y += data[i + 1]\n path[pathIndex] = ('m', data)\n elif cmd in ['L', 'T']:\n for i in range(i, len(data), 2):\n data[i] -= x\n data[i + 1] -= y\n x += data[i]\n y += data[i + 1]\n path[pathIndex] = (cmd.lower(), data)\n elif cmd in ['m']:\n if pathIndex == 0:\n # START OF PATH - this is an absolute moveto\n # followed by relative linetos\n startx, starty = data[0], data[1]\n x, y = startx, starty\n i = 2\n else:\n startx = x + data[0]\n starty = y + data[1]\n for i in range(i, len(data), 2):\n x += data[i]\n y += data[i + 1]\n elif cmd in ['l', 't']:\n x += sum(data[0::2])\n y += sum(data[1::2])\n elif cmd in ['S', 'Q']:\n for i in range(i, len(data), 4):\n data[i] -= x\n data[i + 1] -= y\n data[i + 2] -= x\n data[i + 3] -= y\n x += data[i + 2]\n y += data[i + 3]\n path[pathIndex] = (cmd.lower(), data)\n elif cmd in ['s', 'q']:\n x += sum(data[2::4])\n y += sum(data[3::4])\n elif cmd == 'C':\n for i in range(i, len(data), 6):\n data[i] -= x\n data[i + 1] -= y\n data[i + 2] -= x\n data[i + 3] -= y\n data[i + 4] -= x\n data[i + 5] -= y\n x += data[i + 4]\n y += data[i + 5]\n path[pathIndex] = ('c', data)\n elif cmd == 'c':\n x += sum(data[4::6])\n y += sum(data[5::6])\n elif cmd in ['z', 'Z']:\n x, y = startx, starty\n path[pathIndex] = ('z', data)\n\n # remove empty segments and redundant commands\n # Reuse the data structure 'path' and the coordinate lists, even if we're\n # deleting items, because these deletions are relatively cheap.\n if not has_round_or_square_linecaps:\n # remove empty path segments\n for pathIndex in range(len(path)):\n cmd, data = path[pathIndex]\n i = 0\n if cmd in ['m', 'l', 't']:\n if cmd == 'm':\n # It might be tempting to rewrite \"m0 0 ...\" into\n # \"l...\" here. However, this is an unsound\n # optimization in general as \"m0 0 ... z\" is\n # different from \"l...z\".\n #\n # To do such a rewrite, we need to understand the\n # full subpath. This logic happens after this\n # loop.\n i = 2\n while i < len(data):\n if data[i] == data[i + 1] == 0:\n del data[i:i + 2]\n _num_path_segments_removed += 1\n else:\n i += 2\n elif cmd == 'c':\n while i < len(data):\n if data[i] == data[i + 1] == data[i + 2] == data[i + 3] == data[i + 4] == data[i + 5] == 0:\n del data[i:i + 6]\n _num_path_segments_removed += 1\n else:\n i += 6\n elif cmd == 'a':\n while i < len(data):\n if data[i + 5] == data[i + 6] == 0:\n del data[i:i + 7]\n _num_path_segments_removed += 1\n else:\n i += 7\n elif cmd == 'q':\n while i < len(data):\n if data[i] == data[i + 1] == data[i + 2] == data[i + 3] == 0:\n del data[i:i + 4]\n _num_path_segments_removed += 1\n else:\n i += 4\n elif cmd in ['h', 'v']:\n oldLen = len(data)\n path[pathIndex] = (cmd, [coord for coord in data if coord != 0])\n _num_path_segments_removed += len(path[pathIndex][1]) - oldLen\n\n # remove no-op commands\n pathIndex = len(path)\n subpath_needs_anchor = False\n # NB: We can never rewrite the first m/M command (expect if it\n # is the only command)\n while pathIndex > 1:\n pathIndex -= 1\n cmd, data = path[pathIndex]\n if cmd == 'z':\n next_cmd, next_data = path[pathIndex - 1]\n if next_cmd == 'm' and len(next_data) == 2:\n # mX Yz -> mX Y\n\n # note the len check on next_data as it is not\n # safe to rewrite \"m0 0 1 1z\" in general (it is a\n # question of where the \"pen\" ends - you can\n # continue a draw on the same subpath after a\n # \"z\").\n del path[pathIndex]\n _num_path_segments_removed += 1\n else:\n # it is not safe to rewrite \"m0 0 ...\" to \"l...\"\n # because of this \"z\" command.\n subpath_needs_anchor = True\n elif cmd == 'm':\n if len(path) - 1 == pathIndex and len(data) == 2:\n # Ends with an empty move (but no line/draw\n # following it)\n del path[pathIndex]\n _num_path_segments_removed += 1\n continue\n if subpath_needs_anchor:\n subpath_needs_anchor = False\n elif data[0] == data[1] == 0:\n # unanchored, i.e. we can replace \"m0 0 ...\" with\n # \"l...\" as there is no \"z\" after it.\n path[pathIndex] = ('l', data[2:])\n _num_path_segments_removed += 1\n\n # fixup: Delete subcommands having no coordinates.\n path = [elem for elem in path if len(elem[1]) > 0 or elem[0] == 'z']\n\n # convert straight curves into lines\n newPath = [path[0]]\n for (cmd, data) in path[1:]:\n i = 0\n newData = data\n if cmd == 'c':\n newData = []\n while i < len(data):\n # since all commands are now relative, we can think of previous point as (0,0)\n # and new point (dx,dy) is (data[i+4],data[i+5])\n # eqn of line will be y = (dy/dx)*x or if dx=0 then eqn of line is x=0\n (p1x, p1y) = (data[i], data[i + 1])\n (p2x, p2y) = (data[i + 2], data[i + 3])\n dx = data[i + 4]\n dy = data[i + 5]\n\n foundStraightCurve = False\n\n if dx == 0:\n if p1x == 0 and p2x == 0:\n foundStraightCurve = True\n else:\n m = dy / dx\n if p1y == m * p1x and p2y == m * p2x:\n foundStraightCurve = True\n\n if foundStraightCurve:\n # flush any existing curve coords first\n if newData:\n newPath.append((cmd, newData))\n newData = []\n # now create a straight line segment\n newPath.append(('l', [dx, dy]))\n else:\n newData.extend(data[i:i + 6])\n\n i += 6\n if newData or cmd == 'z' or cmd == 'Z':\n newPath.append((cmd, newData))\n path = newPath\n\n # collapse all consecutive commands of the same type into one command\n prevCmd = ''\n prevData = []\n newPath = []\n for (cmd, data) in path:\n if prevCmd == '':\n # initialize with current path cmd and data\n prevCmd = cmd\n prevData = data\n else:\n # collapse if\n # - cmd is not moveto (explicit moveto commands are not drawn)\n # - the previous and current commands are the same type,\n # - the previous command is moveto and the current is lineto\n # (subsequent moveto pairs are treated as implicit lineto commands)\n if cmd != 'm' and (cmd == prevCmd or (cmd == 'l' and prevCmd == 'm')):\n prevData.extend(data)\n # else flush the previous command if it is not the same type as the current command\n else:\n newPath.append((prevCmd, prevData))\n prevCmd = cmd\n prevData = data\n # flush last command and data\n newPath.append((prevCmd, prevData))\n path = newPath\n\n # convert to shorthand path segments where possible\n newPath = []\n for (cmd, data) in path:\n # convert line segments into h,v where possible\n if cmd == 'l':\n i = 0\n lineTuples = []\n while i < len(data):\n if data[i] == 0:\n # vertical\n if lineTuples:\n # flush the existing line command\n newPath.append(('l', lineTuples))\n lineTuples = []\n # append the v and then the remaining line coords\n newPath.append(('v', [data[i + 1]]))\n _num_path_segments_removed += 1\n elif data[i + 1] == 0:\n if lineTuples:\n # flush the line command, then append the h and then the remaining line coords\n newPath.append(('l', lineTuples))\n lineTuples = []\n newPath.append(('h', [data[i]]))\n _num_path_segments_removed += 1\n else:\n lineTuples.extend(data[i:i + 2])\n i += 2\n if lineTuples:\n newPath.append(('l', lineTuples))\n # also handle implied relative linetos\n elif cmd == 'm':\n i = 2\n lineTuples = [data[0], data[1]]\n while i < len(data):\n if data[i] == 0:\n # vertical\n if lineTuples:\n # flush the existing m/l command\n newPath.append((cmd, lineTuples))\n lineTuples = []\n cmd = 'l' # dealing with linetos now\n # append the v and then the remaining line coords\n newPath.append(('v', [data[i + 1]]))\n _num_path_segments_removed += 1\n elif data[i + 1] == 0:\n if lineTuples:\n # flush the m/l command, then append the h and then the remaining line coords\n newPath.append((cmd, lineTuples))\n lineTuples = []\n cmd = 'l' # dealing with linetos now\n newPath.append(('h', [data[i]]))\n _num_path_segments_removed += 1\n else:\n lineTuples.extend(data[i:i + 2])\n i += 2\n if lineTuples:\n newPath.append((cmd, lineTuples))\n # convert B\u00e9zier curve segments into s where possible\n elif cmd == 'c':\n # set up the assumed bezier control point as the current point,\n # i.e. (0,0) since we're using relative coords\n bez_ctl_pt = (0, 0)\n # however if the previous command was 's'\n # the assumed control point is a reflection of the previous control point at the current point\n if len(newPath):\n (prevCmd, prevData) = newPath[-1]\n if prevCmd == 's':\n bez_ctl_pt = (prevData[-2] - prevData[-4], prevData[-1] - prevData[-3])\n i = 0\n curveTuples = []\n while i < len(data):\n # rotate by 180deg means negate both coordinates\n # if the previous control point is equal then we can substitute a\n # shorthand bezier command\n if bez_ctl_pt[0] == data[i] and bez_ctl_pt[1] == data[i + 1]:\n if curveTuples:\n newPath.append(('c', curveTuples))\n curveTuples = []\n # append the s command\n newPath.append(('s', [data[i + 2], data[i + 3], data[i + 4], data[i + 5]]))\n _num_path_segments_removed += 1\n else:\n j = 0\n while j <= 5:\n curveTuples.append(data[i + j])\n j += 1\n\n # set up control point for next curve segment\n bez_ctl_pt = (data[i + 4] - data[i + 2], data[i + 5] - data[i + 3])\n i += 6\n\n if curveTuples:\n newPath.append(('c', curveTuples))\n # convert quadratic curve segments into t where possible\n elif cmd == 'q':\n quad_ctl_pt = (0, 0)\n i = 0\n curveTuples = []\n while i < len(data):\n if quad_ctl_pt[0] == data[i] and quad_ctl_pt[1] == data[i + 1]:\n if curveTuples:\n newPath.append(('q', curveTuples))\n curveTuples = []\n # append the t command\n newPath.append(('t', [data[i + 2], data[i + 3]]))\n _num_path_segments_removed += 1\n else:\n j = 0\n while j <= 3:\n curveTuples.append(data[i + j])\n j += 1\n\n quad_ctl_pt = (data[i + 2] - data[i], data[i + 3] - data[i + 1])\n i += 4\n\n if curveTuples:\n newPath.append(('q', curveTuples))\n else:\n newPath.append((cmd, data))\n path = newPath\n\n # For each m, l, h or v, collapse unnecessary coordinates that run in the same direction\n # i.e. \"h-100-100\" becomes \"h-200\" but \"h300-100\" does not change.\n # If the path has intermediate markers we have to preserve intermediate nodes, though.\n # Reuse the data structure 'path', since we're not adding or removing subcommands.\n # Also reuse the coordinate lists, even if we're deleting items, because these\n # deletions are relatively cheap.\n if not has_intermediate_markers:\n for pathIndex in range(len(path)):\n cmd, data = path[pathIndex]\n\n # h / v expects only one parameter and we start drawing with the first (so we need at least 2)\n if cmd in ['h', 'v'] and len(data) >= 2:\n coordIndex = 0\n while coordIndex+1 < len(data):\n if is_same_sign(data[coordIndex], data[coordIndex+1]):\n data[coordIndex] += data[coordIndex+1]\n del data[coordIndex+1]\n _num_path_segments_removed += 1\n else:\n coordIndex += 1\n\n # l expects two parameters and we start drawing with the first (so we need at least 4)\n elif cmd == 'l' and len(data) >= 4:\n coordIndex = 0\n while coordIndex+2 < len(data):\n if is_same_direction(*data[coordIndex:coordIndex+4]):\n data[coordIndex] += data[coordIndex+2]\n data[coordIndex+1] += data[coordIndex+3]\n del data[coordIndex+2] # delete the next two elements\n del data[coordIndex+2]\n _num_path_segments_removed += 1\n else:\n coordIndex += 2\n\n # m expects two parameters but we have to skip the first pair as it's not drawn (so we need at least 6)\n elif cmd == 'm' and len(data) >= 6:\n coordIndex = 2\n while coordIndex+2 < len(data):\n if is_same_direction(*data[coordIndex:coordIndex+4]):\n data[coordIndex] += data[coordIndex+2]\n data[coordIndex+1] += data[coordIndex+3]\n del data[coordIndex+2] # delete the next two elements\n del data[coordIndex+2]\n _num_path_segments_removed += 1\n else:\n coordIndex += 2\n\n # it is possible that we have consecutive h, v, c, t commands now\n # so again collapse all consecutive commands of the same type into one command\n prevCmd = ''\n prevData = []\n newPath = [path[0]]\n for (cmd, data) in path[1:]:\n # flush the previous command if it is not the same type as the current command\n if prevCmd != '':\n if cmd != prevCmd or cmd == 'm':\n newPath.append((prevCmd, prevData))\n prevCmd = ''\n prevData = []\n\n # if the previous and current commands are the same type, collapse\n if cmd == prevCmd and cmd != 'm':\n prevData.extend(data)\n\n # save last command and data\n else:\n prevCmd = cmd\n prevData = data\n # flush last command and data\n if prevCmd != '':\n newPath.append((prevCmd, prevData))\n path = newPath\n\n newPathStr = serializePath(path, options)\n\n # if for whatever reason we actually made the path longer don't use it\n # TODO: maybe we could compare path lengths after each optimization step and use the shortest\n if len(newPathStr) <= len(oldPathStr):\n _num_bytes_saved_in_path_data += (len(oldPathStr) - len(newPathStr))\n element.setAttribute('d', newPathStr)", "response": "Cleans the path string of the element and returns a new empty tree structure."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef parseListOfPoints(s):\n i = 0\n\n # (wsp)? comma-or-wsp-separated coordinate pairs (wsp)?\n # coordinate-pair = coordinate comma-or-wsp coordinate\n # coordinate = sign? integer\n # comma-wsp: (wsp+ comma? wsp*) | (comma wsp*)\n ws_nums = re.split(r\"\\s*[\\s,]\\s*\", s.strip())\n nums = []\n\n # also, if 100-100 is found, split it into two also\n # \n for i in range(len(ws_nums)):\n negcoords = ws_nums[i].split(\"-\")\n\n # this string didn't have any negative coordinates\n if len(negcoords) == 1:\n nums.append(negcoords[0])\n # we got negative coords\n else:\n for j in range(len(negcoords)):\n # first number could be positive\n if j == 0:\n if negcoords[0] != '':\n nums.append(negcoords[0])\n # otherwise all other strings will be negative\n else:\n # unless we accidentally split a number that was in scientific notation\n # and had a negative exponent (500.00e-1)\n prev = \"\"\n if len(nums):\n prev = nums[len(nums) - 1]\n if prev and prev[len(prev) - 1] in ['e', 'E']:\n nums[len(nums) - 1] = prev + '-' + negcoords[j]\n else:\n nums.append('-' + negcoords[j])\n\n # if we have an odd number of points, return empty\n if len(nums) % 2 != 0:\n return []\n\n # now resolve into Decimal values\n i = 0\n while i < len(nums):\n try:\n nums[i] = getcontext().create_decimal(nums[i])\n nums[i + 1] = getcontext().create_decimal(nums[i + 1])\n except InvalidOperation: # one of the lengths had a unit or is an invalid number\n return []\n\n i += 2\n\n return nums", "response": "Parse a string into a list of points."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef cleanPolygon(elem, options):\n global _num_points_removed_from_polygon\n\n pts = parseListOfPoints(elem.getAttribute('points'))\n N = len(pts) / 2\n if N >= 2:\n (startx, starty) = pts[:2]\n (endx, endy) = pts[-2:]\n if startx == endx and starty == endy:\n del pts[-2:]\n _num_points_removed_from_polygon += 1\n elem.setAttribute('points', scourCoordinates(pts, options, True))", "response": "Remove unnecessary closing point of polygon points attribute\n "} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef cleanPolyline(elem, options):\n pts = parseListOfPoints(elem.getAttribute('points'))\n elem.setAttribute('points', scourCoordinates(pts, options, True))", "response": "Scour the polyline points attribute\n "} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if there are control points in the path data", "response": "def controlPoints(cmd, data):\n \"\"\"\n Checks if there are control points in the path data\n\n Returns the indices of all values in the path data which are control points\n \"\"\"\n cmd = cmd.lower()\n if cmd in ['c', 's', 'q']:\n indices = range(len(data))\n if cmd == 'c': # c: (x1 y1 x2 y2 x y)+\n return [index for index in indices if (index % 6) < 4]\n elif cmd in ['s', 'q']: # s: (x2 y2 x y)+ q: (x1 y1 x y)+\n return [index for index in indices if (index % 4) < 2]\n\n return []"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck if there are flags in the path data", "response": "def flags(cmd, data):\n \"\"\"\n Checks if there are flags in the path data\n\n Returns the indices of all values in the path data which are flags\n \"\"\"\n if cmd.lower() == 'a': # a: (rx ry x-axis-rotation large-arc-flag sweep-flag x y)+\n indices = range(len(data))\n return [index for index in indices if (index % 7) in [3, 4]]\n\n return []"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nserialize the path data with some cleanups.", "response": "def serializePath(pathObj, options):\n \"\"\"\n Reserializes the path data with some cleanups.\n \"\"\"\n # elliptical arc commands must have comma/wsp separating the coordinates\n # this fixes an issue outlined in Fix https://bugs.launchpad.net/scour/+bug/412754\n return ''.join([cmd + scourCoordinates(data, options,\n control_points=controlPoints(cmd, data),\n flags=flags(cmd, data))\n for cmd, data in pathObj])"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef serializeTransform(transformObj):\n return ' '.join([command + '(' + ' '.join([scourUnitlessLength(number) for number in numbers]) + ')'\n for command, numbers in transformObj])", "response": "Serializes the transform data with some cleanups."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nscouring the coordinates of the given data into a list of strings.", "response": "def scourCoordinates(data, options, force_whitespace=False, control_points=[], flags=[]):\n \"\"\"\n Serializes coordinate data with some cleanups:\n - removes all trailing zeros after the decimal\n - integerize coordinates if possible\n - removes extraneous whitespace\n - adds spaces between values in a subcommand if required (or if force_whitespace is True)\n \"\"\"\n if data is not None:\n newData = []\n c = 0\n previousCoord = ''\n for coord in data:\n is_control_point = c in control_points\n scouredCoord = scourUnitlessLength(coord,\n renderer_workaround=options.renderer_workaround,\n is_control_point=is_control_point)\n # don't output a space if this number starts with a dot (.) or minus sign (-); we only need a space if\n # - this number starts with a digit\n # - this number starts with a dot but the previous number had *no* dot or exponent\n # i.e. '1.3 0.5' -> '1.3.5' or '1e3 0.5' -> '1e3.5' is fine but '123 0.5' -> '123.5' is obviously not\n # - 'force_whitespace' is explicitly set to 'True'\n # we never need a space after flags (occuring in elliptical arcs), but librsvg struggles without it\n if (c > 0\n and (force_whitespace\n or scouredCoord[0].isdigit()\n or (scouredCoord[0] == '.' and not ('.' in previousCoord or 'e' in previousCoord)))\n and ((c-1 not in flags) or options.renderer_workaround)):\n newData.append(' ')\n\n # add the scoured coordinate to the path string\n newData.append(scouredCoord)\n previousCoord = scouredCoord\n c += 1\n\n # What we need to do to work around GNOME bugs 548494, 563933 and 620565, is to make sure that a dot doesn't\n # immediately follow a command (so 'h50' and 'h0.5' are allowed, but not 'h.5').\n # Then, we need to add a space character after any coordinates having an 'e' (scientific notation),\n # so as to have the exponent separate from the next number.\n # TODO: Check whether this is still required (bugs all marked as fixed, might be time to phase it out)\n if options.renderer_workaround:\n if len(newData) > 0:\n for i in range(1, len(newData)):\n if newData[i][0] == '-' and 'e' in newData[i - 1]:\n newData[i - 1] += ' '\n return ''.join(newData)\n else:\n return ''.join(newData)\n\n return ''"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nscours a length. Accepts units.", "response": "def scourLength(length):\n \"\"\"\n Scours a length. Accepts units.\n \"\"\"\n length = SVGLength(length)\n\n return scourUnitlessLength(length.value) + Unit.str(length.units)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nscours the length of a single element and returns the length of the element.", "response": "def scourUnitlessLength(length, renderer_workaround=False, is_control_point=False): # length is of a numeric type\n \"\"\"\n Scours the numeric part of a length only. Does not accept units.\n\n This is faster than scourLength on elements guaranteed not to\n contain units.\n \"\"\"\n if not isinstance(length, Decimal):\n length = getcontext().create_decimal(str(length))\n initial_length = length\n\n # reduce numeric precision\n # plus() corresponds to the unary prefix plus operator and applies context precision and rounding\n if is_control_point:\n length = scouringContextC.plus(length)\n else:\n length = scouringContext.plus(length)\n\n # remove trailing zeroes as we do not care for significance\n intLength = length.to_integral_value()\n if length == intLength:\n length = Decimal(intLength)\n else:\n length = length.normalize()\n\n # Gather the non-scientific notation version of the coordinate.\n # Re-quantize from the initial value to prevent unnecessary loss of precision\n # (e.g. 123.4 should become 123, not 120 or even 100)\n nonsci = '{0:f}'.format(length)\n nonsci = '{0:f}'.format(initial_length.quantize(Decimal(nonsci)))\n if not renderer_workaround:\n if len(nonsci) > 2 and nonsci[:2] == '0.':\n nonsci = nonsci[1:] # remove the 0, leave the dot\n elif len(nonsci) > 3 and nonsci[:3] == '-0.':\n nonsci = '-' + nonsci[2:] # remove the 0, leave the minus and dot\n return_value = nonsci\n\n # Gather the scientific notation version of the coordinate which\n # can only be shorter if the length of the number is at least 4 characters (e.g. 1000 = 1e3).\n if len(nonsci) > 3:\n # We have to implement this ourselves since both 'normalize()' and 'to_sci_string()'\n # don't handle negative exponents in a reasonable way (e.g. 0.000001 remains unchanged)\n exponent = length.adjusted() # how far do we have to shift the dot?\n length = length.scaleb(-exponent).normalize() # shift the dot and remove potential trailing zeroes\n\n sci = six.text_type(length) + 'e' + six.text_type(exponent)\n\n if len(sci) < len(nonsci):\n return_value = sci\n\n return return_value"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreduce the precision of the given element node and its children to the current Decimal context s precision.", "response": "def reducePrecision(element):\n \"\"\"\n Because opacities, letter spacings, stroke widths and all that don't need\n to be preserved in SVG files with 9 digits of precision.\n\n Takes all of these attributes, in the given element node and its children,\n and reduces their precision to the current Decimal context's precision.\n Also checks for the attributes actually being lengths, not 'inherit', 'none'\n or anything that isn't an SVGLength.\n\n Returns the number of bytes saved after performing these reductions.\n \"\"\"\n num = 0\n\n styles = _getStyle(element)\n for lengthAttr in ['opacity', 'flood-opacity', 'fill-opacity',\n 'stroke-opacity', 'stop-opacity', 'stroke-miterlimit',\n 'stroke-dashoffset', 'letter-spacing', 'word-spacing',\n 'kerning', 'font-size-adjust', 'font-size',\n 'stroke-width']:\n val = element.getAttribute(lengthAttr)\n if val != '':\n valLen = SVGLength(val)\n if valLen.units != Unit.INVALID: # not an absolute/relative size or inherit, can be % though\n newVal = scourLength(val)\n if len(newVal) < len(val):\n num += len(val) - len(newVal)\n element.setAttribute(lengthAttr, newVal)\n # repeat for attributes hidden in styles\n if lengthAttr in styles:\n val = styles[lengthAttr]\n valLen = SVGLength(val)\n if valLen.units != Unit.INVALID:\n newVal = scourLength(val)\n if len(newVal) < len(val):\n num += len(val) - len(newVal)\n styles[lengthAttr] = newVal\n _setStyle(element, styles)\n\n for child in element.childNodes:\n if child.nodeType == Node.ELEMENT_NODE:\n num += reducePrecision(child)\n\n return num"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef optimizeAngle(angle):\n # First, we put the new angle in the range ]-360, 360[.\n # The modulo operator yields results with the sign of the\n # divisor, so for negative dividends, we preserve the sign\n # of the angle.\n if angle < 0:\n angle %= -360\n else:\n angle %= 360\n # 720 degrees is unnecessary, as 360 covers all angles.\n # As \"-x\" is shorter than \"35x\" and \"-xxx\" one character\n # longer than positive angles <= 260, we constrain angle\n # range to [-90, 270[ (or, equally valid: ]-100, 260]).\n if angle >= 270:\n angle -= 360\n elif angle < -90:\n angle += 360\n return angle", "response": "This function is used to optimize the angle of a single - character system."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\noptimizes a series of transformations parsed from a single transform = transform attribute.", "response": "def optimizeTransform(transform):\n \"\"\"\n Optimises a series of transformations parsed from a single\n transform=\"\" attribute.\n\n The transformation list is modified in-place.\n \"\"\"\n # FIXME: reordering these would optimize even more cases:\n # first: Fold consecutive runs of the same transformation\n # extra: Attempt to cast between types to create sameness:\n # \"matrix(0 1 -1 0 0 0) rotate(180) scale(-1)\" all\n # are rotations (90, 180, 180) -- thus \"rotate(90)\"\n # second: Simplify transforms where numbers are optional.\n # third: Attempt to simplify any single remaining matrix()\n #\n # if there's only one transformation and it's a matrix,\n # try to make it a shorter non-matrix transformation\n # NOTE: as matrix(a b c d e f) in SVG means the matrix:\n # |\u00af a c e \u00af| make constants |\u00af A1 A2 A3 \u00af|\n # | b d f | translating them | B1 B2 B3 |\n # |_ 0 0 1 _| to more readable |_ 0 0 1 _|\n if len(transform) == 1 and transform[0][0] == 'matrix':\n matrix = A1, B1, A2, B2, A3, B3 = transform[0][1]\n # |\u00af 1 0 0 \u00af|\n # | 0 1 0 | Identity matrix (no transformation)\n # |_ 0 0 1 _|\n if matrix == [1, 0, 0, 1, 0, 0]:\n del transform[0]\n # |\u00af 1 0 X \u00af|\n # | 0 1 Y | Translation by (X, Y).\n # |_ 0 0 1 _|\n elif (A1 == 1 and A2 == 0\n and B1 == 0 and B2 == 1):\n transform[0] = ('translate', [A3, B3])\n # |\u00af X 0 0 \u00af|\n # | 0 Y 0 | Scaling by (X, Y).\n # |_ 0 0 1 _|\n elif (A2 == 0 and A3 == 0\n and B1 == 0 and B3 == 0):\n transform[0] = ('scale', [A1, B2])\n # |\u00af cos(A) -sin(A) 0 \u00af| Rotation by angle A,\n # | sin(A) cos(A) 0 | clockwise, about the origin.\n # |_ 0 0 1 _| A is in degrees, [-180...180].\n elif (A1 == B2 and -1 <= A1 <= 1 and A3 == 0\n and -B1 == A2 and -1 <= B1 <= 1 and B3 == 0\n # as cos\u00b2 A + sin\u00b2 A == 1 and as decimal trig is approximate:\n # FIXME: the \"epsilon\" term here should really be some function\n # of the precision of the (sin|cos)_A terms, not 1e-15:\n and abs((B1 ** 2) + (A1 ** 2) - 1) < Decimal(\"1e-15\")):\n sin_A, cos_A = B1, A1\n # while asin(A) and acos(A) both only have an 180\u00b0 range\n # the sign of sin(A) and cos(A) varies across quadrants,\n # letting us hone in on the angle the matrix represents:\n # -- => < -90 | -+ => -90..0 | ++ => 0..90 | +- => >= 90\n #\n # http://en.wikipedia.org/wiki/File:Sine_cosine_plot.svg\n # shows asin has the correct angle the middle quadrants:\n A = Decimal(str(math.degrees(math.asin(float(sin_A)))))\n if cos_A < 0: # otherwise needs adjusting from the edges\n if sin_A < 0:\n A = -180 - A\n else:\n A = 180 - A\n transform[0] = ('rotate', [A])\n\n # Simplify transformations where numbers are optional.\n for type, args in transform:\n if type == 'translate':\n # Only the X coordinate is required for translations.\n # If the Y coordinate is unspecified, it's 0.\n if len(args) == 2 and args[1] == 0:\n del args[1]\n elif type == 'rotate':\n args[0] = optimizeAngle(args[0]) # angle\n # Only the angle is required for rotations.\n # If the coordinates are unspecified, it's the origin (0, 0).\n if len(args) == 3 and args[1] == args[2] == 0:\n del args[1:]\n elif type == 'scale':\n # Only the X scaling factor is required.\n # If the Y factor is unspecified, it's the same as X.\n if len(args) == 2 and args[0] == args[1]:\n del args[1]\n\n # Attempt to coalesce runs of the same transformation.\n # Translations followed immediately by other translations,\n # rotations followed immediately by other rotations,\n # scaling followed immediately by other scaling,\n # are safe to add.\n # Identity skewX/skewY are safe to remove, but how do they accrete?\n # |\u00af 1 0 0 \u00af|\n # | tan(A) 1 0 | skews X coordinates by angle A\n # |_ 0 0 1 _|\n #\n # |\u00af 1 tan(A) 0 \u00af|\n # | 0 1 0 | skews Y coordinates by angle A\n # |_ 0 0 1 _|\n #\n # FIXME: A matrix followed immediately by another matrix\n # would be safe to multiply together, too.\n i = 1\n while i < len(transform):\n currType, currArgs = transform[i]\n prevType, prevArgs = transform[i - 1]\n if currType == prevType == 'translate':\n prevArgs[0] += currArgs[0] # x\n # for y, only add if the second translation has an explicit y\n if len(currArgs) == 2:\n if len(prevArgs) == 2:\n prevArgs[1] += currArgs[1] # y\n elif len(prevArgs) == 1:\n prevArgs.append(currArgs[1]) # y\n del transform[i]\n if prevArgs[0] == prevArgs[1] == 0:\n # Identity translation!\n i -= 1\n del transform[i]\n elif (currType == prevType == 'rotate'\n and len(prevArgs) == len(currArgs) == 1):\n # Only coalesce if both rotations are from the origin.\n prevArgs[0] = optimizeAngle(prevArgs[0] + currArgs[0])\n del transform[i]\n elif currType == prevType == 'scale':\n prevArgs[0] *= currArgs[0] # x\n # handle an implicit y\n if len(prevArgs) == 2 and len(currArgs) == 2:\n # y1 * y2\n prevArgs[1] *= currArgs[1]\n elif len(prevArgs) == 1 and len(currArgs) == 2:\n # create y2 = uniformscalefactor1 * y2\n prevArgs.append(prevArgs[0] * currArgs[1])\n elif len(prevArgs) == 2 and len(currArgs) == 1:\n # y1 * uniformscalefactor2\n prevArgs[1] *= currArgs[0]\n del transform[i]\n # if prevArgs is [1] or [1, 1], then it is effectively an\n # identity matrix and can be removed.\n if prevArgs[0] == 1 and (len(prevArgs) == 1 or prevArgs[1] == 1):\n # Identity scale!\n i -= 1\n del transform[i]\n else:\n i += 1\n\n # Some fixups are needed for single-element transformation lists, since\n # the loop above was to coalesce elements with their predecessors in the\n # list, and thus it required 2 elements.\n i = 0\n while i < len(transform):\n currType, currArgs = transform[i]\n if ((currType == 'skewX' or currType == 'skewY')\n and len(currArgs) == 1 and currArgs[0] == 0):\n # Identity skew!\n del transform[i]\n elif ((currType == 'rotate')\n and len(currArgs) == 1 and currArgs[0] == 0):\n # Identity rotation!\n del transform[i]\n else:\n i += 1"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef optimizeTransforms(element, options):\n num = 0\n\n for transformAttr in ['transform', 'patternTransform', 'gradientTransform']:\n val = element.getAttribute(transformAttr)\n if val != '':\n transform = svg_transform_parser.parse(val)\n\n optimizeTransform(transform)\n\n newVal = serializeTransform(transform)\n\n if len(newVal) < len(val):\n if len(newVal):\n element.setAttribute(transformAttr, newVal)\n else:\n element.removeAttribute(transformAttr)\n num += len(val) - len(newVal)\n\n for child in element.childNodes:\n if child.nodeType == Node.ELEMENT_NODE:\n num += optimizeTransforms(child, options)\n\n return num", "response": "Optimize the transform specifications on the given node and its children."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nremoves comments from the element and its children.", "response": "def removeComments(element):\n \"\"\"\n Removes comments from the element and its children.\n \"\"\"\n global _num_bytes_saved_in_comments\n num = 0\n\n if isinstance(element, xml.dom.minidom.Comment):\n _num_bytes_saved_in_comments += len(element.data)\n element.parentNode.removeChild(element)\n num += 1\n else:\n for subelement in element.childNodes[:]:\n num += removeComments(subelement)\n\n return num"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting raster references to inline images. NOTE: there are size limits to base64-encoding handling in browsers", "response": "def embedRasters(element, options):\n import base64\n \"\"\"\n Converts raster references to inline images.\n NOTE: there are size limits to base64-encoding handling in browsers\n \"\"\"\n global _num_rasters_embedded\n\n href = element.getAttributeNS(NS['XLINK'], 'href')\n\n # if xlink:href is set, then grab the id\n if href != '' and len(href) > 1:\n ext = os.path.splitext(os.path.basename(href))[1].lower()[1:]\n\n # only operate on files with 'png', 'jpg', and 'gif' file extensions\n if ext in ['png', 'jpg', 'gif']:\n # fix common issues with file paths\n # TODO: should we warn the user instead of trying to correct those invalid URIs?\n # convert backslashes to slashes\n href_fixed = href.replace('\\\\', '/')\n # absolute 'file:' URIs have to use three slashes (unless specifying a host which I've never seen)\n href_fixed = re.sub('file:/+', 'file:///', href_fixed)\n\n # parse the URI to get scheme and path\n # in principle it would make sense to work only with this ParseResult and call 'urlunparse()' in the end\n # however 'urlunparse(urlparse(file:raster.png))' -> 'file:///raster.png' which is nonsense\n parsed_href = urllib.parse.urlparse(href_fixed)\n\n # assume locations without protocol point to local files (and should use the 'file:' protocol)\n if parsed_href.scheme == '':\n parsed_href = parsed_href._replace(scheme='file')\n if href_fixed[0] == '/':\n href_fixed = 'file://' + href_fixed\n else:\n href_fixed = 'file:' + href_fixed\n\n # relative local paths are relative to the input file, therefore temporarily change the working dir\n working_dir_old = None\n if parsed_href.scheme == 'file' and parsed_href.path[0] != '/':\n if options.infilename:\n working_dir_old = os.getcwd()\n working_dir_new = os.path.abspath(os.path.dirname(options.infilename))\n os.chdir(working_dir_new)\n\n # open/download the file\n try:\n file = urllib.request.urlopen(href_fixed)\n rasterdata = file.read()\n file.close()\n except Exception as e:\n print(\"WARNING: Could not open file '\" + href + \"' for embedding. \"\n \"The raster image will be kept as a reference but might be invalid. \"\n \"(Exception details: \" + str(e) + \")\", file=options.ensure_value(\"stdout\", sys.stdout))\n rasterdata = ''\n finally:\n # always restore initial working directory if we changed it above\n if working_dir_old is not None:\n os.chdir(working_dir_old)\n\n # TODO: should we remove all images which don't resolve?\n # then we also have to consider unreachable remote locations (i.e. if there is no internet connection)\n if rasterdata != '':\n # base64-encode raster\n b64eRaster = base64.b64encode(rasterdata)\n\n # set href attribute to base64-encoded equivalent\n if b64eRaster != '':\n # PNG and GIF both have MIME Type 'image/[ext]', but\n # JPEG has MIME Type 'image/jpeg'\n if ext == 'jpg':\n ext = 'jpeg'\n\n element.setAttributeNS(NS['XLINK'], 'href',\n 'data:image/' + ext + ';base64,' + b64eRaster.decode())\n _num_rasters_embedded += 1\n del b64eRaster"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_format(format_):\n format_path = os.getenv('FORMAT_PATH', 'format_files')\n try:\n spec = importlib.util.spec_from_file_location(\n \"format_mod\", f\"{format_path}/{format_}.py\")\n format_mod = importlib.util.module_from_spec(spec)\n spec.loader.exec_module(format_mod)\n except FileNotFoundError:\n try:\n format_mod = import_module(f'nerium.schema.{format_}')\n except ModuleNotFoundError:\n format_mod = import_module('nerium.schema.default')\n schema = format_mod.ResultSchema()\n return schema", "response": "Find format schema in nerium format_files or nerium. schema. default"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nset a new key value pair.", "response": "def set_property(self, key, value):\n '''Set a new (or updating existing) key value pair.\n\n Args:\n key: A string containing the key namespace\n value: A str, int, or bool value\n\n Raises:\n NotImplementedError: an unsupported value-type was provided\n '''\n value_type = type(value)\n if value_type not in [str, int, bool]:\n raise NotImplementedError(\n 'Only string, integer, and boolean properties are implemented')\n\n key_object = self.properties.findChild(name='key', text=key)\n\n # Key (and value, if it's a valid property list) don't exist\n if key_object is None:\n key_object = self.soup.new_tag('key')\n key_object.string = key\n\n self.properties.append(key_object)\n\n value_object = self.soup.new_tag(\n {str: 'string', int: 'integer', bool: str(value).lower()}[\n value_type])\n if value_type is not bool:\n value_object.string = str(value)\n\n self.properties.append(value_object)\n\n return\n\n # Key (and value, if it's a valid property list) exist\n # Eh, just remove the key+value tags from the tree and re-add them\n # (with the new value)\n value_object = key_object.find_next_sibling()\n\n key_object.decompose()\n value_object.decompose()\n\n self.set_property(key, value)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef save(self):\n '''Save current property list representation to the original file.'''\n with open(self.filename, 'w') as plist_file:\n plist_file.write(str(self.soup))", "response": "Save current property list representation to the original file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fetch_userid(self, side):\n for user in self.users:\n obj = self.users[user]\n if obj.side == side:\n return user", "response": "Return the userid for the specified bed side."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfetches new session token from api.", "response": "async def fetch_token(self):\n \"\"\"Fetch new session token from api.\"\"\"\n url = '{}/login'.format(API_URL)\n payload = 'email={}&password={}'.format(self._email, self._password)\n\n reg = await self.api_post(url, None, payload)\n if reg is None:\n _LOGGER.error('Unable to authenticate and fetch eight token.')\n else:\n self._userid = reg['session']['userId']\n self._token = reg['session']['token']\n self._expdate = reg['session']['expirationDate']\n _LOGGER.debug('UserID: %s, Token: %s', self._userid, self.token)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\nasync def fetch_device_list(self):\n url = '{}/users/me'.format(API_URL)\n\n dlist = await self.api_get(url)\n if dlist is None:\n _LOGGER.error('Unable to fetch eight devices.')\n else:\n self._devices = dlist['user']['devices']\n _LOGGER.debug('Devices: %s', self._devices)", "response": "Fetch list of devices."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\nasync def assign_users(self):\n device = self._devices[0]\n url = '{}/devices/{}?filter=ownerId,leftUserId,rightUserId' \\\n .format(API_URL, device)\n\n data = await self.api_get(url)\n if data is None:\n _LOGGER.error('Unable to assign eight device users.')\n else:\n # Find the side to the known userid\n if data['result']['rightUserId'] == self._userid:\n self.users[data['result']['rightUserId']] = \\\n EightUser(self, data['result']['rightUserId'], 'right')\n user_side = 'right'\n elif data['result']['leftUserId'] == self._userid:\n self.users[data['result']['leftUserId']] = \\\n EightUser(self, data['result']['leftUserId'], 'left')\n user_side = 'left'\n else:\n _LOGGER.error('Unable to assign eight device users.')\n\n if self._partner:\n if user_side == 'right':\n self.users[data['result']['leftUserId']] = \\\n EightUser(self, data['result']['leftUserId'], 'left')\n else:\n self.users[data['result']['rightUserId']] = \\\n EightUser(self, data['result']['rightUserId'], 'right')", "response": "Assign users to the eight device."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef room_temperature(self):\n # Check which side is active, if both are return the average\n tmp = None\n tmp2 = None\n for user in self.users:\n obj = self.users[user]\n if obj.current_values['processing']:\n if tmp is None:\n tmp = obj.current_values['room_temp']\n else:\n tmp = (tmp + obj.current_values['room_temp']) / 2\n else:\n if tmp2 is None:\n tmp2 = obj.current_values['room_temp']\n else:\n tmp2 = (tmp2 + obj.current_values['room_temp']) / 2\n\n if tmp is not None:\n return tmp\n elif tmp2 is not None:\n return tmp2", "response": "Return room temperature for both sides of bed."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nmanages the device json list.", "response": "def handle_device_json(self, data):\n \"\"\"Manage the device json list.\"\"\"\n self._device_json.insert(0, data)\n self._device_json.pop()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\nasync def update_device_data(self):\n url = '{}/devices/{}?offlineView=true'.format(API_URL, self.deviceid)\n\n # Check for access token expiration (every 15days)\n exp_delta = datetime.strptime(self._expdate, '%Y-%m-%dT%H:%M:%S.%fZ') \\\n - datetime.fromtimestamp(time.time())\n # Renew 1hr before expiration\n if exp_delta.total_seconds() < 3600:\n _LOGGER.debug('Fetching new access token before expiration.')\n await self.fetch_token()\n\n device_resp = await self.api_get(url)\n if device_resp is None:\n _LOGGER.error('Unable to fetch eight device data.')\n else:\n # Want to keep last 10 readings so purge the last after we add\n self.handle_device_json(device_resp['result'])\n for user in self.users:\n self.users[user].dynamic_presence()", "response": "Update device data json."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\nasync def api_get(self, url, params=None):\n request = None\n headers = DEFAULT_HEADERS.copy()\n headers.update({'Session-Token': self._token})\n\n try:\n with async_timeout.timeout(DEFAULT_TIMEOUT, loop=self._event_loop):\n request = await self._api_session.get(\n url, headers=headers, params=params)\n # _LOGGER.debug('Get URL: %s', request.url)\n if request.status != 200:\n _LOGGER.error('Error fetching Eight data: %s', request.status)\n return None\n\n if 'application/json' in request.headers['content-type']:\n request_json = await request.json()\n else:\n _LOGGER.debug('Response was not JSON, returning text.')\n request_json = await request.text()\n\n return request_json\n\n except (aiohttp.ClientError, asyncio.TimeoutError,\n ConnectionRefusedError) as err:\n _LOGGER.error('Error fetching Eight data. %s', err)\n return None", "response": "Make api fetch request."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nmaking api post request.", "response": "async def api_put(self, url, data=None):\n \"\"\"Make api post request.\"\"\"\n put = None\n headers = DEFAULT_HEADERS.copy()\n headers.update({'Session-Token': self._token})\n\n try:\n with async_timeout.timeout(DEFAULT_TIMEOUT, loop=self._event_loop):\n put = await self._api_session.put(\n url, headers=headers, data=data)\n if put.status != 200:\n _LOGGER.error('Error putting Eight data: %s', put.status)\n return None\n\n if 'application/json' in put.headers['content-type']:\n put_result = await put.json()\n else:\n _LOGGER.debug('Response was not JSON, returning text.')\n put_result = await put.text()\n\n return put_result\n\n except (aiohttp.ClientError, asyncio.TimeoutError,\n ConnectionRefusedError) as err:\n _LOGGER.error('Error putting Eight data. %s', err)\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef satisfy(self, *requirements):\n versions = []\n for req_range in requirements:\n try:\n version = self._detected_versions[req_range.name]\n except KeyError:\n try:\n version = req_range.requirement.detect_version()\n except ArcanaRequirementNotFoundError as e:\n if self._fail_on_missing:\n raise\n else:\n logger.warning(e)\n except ArcanaVersionNotDetectableError as e:\n if self._fail_on_undetectable:\n raise\n else:\n logger.warning(e)\n else:\n self._detected_versions[req_range.name] = version\n if not req_range.within(version):\n raise ArcanaVersionError(\n \"Detected {} version {} is not within requested range {}\"\n .format(req_range.requirement, version, req_range))\n versions.append(version)\n return versions", "response": "Returns a list of versions that satisfy the given requirements."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting a X10 housecode to a zero - based index", "response": "def housecode_to_index(housecode):\n \"\"\"Convert a X10 housecode to a zero-based index\"\"\"\n match = re.search(r'^([A-P])(\\d{1,2})$', housecode.upper())\n if match:\n house_index = int(match.group(2))\n if 1 <= house_index <= 16:\n return (ord(match.group(1)) - ord('A')) * 16 + house_index - 1\n raise ValueError(\"Invalid X10 housecode: %s\" % housecode)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts a zero - based index to a X10 housecode.", "response": "def index_to_housecode(index):\n \"\"\"Convert a zero-based index to a X10 housecode.\"\"\"\n if index < 0 or index > 255:\n raise ValueError\n quotient, remainder = divmod(index, 16)\n return chr(quotient+ord('A')) + '{:02d}'.format(remainder+1)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _check_checksum(msg):\n checksum = int(msg[-2:], 16)\n for char in msg[:-2]:\n checksum += ord(char)\n if (checksum % 256) != 0:\n raise ValueError(\"Elk message checksum invalid\")", "response": "Ensure checksum in message is good."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nchecking that the message is valid and that checksum is good.", "response": "def _check_message_valid(msg):\n \"\"\"Check packet length valid and that checksum is good.\"\"\"\n try:\n if int(msg[:2], 16) != (len(msg) - 2):\n raise ValueError(\"Elk message length incorrect\")\n _check_checksum(msg)\n except IndexError:\n raise ValueError(\"Elk message length incorrect\")"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef cw_encode(index, value, value_format):\n if value_format == 2:\n value = value[0] << 8 + value[1]\n return MessageEncode('0Dcw{:02d}{:05d}00'.format(index+1, value), None)", "response": "Encode a CWR message."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef dm_encode(keypad_area, clear, beep, timeout, line1, line2):\n return MessageEncode(\n '2Edm{:1d}{:1d}{:1d}{:05d}{:^<16.16}{:^<16.16}00'\n .format(keypad_area+1, clear, beep, timeout, line1, line2), None\n )", "response": "dm - Display message on keypad."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef zb_encode(zone, area, user_code):\n if zone < 0:\n zone = 0\n elif zone > Max.ZONES.value:\n zone = 999\n else:\n zone += 1\n return MessageEncode('10zb{zone:03d}{area:1d}{code:06d}00'.format(\n zone=zone, area=area+1, code=user_code), 'ZB')", "response": "zb - ZB encode"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_handler(self, message_type, handler):\n if message_type not in self._handlers:\n self._handlers[message_type] = []\n\n if handler not in self._handlers[message_type]:\n self._handlers[message_type].append(handler)", "response": "Manage callbacks for message handlers."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef decode(self, msg):\n _check_message_valid(msg)\n cmd = msg[2:4]\n decoder = getattr(self, '_{}_decode'.format(cmd.lower()), None)\n if not decoder:\n cmd = 'unknown'\n decoder = self._unknown_decode\n decoded_msg = decoder(msg)\n for handler in self._handlers.get(cmd, []):\n handler(**decoded_msg)", "response": "Decode an Elk message by passing to appropriate decoder"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _ee_decode(self, msg):\n return {'area': int(msg[4:5])-1, 'is_exit': msg[5:6] == '0',\n 'timer1': int(msg[6:9]), 'timer2': int(msg[9:12]),\n 'armed_status': msg[12:13]}", "response": "EE - > Entry / Exit timer report."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _ic_decode(self, msg):\n code = msg[4:16]\n if re.match(r'(0\\d){6}', code):\n code = re.sub(r'0(\\d)', r'\\1', code)\n return {'code': code, 'user': int(msg[16:19])-1,\n 'keypad': int(msg[19:21])-1}", "response": "IC: Send Valid Or Invalid User Code Format."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _ka_decode(self, msg):\n return {'keypad_areas': [ord(x)-0x31 for x in msg[4:4+Max.KEYPADS.value]]}", "response": "Decode a KA message."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndecoding the LW message into a dictionary.", "response": "def _lw_decode(self, msg):\n \"\"\"LW: temperatures from all keypads and zones 1-16.\"\"\"\n keypad_temps = []\n zone_temps = []\n for i in range(16):\n keypad_temps.append(int(msg[4+3*i:7+3*i]) - 40)\n zone_temps.append(int(msg[52+3*i:55+3*i]) - 60)\n return {'keypad_temps': keypad_temps, 'zone_temps': zone_temps}"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _sd_decode(self, msg):\n desc_ch1 = msg[9]\n show_on_keypad = ord(desc_ch1) >= 0x80\n if show_on_keypad:\n desc_ch1 = chr(ord(desc_ch1) & 0x7f)\n return {'desc_type': int(msg[4:6]), 'unit': int(msg[6:9])-1,\n 'desc': (desc_ch1+msg[10:25]).rstrip(),\n 'show_on_keypad': show_on_keypad}", "response": "SD : Description text."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _st_decode(self, msg):\n group = int(msg[4:5])\n temperature = int(msg[7:10])\n if group == 0:\n temperature -= 60\n elif group == 1:\n temperature -= 40\n return {'group': group, 'device': int(msg[5:7])-1,\n 'temperature': temperature}", "response": "ST : Temperature update."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndecoding the VN message into a dictionary.", "response": "def _vn_decode(self, msg):\n \"\"\"VN: Version information.\"\"\"\n elkm1_version = \"{}.{}.{}\".format(int(msg[4:6], 16), int(msg[6:8], 16),\n int(msg[8:10], 16))\n xep_version = \"{}.{}.{}\".format(int(msg[10:12], 16), int(msg[12:14], 16),\n int(msg[14:16], 16))\n return {'elkm1_version': elkm1_version, 'xep_version': xep_version}"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _zp_decode(self, msg):\n zone_partitions = [ord(x)-0x31 for x in msg[4:4+Max.ZONES.value]]\n return {'zone_partitions': zone_partitions}", "response": "ZP - > Zone partitions."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _zs_decode(self, msg):\n status = [_status_decode(int(x, 16)) for x in msg[4:4+Max.ZONES.value]]\n return {'zone_statuses': status}", "response": "ZS : Zone statuses."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef connect(self):\n sess_kwargs = {}\n if self._user is not None:\n sess_kwargs['user'] = self._user\n if self._password is not None:\n sess_kwargs['password'] = self._password\n self._login = xnat.connect(server=self._server, **sess_kwargs)", "response": "Connects to the XNAT server."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget a single file from the fileset and return a dict containing the information about the file.", "response": "def get_fileset(self, fileset):\n \"\"\"\n Caches a single fileset (if the 'path' attribute is accessed\n and it has not been previously cached for example\n\n Parameters\n ----------\n fileset : Fileset\n The fileset to cache\n prev_login : xnat.XNATSession\n An XNATSession object to use for the connection. A new\n one is created if one isn't provided\n\n Returns\n -------\n primary_path : str\n The path of the primary file once it has been cached\n aux_paths : dict[str, str]\n A dictionary containing a mapping of auxiliary file names to\n paths\n \"\"\"\n if fileset.format is None:\n raise ArcanaUsageError(\n \"Attempting to download {}, which has not been assigned a \"\n \"file format (see Fileset.formatted)\".format(fileset))\n self._check_repository(fileset)\n with self: # Connect to the XNAT repository if haven't already\n xsession = self.get_xsession(fileset)\n xscan = xsession.scans[fileset.name]\n # Set URI so we can retrieve checksums if required\n fileset.uri = xscan.uri\n fileset.id = xscan.id\n cache_path = self._cache_path(fileset)\n need_to_download = True\n if op.exists(cache_path):\n if self._check_md5:\n md5_path = cache_path + XnatRepo.MD5_SUFFIX\n try:\n with open(md5_path, 'r') as f:\n cached_checksums = json.load(f)\n if cached_checksums == fileset.checksums:\n need_to_download = False\n except IOError:\n pass\n else:\n need_to_download = False\n if need_to_download:\n# if fileset._resource_name is not None:\n xresource = xscan.resources[fileset._resource_name]\n# else:\n# xresources = []\n# for resource_name in fileset.format.xnat_resource_names:\n# try:\n# xresources.append(xscan.resources[resource_name])\n# except KeyError:\n# pass\n# if not xresources:\n# raise ArcanaError(\n# \"Could not find matching resource for {} ('{}') \"\n# \"in {}, available resources are '{}'\"\n# .format(\n# self.format,\n# \"', '\".join(\n# fileset.format.xnat_resource_names),\n# xscan.uri,\n# \"', '\".join(\n# r.label\n# for r in list(xscan.resources.values()))))\n# elif len(xresources) > 1:\n# logger.warning(\n# \"Found multiple acceptable resources for {}: {}\"\n# .format(fileset,\n# ', '.join(str(r) for r in xresources)))\n# xresource = xresources[0]\n # The path to the directory which the files will be\n # downloaded to.\n tmp_dir = cache_path + '.download'\n try:\n # Attempt to make tmp download directory. This will\n # fail if another process (or previous attempt) has\n # already created it. In that case this process will\n # wait to see if that download finishes successfully,\n # and if so use the cached version.\n os.mkdir(tmp_dir)\n except OSError as e:\n if e.errno == errno.EEXIST:\n # Another process may be concurrently downloading\n # the same file to the cache. Wait for\n # 'race_cond_delay' seconds and then check that it\n # has been completed or assume interrupted and\n # redownload.\n self._delayed_download(\n tmp_dir, xresource, xscan, fileset,\n xsession.label, cache_path,\n delay=self._race_cond_delay)\n else:\n raise\n else:\n self.download_fileset(\n tmp_dir, xresource, xscan, fileset,\n xsession.label, cache_path)\n shutil.rmtree(tmp_dir)\n if not fileset.format.directory:\n (primary_path, aux_paths) = fileset.format.assort_files(\n op.join(cache_path, f) for f in os.listdir(cache_path))\n else:\n primary_path = cache_path\n aux_paths = None\n return primary_path, aux_paths"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nretrieves the MD5 digests associated with the files in the file - set and returns the checksums as a dictionary.", "response": "def get_checksums(self, fileset):\n \"\"\"\n Downloads the MD5 digests associated with the files in the file-set.\n These are saved with the downloaded files in the cache and used to\n check if the files have been updated on the server\n\n Parameters\n ----------\n resource : xnat.ResourceCatalog\n The xnat resource\n file_format : FileFormat\n The format of the fileset to get the checksums for. Used to\n determine the primary file within the resource and change the\n corresponding key in the checksums dictionary to '.' to match\n the way it is generated locally by Arcana.\n \"\"\"\n if fileset.uri is None:\n raise ArcanaUsageError(\n \"Can't retrieve checksums as URI has not been set for {}\"\n .format(fileset))\n with self:\n checksums = {r['Name']: r['digest']\n for r in self._login.get_json(fileset.uri + '/files')[\n 'ResultSet']['Result']}\n if not fileset.format.directory:\n # Replace the key corresponding to the primary file with '.' to\n # match the way that checksums are created by Arcana\n primary = fileset.format.assort_files(checksums.keys())[0]\n checksums['.'] = checksums.pop(primary)\n return checksums"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef find_data(self, subject_ids=None, visit_ids=None, **kwargs):\n subject_ids = self.convert_subject_ids(subject_ids)\n # Add derived visit IDs to list of visit ids to filter\n all_filesets = []\n all_fields = []\n all_records = []\n # Note we prefer the use of raw REST API calls here for performance\n # reasons over using XnatPy's data structures.\n with self:\n # Get map of internal subject IDs to subject labels in project\n subject_xids_to_labels = {\n s['ID']: s['label'] for s in self._login.get_json(\n '/data/projects/{}/subjects'.format(self.project_id))[\n 'ResultSet']['Result']}\n # Get list of all sessions within project\n session_xids = [\n s['ID'] for s in self._login.get_json(\n '/data/projects/{}/experiments'.format(self.project_id))[\n 'ResultSet']['Result']\n if (self.session_filter is None or\n self.session_filter.match(s['label']))]\n for session_xid in session_xids:\n session_json = self._login.get_json(\n '/data/projects/{}/experiments/{}'.format(\n self.project_id, session_xid))['items'][0]\n subject_xid = session_json['data_fields']['subject_ID']\n subject_id = subject_xids_to_labels[subject_xid]\n session_label = session_json['data_fields']['label']\n session_uri = (\n '/data/archive/projects/{}/subjects/{}/experiments/{}'\n .format(self.project_id, subject_xid, session_xid))\n # Get field values. We do this first so we can check for the\n # DERIVED_FROM_FIELD to determine the correct session label and\n # study name\n field_values = {}\n try:\n fields_json = next(\n c['items'] for c in session_json['children']\n if c['field'] == 'fields/field')\n except StopIteration:\n pass\n else:\n for js in fields_json:\n try:\n value = js['data_fields']['field']\n except KeyError:\n pass\n else:\n field_values[js['data_fields']['name']] = value\n # Extract study name and derived-from session\n if self.DERIVED_FROM_FIELD in field_values:\n df_sess_label = field_values.pop(self.DERIVED_FROM_FIELD)\n from_study = session_label[len(df_sess_label) + 1:]\n session_label = df_sess_label\n else:\n from_study = None\n # Strip subject ID from session label if required\n if session_label.startswith(subject_id + '_'):\n visit_id = session_label[len(subject_id) + 1:]\n else:\n visit_id = session_label\n # Strip project ID from subject ID if required\n if subject_id.startswith(self.project_id + '_'):\n subject_id = subject_id[len(self.project_id) + 1:]\n # Check subject is summary or not and whether it is to be\n # filtered\n if subject_id == XnatRepo.SUMMARY_NAME:\n subject_id = None\n elif not (subject_ids is None or subject_id in subject_ids):\n continue\n # Check visit is summary or not and whether it is to be\n # filtered\n if visit_id == XnatRepo.SUMMARY_NAME:\n visit_id = None\n elif not (visit_ids is None or visit_id in visit_ids):\n continue\n # Determine frequency\n if (subject_id, visit_id) == (None, None):\n frequency = 'per_study'\n elif visit_id is None:\n frequency = 'per_subject'\n elif subject_id is None:\n frequency = 'per_visit'\n else:\n frequency = 'per_session'\n # Append fields\n for name, value in field_values.items():\n value = value.replace('"', '\"')\n all_fields.append(Field(\n name=name, value=value, repository=self,\n frequency=frequency,\n subject_id=subject_id,\n visit_id=visit_id,\n from_study=from_study,\n **kwargs))\n # Extract part of JSON relating to files\n try:\n scans_json = next(\n c['items'] for c in session_json['children']\n if c['field'] == 'scans/scan')\n except StopIteration:\n scans_json = []\n for scan_json in scans_json:\n scan_id = scan_json['data_fields']['ID']\n scan_type = scan_json['data_fields'].get('type', '')\n scan_uri = '{}/scans/{}'.format(session_uri, scan_id)\n try:\n resources_json = next(\n c['items'] for c in scan_json['children']\n if c['field'] == 'file')\n except StopIteration:\n resources = {}\n else:\n resources = {js['data_fields']['label']:\n js['data_fields'].get('format', None)\n for js in resources_json}\n # Remove auto-generated snapshots directory\n resources.pop('SNAPSHOTS', None)\n if scan_type == self.PROV_SCAN:\n # Download provenance JSON files and parse into\n # records\n temp_dir = tempfile.mkdtemp()\n try:\n with tempfile.TemporaryFile() as temp_zip:\n self._login.download_stream(\n scan_uri + '/files', temp_zip,\n format='zip')\n with ZipFile(temp_zip) as zip_file:\n zip_file.extractall(temp_dir)\n for base_dir, _, fnames in os.walk(temp_dir):\n for fname in fnames:\n if fname.endswith('.json'):\n pipeline_name = fname[:-len('.json')]\n json_path = op.join(base_dir, fname)\n all_records.append(\n Record.load(\n pipeline_name, frequency,\n subject_id, visit_id,\n from_study, json_path))\n finally:\n shutil.rmtree(temp_dir, ignore_errors=True)\n else:\n for resource in resources:\n all_filesets.append(Fileset(\n scan_type, id=scan_id, uri=scan_uri,\n repository=self, frequency=frequency,\n subject_id=subject_id, visit_id=visit_id,\n from_study=from_study,\n resource_name=resource, **kwargs))\n logger.debug(\"Found node {}:{} on {}:{}\".format(\n subject_id, visit_id, self.server, self.project_id))\n return all_filesets, all_fields, all_records", "response": "Find all filesets fields and provenance records within an XNAT project."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting subject ids to strings if they are integers", "response": "def convert_subject_ids(self, subject_ids):\n \"\"\"\n Convert subject ids to strings if they are integers\n \"\"\"\n # TODO: need to make this generalisable via a\n # splitting+mapping function passed to the repository\n if subject_ids is not None:\n subject_ids = set(\n ('{:03d}'.format(s)\n if isinstance(s, int) else s) for s in subject_ids)\n return subject_ids"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the XNAT session and cache dir corresponding to the item.", "response": "def get_xsession(self, item):\n \"\"\"\n Returns the XNAT session and cache dir corresponding to the\n item.\n \"\"\"\n subj_label, sess_label = self._get_item_labels(item)\n with self:\n xproject = self._login.projects[self.project_id]\n try:\n xsubject = xproject.subjects[subj_label]\n except KeyError:\n xsubject = self._login.classes.SubjectData(\n label=subj_label, parent=xproject)\n try:\n xsession = xsubject.experiments[sess_label]\n except KeyError:\n xsession = self._login.classes.MrSessionData(\n label=sess_label, parent=xsubject)\n if item.derived:\n xsession.fields[\n self.DERIVED_FROM_FIELD] = self._get_item_labels(\n item, no_from_study=True)[1]\n return xsession"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_item_labels(self, item, no_from_study=False):\n subject_id = self.inv_map_subject_id(item.subject_id)\n visit_id = self.inv_map_visit_id(item.visit_id)\n subj_label, sess_label = self._get_labels(\n item.frequency, subject_id, visit_id)\n if not no_from_study and item.from_study is not None:\n sess_label += '_' + item.from_study\n return (subj_label, sess_label)", "response": "Returns the subject and session labels given the frequency and IDs of the items."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the labels for the XNAT subject and session given the frequency and provided IDs.", "response": "def _get_labels(self, frequency, subject_id=None, visit_id=None):\n \"\"\"\n Returns the labels for the XNAT subject and sessions given\n the frequency and provided IDs.\n \"\"\"\n if frequency == 'per_session':\n subj_label = '{}_{}'.format(self.project_id,\n subject_id)\n sess_label = '{}_{}_{}'.format(self.project_id,\n subject_id,\n visit_id)\n elif frequency == 'per_subject':\n subj_label = '{}_{}'.format(self.project_id,\n subject_id)\n sess_label = '{}_{}_{}'.format(self.project_id,\n subject_id,\n self.SUMMARY_NAME)\n elif frequency == 'per_visit':\n subj_label = '{}_{}'.format(self.project_id,\n self.SUMMARY_NAME)\n sess_label = '{}_{}_{}'.format(self.project_id,\n self.SUMMARY_NAME,\n visit_id)\n elif frequency == 'per_study':\n subj_label = '{}_{}'.format(self.project_id,\n self.SUMMARY_NAME)\n sess_label = '{}_{}_{}'.format(self.project_id,\n self.SUMMARY_NAME,\n self.SUMMARY_NAME)\n else:\n assert False\n return (subj_label, sess_label)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef turn_on(self, time):\n self._elk.send(cn_encode(self._index, time))", "response": "Turn on an output"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef sync(self):\n self.elk.send(cs_encode())\n self.get_descriptions(TextDescriptions.OUTPUT.value)", "response": "Retrieve areas from ElkM1"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef detect_version_str(self):\n help_text = run_matlab_cmd(\"help('{}')\".format(self.test_func))\n if not help_text:\n raise ArcanaRequirementNotFoundError(\n \"Did not find test function '{}' for {}\"\n .format(self.test_func, self))\n return self.parse_help_text(help_text)", "response": "Try to detect version of package from command help text."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef target_heating_level(self):\n try:\n if self.side == 'left':\n level = self.device.device_data['leftTargetHeatingLevel']\n elif self.side == 'right':\n level = self.device.device_data['rightTargetHeatingLevel']\n return level\n except TypeError:\n return None", "response": "Return target heating level."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef past_heating_level(self, num):\n if num > 9:\n return 0\n\n try:\n if self.side == 'left':\n level = self.device.device_data_history[\n num]['leftHeatingLevel']\n elif self.side == 'right':\n level = self.device.device_data_history[\n num]['rightHeatingLevel']\n return level\n except TypeError:\n return 0", "response": "Return a heating level from the past."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef now_heating(self):\n try:\n if self.side == 'left':\n heat = self.device.device_data['leftNowHeating']\n elif self.side == 'right':\n heat = self.device.device_data['rightNowHeating']\n return heat\n except TypeError:\n return None", "response": "Return current heating state."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn seconds of heat time remaining.", "response": "def heating_remaining(self):\n \"\"\"Return seconds of heat time remaining.\"\"\"\n try:\n if self.side == 'left':\n timerem = self.device.device_data['leftHeatingDuration']\n elif self.side == 'right':\n timerem = self.device.device_data['rightHeatingDuration']\n return timerem\n except TypeError:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn last seen time.", "response": "def last_seen(self):\n \"\"\"Return mattress last seen time.\"\"\"\n \"\"\"\n These values seem to be rarely updated correctly in the API.\n Don't expect accurate results from this property.\n \"\"\"\n try:\n if self.side == 'left':\n lastseen = self.device.device_data['leftPresenceEnd']\n elif self.side == 'right':\n lastseen = self.device.device_data['rightPresenceEnd']\n\n date = datetime.fromtimestamp(int(lastseen)) \\\n .strftime('%Y-%m-%dT%H:%M:%S')\n return date\n except TypeError:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a dict of all the current heating values.", "response": "def heating_values(self):\n \"\"\"Return a dict of all the current heating values.\"\"\"\n heating_dict = {\n 'level': self.heating_level,\n 'target': self.target_heating_level,\n 'active': self.now_heating,\n 'remaining': self.heating_remaining,\n 'last_seen': self.last_seen,\n }\n return heating_dict"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef current_sleep_stage(self):\n try:\n stages = self.intervals[0]['stages']\n num_stages = len(stages)\n\n if num_stages == 0:\n return None\n\n # API now always has an awake state last in the dict\n # so always pull the second to last stage while we are\n # in a processing state\n if self.current_session_processing:\n stage = stages[num_stages-2]['stage']\n else:\n stage = stages[num_stages-1]['stage']\n\n # UNRELIABLE... Removing for now.\n # Check sleep stage against last_seen time to make\n # sure we don't get stuck in a non-awake state.\n #delta_elap = datetime.fromtimestamp(time.time()) \\\n # - datetime.strptime(self.last_seen, '%Y-%m-%dT%H:%M:%S')\n #_LOGGER.debug('User elap: %s', delta_elap.total_seconds())\n #if stage != 'awake' and delta_elap.total_seconds() > 1800:\n # Bed hasn't seen us for 30min so set awake.\n # stage = 'awake'\n except KeyError:\n stage = None\n return stage", "response": "Return sleep stage for in - progress session."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef current_sleep_breakdown(self):\n try:\n stages = self.intervals[0]['stages']\n breakdown = {'awake': 0, 'light': 0, 'deep': 0, 'rem': 0}\n for stage in stages:\n if stage['stage'] == 'awake':\n breakdown['awake'] += stage['duration']\n elif stage['stage'] == 'light':\n breakdown['light'] += stage['duration']\n elif stage['stage'] == 'deep':\n breakdown['deep'] += stage['duration']\n elif stage['stage'] == 'rem':\n breakdown['rem'] += stage['duration']\n except KeyError:\n breakdown = None\n return breakdown", "response": "Return durations of sleep stages for in - progress session."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef current_bed_temp(self):\n try:\n bedtemps = self.intervals[0]['timeseries']['tempBedC']\n num_temps = len(bedtemps)\n\n if num_temps == 0:\n return None\n\n bedtemp = bedtemps[num_temps-1][1]\n except KeyError:\n bedtemp = None\n return bedtemp", "response": "Return current bed temperature for in - progress session."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns current room temperature for in - progress session.", "response": "def current_room_temp(self):\n \"\"\"Return current room temperature for in-progress session.\"\"\"\n try:\n rmtemps = self.intervals[0]['timeseries']['tempRoomC']\n num_temps = len(rmtemps)\n\n if num_temps == 0:\n return None\n\n rmtemp = rmtemps[num_temps-1][1]\n except KeyError:\n rmtemp = None\n return rmtemp"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef current_resp_rate(self):\n try:\n rates = self.intervals[0]['timeseries']['respiratoryRate']\n num_rates = len(rates)\n\n if num_rates == 0:\n return None\n\n rate = rates[num_rates-1][1]\n except KeyError:\n rate = None\n return rate", "response": "Return current respiratory rate for in - progress session."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning current heart rate for in - progress session.", "response": "def current_heart_rate(self):\n \"\"\"Return current heart rate for in-progress session.\"\"\"\n try:\n rates = self.intervals[0]['timeseries']['heartRate']\n num_rates = len(rates)\n\n if num_rates == 0:\n return None\n\n rate = rates[num_rates-1][1]\n except KeyError:\n rate = None\n return rate"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef current_values(self):\n current_dict = {\n 'date': self.current_session_date,\n 'score': self.current_sleep_score,\n 'stage': self.current_sleep_stage,\n 'breakdown': self.current_sleep_breakdown,\n 'tnt': self.current_tnt,\n 'bed_temp': self.current_bed_temp,\n 'room_temp': self.current_room_temp,\n 'resp_rate': self.current_resp_rate,\n 'heart_rate': self.current_heart_rate,\n 'processing': self.current_session_processing,\n }\n return current_dict", "response": "Return a dict of all the current parameters."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns date and time for start of last session data.", "response": "def last_session_date(self):\n \"\"\"Return date/time for start of last session data.\"\"\"\n try:\n date = self.intervals[1]['ts']\n except KeyError:\n return None\n date_f = datetime.strptime(date, '%Y-%m-%dT%H:%M:%S.%fZ')\n now = time.time()\n offset = datetime.fromtimestamp(now) - datetime.utcfromtimestamp(now)\n return date_f + offset"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning durations of sleep stages for last complete session.", "response": "def last_sleep_breakdown(self):\n \"\"\"Return durations of sleep stages for last complete session.\"\"\"\n try:\n stages = self.intervals[1]['stages']\n except KeyError:\n return None\n\n breakdown = {'awake': 0, 'light': 0, 'deep': 0, 'rem': 0}\n for stage in stages:\n if stage['stage'] == 'awake':\n breakdown['awake'] += stage['duration']\n elif stage['stage'] == 'light':\n breakdown['light'] += stage['duration']\n elif stage['stage'] == 'deep':\n breakdown['deep'] += stage['duration']\n elif stage['stage'] == 'rem':\n breakdown['rem'] += stage['duration']\n return breakdown"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn avg bed temperature for last session.", "response": "def last_bed_temp(self):\n \"\"\"Return avg bed temperature for last session.\"\"\"\n try:\n bedtemps = self.intervals[1]['timeseries']['tempBedC']\n except KeyError:\n return None\n tmp = 0\n num_temps = len(bedtemps)\n\n if num_temps == 0:\n return None\n\n for temp in bedtemps:\n tmp += temp[1]\n bedtemp = tmp/num_temps\n return bedtemp"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn avg room temperature for last session.", "response": "def last_room_temp(self):\n \"\"\"Return avg room temperature for last session.\"\"\"\n try:\n rmtemps = self.intervals[1]['timeseries']['tempRoomC']\n except KeyError:\n return None\n tmp = 0\n num_temps = len(rmtemps)\n\n if num_temps == 0:\n return None\n\n for temp in rmtemps:\n tmp += temp[1]\n rmtemp = tmp/num_temps\n return rmtemp"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the average heart rate for last session.", "response": "def last_heart_rate(self):\n \"\"\"Return avg heart rate for last session.\"\"\"\n try:\n rates = self.intervals[1]['timeseries']['heartRate']\n except KeyError:\n return None\n tmp = 0\n num_rates = len(rates)\n\n if num_rates == 0:\n return None\n\n for rate in rates:\n tmp += rate[1]\n rateavg = tmp/num_rates\n return rateavg"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef last_values(self):\n last_dict = {\n 'date': self.last_session_date,\n 'score': self.last_sleep_score,\n 'breakdown': self.last_sleep_breakdown,\n 'tnt': self.last_tnt,\n 'bed_temp': self.last_bed_temp,\n 'room_temp': self.last_room_temp,\n 'resp_rate': self.last_resp_rate,\n 'heart_rate': self.last_heart_rate,\n 'processing': self.last_session_processing,\n }\n return last_dict", "response": "Return a dict of all the last parameters."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncalculates some heating data stats.", "response": "def heating_stats(self):\n \"\"\"Calculate some heating data stats.\"\"\"\n local_5 = []\n local_10 = []\n\n for i in range(0, 10):\n level = self.past_heating_level(i)\n if level == 0:\n _LOGGER.debug('Cant calculate stats yet...')\n return\n if i < 5:\n local_5.append(level)\n local_10.append(level)\n\n _LOGGER.debug('%s Heating History: %s', self.side, local_10)\n\n try:\n # Average of 5min on the history dict.\n fiveminavg = statistics.mean(local_5)\n tenminavg = statistics.mean(local_10)\n _LOGGER.debug('%s Heating 5 min avg: %s', self.side, fiveminavg)\n _LOGGER.debug('%s Heating 10 min avg: %s', self.side, tenminavg)\n\n # Standard deviation\n fivestdev = statistics.stdev(local_5)\n tenstdev = statistics.stdev(local_10)\n _LOGGER.debug('%s Heating 5 min stdev: %s', self.side, fivestdev)\n _LOGGER.debug('%s Heating 10 min stdev: %s', self.side, tenstdev)\n\n # Variance\n fivevar = statistics.variance(local_5)\n tenvar = statistics.variance(local_10)\n _LOGGER.debug('%s Heating 5 min variance: %s', self.side, fivevar)\n _LOGGER.debug('%s Heating 10 min variance: %s', self.side, tenvar)\n except:\n _LOGGER.debug('Cant calculate stats yet...')"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndetermines presence based on the heating level and end presence of the resource.", "response": "def dynamic_presence(self):\n \"\"\"\n Determine presence based on bed heating level and end presence\n time reported by the api.\n\n Idea originated from Alex Lee Yuk Cheung SmartThings Code.\n \"\"\"\n\n # self.heating_stats()\n\n if not self.presence:\n if self.heating_level > 50:\n # Can likely make this better\n if not self.now_heating:\n self.presence = True\n elif self.heating_level - self.target_heating_level >= 8:\n self.presence = True\n elif self.heating_level > 25:\n # Catch rising edge\n if self.past_heating_level(0) - self.past_heating_level(1) >= 2 \\\n and self.past_heating_level(1) - self.past_heating_level(2) >= 2 \\\n and self.past_heating_level(2) - self.past_heating_level(3) >= 2:\n # Values are increasing so we are likely in bed\n if not self.now_heating:\n self.presence = True\n elif self.heating_level - self.target_heating_level >= 8:\n self.presence = True\n\n elif self.presence:\n if self.heating_level <= 15:\n # Failsafe, very slow\n self.presence = False\n elif self.heating_level < 50:\n if self.past_heating_level(0) - self.past_heating_level(1) < 0 \\\n and self.past_heating_level(1) - self.past_heating_level(2) < 0 \\\n and self.past_heating_level(2) - self.past_heating_level(3) < 0:\n # Values are decreasing so we are likely out of bed\n self.presence = False\n\n # Last seen can lag real-time by up to 35min so this is\n # mostly a backup to using the heat values.\n # seen_delta = datetime.fromtimestamp(time.time()) \\\n # - datetime.strptime(self.last_seen, '%Y-%m-%dT%H:%M:%S')\n # _LOGGER.debug('%s Last seen time delta: %s', self.side,\n # seen_delta.total_seconds())\n # if self.presence and seen_delta.total_seconds() > 2100:\n # self.presence = False\n\n _LOGGER.debug('%s Presence Results: %s', self.side, self.presence)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\nasync def set_heating_level(self, level, duration=0):\n url = '{}/devices/{}'.format(API_URL, self.device.deviceid)\n\n # Catch bad inputs\n level = 10 if level < 10 else level\n level = 100 if level > 100 else level\n\n if self.side == 'left':\n data = {\n 'leftHeatingDuration': duration,\n 'leftTargetHeatingLevel': level\n }\n elif self.side == 'right':\n data = {\n 'rightHeatingDuration': duration,\n 'rightTargetHeatingLevel': level\n }\n\n set_heat = await self.device.api_put(url, data)\n if set_heat is None:\n _LOGGER.error('Unable to set eight heating level.')\n else:\n # Standard device json is returned after setting\n self.device.handle_device_json(set_heat['device'])", "response": "Update heating data json."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\nasync def update_trend_data(self, startdate, enddate):\n url = '{}/users/{}/trends'.format(API_URL, self.userid)\n params = {\n 'tz': self.device.tzone,\n 'from': startdate,\n 'to': enddate\n }\n\n trends = await self.device.api_get(url, params)\n if trends is None:\n _LOGGER.error('Unable to fetch eight trend data.')\n else:\n self.trends = trends['days']", "response": "Update trends data json for specified time period."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\nasync def update_intervals_data(self):\n url = '{}/users/{}/intervals'.format(API_URL, self.userid)\n\n intervals = await self.device.api_get(url)\n if intervals is None:\n _LOGGER.error('Unable to fetch eight intervals data.')\n else:\n self.intervals = intervals['intervals']", "response": "Update intervals data json for specified time period."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef save(obj=None, m2m_data=None):\n m2m_data = {} if m2m_data is None else m2m_data\n obj.save_base(raw=True)\n for attr, values in m2m_data.items():\n for value in values:\n getattr(obj, attr).add(value)", "response": "Saves a deserialized model object."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef custom_parser(self, json_text=None):\n app_config = django_apps.get_app_config(\"django_collect_offline\")\n for json_parser in app_config.custom_json_parsers:\n json_text = json_parser(json_text)\n return json_text", "response": "Runs json_text thru custom parsers."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets channel lazy property", "response": "def channel(self):\n \"\"\" channel lazy property \"\"\"\n # start shell, non-blocking channel\n chan = self.cli.invoke_shell(width=360, height=80)\n chan.setblocking(0)\n # set channel timeout\n chan.settimeout(10)\n # now waiting for shell prompt ('username@')\n result = \"\"\n count = 0\n while count < 10:\n try:\n recv_part = chan.recv(16384).decode()\n result += recv_part\n except socket.timeout:\n # socket.timeout here means 'no more data'\n pass\n\n if result.find('%s@' % self.username) != -1:\n return chan\n time.sleep(1)\n count += 1\n # failed to get shell prompt on channel :-(\n raise StitchesConnectionException(\"Failed to get shell prompt\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndisconnect from the internal state of the object.", "response": "def disconnect(self):\n \"\"\"\n Close the connection\n \"\"\"\n if hasattr(self, '_lazy_sftp'):\n if self.sftp is not None:\n self.sftp.close()\n delattr(self, '_lazy_sftp')\n if hasattr(self, '_lazy_channel'):\n if self.channel is not None:\n self.channel.close()\n delattr(self, '_lazy_channel')\n if hasattr(self, '_lazy_cli'):\n if self.cli is not None:\n self.cli.close()\n delattr(self, '_lazy_cli')\n if hasattr(self, '_lazy_pbm'):\n if self.pbm is not None:\n self.pbm.close()\n delattr(self, '_lazy_pbm')\n if hasattr(self, '_lazy_rpyc'):\n if self.rpyc is not None:\n self.rpyc.close()\n delattr(self, '_lazy_rpyc')"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nexecute a command in the connection and return the stdout and stderr of the command.", "response": "def exec_command(self, command, bufsize=-1, get_pty=False):\n \"\"\"\n Execute a command in the connection\n\n @param command: command to execute\n @type command: str\n\n @param bufsize: buffer size\n @type bufsize: int\n\n @param get_pty: get pty\n @type get_pty: bool\n\n @return: the stdin, stdout, and stderr of the executing command\n @rtype: tuple(L{paramiko.ChannelFile}, L{paramiko.ChannelFile},\n L{paramiko.ChannelFile})\n\n @raise SSHException: if the server fails to execute the command\n \"\"\"\n self.last_command = command\n return self.cli.exec_command(command, bufsize, get_pty=get_pty)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nexecutes a command and get its return value", "response": "def recv_exit_status(self, command, timeout=10, get_pty=False):\n \"\"\"\n Execute a command and get its return value\n\n @param command: command to execute\n @type command: str\n\n @param timeout: command execution timeout\n @type timeout: int\n\n @param get_pty: get pty\n @type get_pty: bool\n\n @return: the exit code of the process or None in case of timeout\n @rtype: int or None\n \"\"\"\n status = None\n self.last_command = command\n stdin, stdout, stderr = self.cli.exec_command(command, get_pty=get_pty)\n if stdout and stderr and stdin:\n for _ in range(timeout):\n if stdout.channel.exit_status_ready():\n status = stdout.channel.recv_exit_status()\n break\n time.sleep(1)\n\n self.last_stdout = stdout.read()\n self.last_stderr = stderr.read()\n\n stdin.close()\n stdout.close()\n stderr.close()\n return status"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set(self, value):\n self._elk.send(cx_encode(self._index, value))", "response": "Set the value of the current counter."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef save(self, path):\n with open(path, 'w') as f:\n try:\n json.dump(self.prov, f, indent=2)\n except TypeError:\n raise ArcanaError(\n \"Could not serialise provenance record dictionary:\\n{}\"\n .format(pformat(self.prov)))", "response": "Saves the provenance object to a JSON file optionally including checksums for inputs and outputs."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef load(cls, pipeline_name, frequency, subject_id, visit_id, from_study,\n path):\n \"\"\"\n Loads a saved provenance object from a JSON file\n\n Parameters\n ----------\n path : str\n Path to the provenance file\n frequency : str\n The frequency of the record\n subject_id : str | None\n The subject ID of the provenance record\n visit_id : str | None\n The visit ID of the provenance record\n from_study : str\n Name of the study the derivatives were created for\n\n Returns\n -------\n record : Record\n The loaded provenance record\n \"\"\"\n with open(path) as f:\n prov = json.load(f)\n return Record(pipeline_name, frequency, subject_id, visit_id,\n from_study, prov)", "response": "Loads a saved provenance object from a JSON file."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef mismatches(self, other, include=None, exclude=None):\n if include is not None:\n include_res = [self._gen_prov_path_regex(p) for p in include]\n if exclude is not None:\n exclude_res = [self._gen_prov_path_regex(p) for p in exclude]\n diff = DeepDiff(self._prov, other._prov, ignore_order=True)\n # Create regular expresssions for the include and exclude paths in\n # the format that deepdiff uses for nested dictionary/lists\n\n def include_change(change):\n if include is None:\n included = True\n else:\n included = any(rx.match(change) for rx in include_res)\n if included and exclude is not None:\n included = not any(rx.match(change) for rx in exclude_res)\n return included\n\n filtered_diff = {}\n for change_type, changes in diff.items():\n if isinstance(changes, dict):\n filtered = dict((k, v) for k, v in changes.items()\n if include_change(k))\n else:\n filtered = [c for c in changes if include_change(c)]\n if filtered:\n filtered_diff[change_type] = filtered\n return filtered_diff", "response": "Compares two provenance objects and returns a dictionary of differences between those objects."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef translate(cls, substudy_name, pipeline_getter, auto_added=False):\n assert isinstance(substudy_name, basestring)\n assert isinstance(pipeline_getter, basestring)\n\n def translated_getter(self, **name_maps):\n substudy_spec = self.substudy_spec(substudy_name)\n # Combine mapping of names of sub-study specs with\n return getattr(self.substudy(substudy_name), pipeline_getter)(\n prefix=substudy_name + '_',\n input_map=substudy_spec.name_map,\n output_map=substudy_spec.name_map,\n study=self, name_maps=name_maps)\n # Add reduce method to allow it to be pickled\n translated_getter.auto_added = auto_added\n return translated_getter", "response": "Returns a method that returns a new method that calls the pipeline_getter with the given name_maps and returns the new method that returns the new instance of the class with the translated name_maps."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning an iterator over the data specs that are not explicitly provided by the name map.", "response": "def auto_data_specs(self):\n \"\"\"\n Data specs in the sub-study class that are not explicitly provided\n in the name map\n \"\"\"\n for spec in self.study_class.data_specs():\n if spec.name not in self._name_map:\n yield spec"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _make_nodes(self, cwd=None):\n for i, node in NipypeMapNode._make_nodes(self, cwd=cwd):\n # \"Cast\" NiPype node to a Arcana Node and set Arcana Node\n # parameters\n node.__class__ = self.node_cls\n node._environment = self._environment\n node._versions = self._versions\n node._wall_time = self._wall_time\n node._annotations = self._annotations\n yield i, node", "response": "Cast generated nodes to Arcana Nodes"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_query(query_name):\n query_file_match = list(\n filter(lambda i: query_name == i.stem, FLAT_QUERIES))\n if not query_file_match:\n return None\n # TODO: Log warning if more than one match\n query_file = query_file_match[0]\n with open(query_file) as f:\n metadata, query_body = frontmatter.parse(f.read())\n result_mod = query_file.suffix.strip('.')\n query_obj = SimpleNamespace(\n name=query_name,\n metadata=metadata,\n path=query_file,\n result_mod=result_mod,\n body=query_body,\n error=False,\n executed=datetime.utcnow().isoformat())\n return query_obj", "response": "Find file matching query_name read and return query object\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncall get_query then submit query to resultset module", "response": "def get_result_set(query_name, **kwargs):\n \"\"\" Call get_query, then submit query from file to resultset module\n \"\"\"\n query = get_query(query_name)\n if not query:\n query = SimpleNamespace()\n query.error = f\"No query found matching '{query_name}'\"\n return query\n try:\n result_mod = import_module(\n f'nerium.contrib.resultset.{query.result_mod}')\n except ModuleNotFoundError:\n result_mod = import_module('nerium.resultset.sql')\n query.params = {**kwargs}\n query.body = process_template(sql=query.body, **query.params)\n result = result_mod.result(query, **query.params)\n # Dumping and reloading via json here gets us datetime and decimal\n # serialization handling courtesy of `tablib`\n query.result = json.loads(json.dumps(result, default=serialize_objects_handler))\n try:\n if 'error' in query.result[0].keys():\n query.error = query.result[0]['error']\n except IndexError:\n pass\n return query"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngenerates CSV from result set", "response": "def results_to_csv(query_name, **kwargs):\n \"\"\" Generate CSV from result data\n \"\"\"\n query = get_result_set(query_name, **kwargs)\n result = query.result\n columns = list(result[0].keys())\n data = [tuple(row.values()) for row in result]\n frame = tablib.Dataset()\n frame.headers = columns\n for row in data:\n frame.append(row)\n csvs = frame.export('csv')\n return csvs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nregister with app_label. modelname wrapper_cls.", "response": "def register(self, models=None, wrapper_cls=None):\n \"\"\"Registers with app_label.modelname, wrapper_cls.\n \"\"\"\n self.loaded = True\n for model in models:\n model = model.lower()\n if model not in self.registry:\n self.registry.update({model: wrapper_cls or self.wrapper_cls})\n if self.register_historical:\n historical_model = \".historical\".join(model.split(\".\"))\n self.registry.update(\n {historical_model: wrapper_cls or self.wrapper_cls}\n )\n else:\n raise AlreadyRegistered(f\"Model is already registered. Got {model}.\")"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef register_for_app(\n self, app_label=None, exclude_models=None, exclude_model_classes=None\n ):\n \"\"\"Registers all models for this app_label.\n \"\"\"\n models = []\n exclude_models = exclude_models or []\n app_config = django_apps.get_app_config(app_label)\n for model in app_config.get_models():\n if model._meta.label_lower in exclude_models:\n pass\n elif exclude_model_classes and issubclass(model, exclude_model_classes):\n pass\n else:\n models.append(model._meta.label_lower)\n self.register(models)", "response": "Registers all models for this app_label."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a wrapped model instance.", "response": "def get_wrapped_instance(self, instance=None):\n \"\"\"Returns a wrapped model instance.\n \"\"\"\n if instance._meta.label_lower not in self.registry:\n raise ModelNotRegistered(f\"{repr(instance)} is not registered with {self}.\")\n wrapper_cls = self.registry.get(instance._meta.label_lower) or self.wrapper_cls\n if wrapper_cls:\n return wrapper_cls(instance)\n return instance"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef site_models(self, app_label=None):\n site_models = {}\n app_configs = (\n django_apps.get_app_configs()\n if app_label is None\n else [django_apps.get_app_config(app_label)]\n )\n for app_config in app_configs:\n model_list = [\n model\n for model in app_config.get_models()\n if model._meta.label_lower in self.registry\n ]\n if model_list:\n model_list.sort(key=lambda m: m._meta.verbose_name)\n site_models.update({app_config.name: model_list})\n return site_models", "response": "Returns a dictionary of registered models."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_fileset(self, fileset):\n # Don't need to cache fileset as it is already local as long\n # as the path is set\n if fileset._path is None:\n primary_path = self.fileset_path(fileset)\n aux_files = fileset.format.default_aux_file_paths(primary_path)\n if not op.exists(primary_path):\n raise ArcanaMissingDataException(\n \"{} does not exist in {}\"\n .format(fileset, self))\n for aux_name, aux_path in aux_files.items():\n if not op.exists(aux_path):\n raise ArcanaMissingDataException(\n \"{} is missing '{}' side car in {}\"\n .format(fileset, aux_name, self))\n else:\n primary_path = fileset.path\n aux_files = fileset.aux_files\n return primary_path, aux_files", "response": "Get the path of the fileset from the repository."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting the value of the field from the repository", "response": "def get_field(self, field):\n \"\"\"\n Update the value of the field from the repository\n \"\"\"\n # Load fields JSON, locking to prevent read/write conflicts\n # Would be better if only checked if locked to allow\n # concurrent reads but not possible with multi-process\n # locks (in my understanding at least).\n fpath = self.fields_json_path(field)\n try:\n with InterProcessLock(fpath + self.LOCK_SUFFIX,\n logger=logger), open(fpath, 'r') as f:\n dct = json.load(f)\n val = dct[field.name]\n if field.array:\n val = [field.dtype(v) for v in val]\n else:\n val = field.dtype(val)\n except (KeyError, IOError) as e:\n try:\n # Check to see if the IOError wasn't just because of a\n # missing file\n if e.errno != errno.ENOENT:\n raise\n except AttributeError:\n pass\n raise ArcanaMissingDataException(\n \"{} does not exist in the local repository {}\"\n .format(field.name, self))\n return val"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ninserting or updates a fileset in the repository.", "response": "def put_fileset(self, fileset):\n \"\"\"\n Inserts or updates a fileset in the repository\n \"\"\"\n target_path = self.fileset_path(fileset)\n if op.isfile(fileset.path):\n shutil.copyfile(fileset.path, target_path)\n # Copy side car files into repository\n for aux_name, aux_path in fileset.format.default_aux_file_paths(\n target_path).items():\n shutil.copyfile(self.aux_file[aux_name], aux_path)\n elif op.isdir(fileset.path):\n if op.exists(target_path):\n shutil.rmtree(target_path)\n shutil.copytree(fileset.path, target_path)\n else:\n assert False"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ninsert or updates a field in the repository", "response": "def put_field(self, field):\n \"\"\"\n Inserts or updates a field in the repository\n \"\"\"\n fpath = self.fields_json_path(field)\n # Open fields JSON, locking to prevent other processes\n # reading or writing\n with InterProcessLock(fpath + self.LOCK_SUFFIX, logger=logger):\n try:\n with open(fpath, 'r') as f:\n dct = json.load(f)\n except IOError as e:\n if e.errno == errno.ENOENT:\n dct = {}\n else:\n raise\n if field.array:\n dct[field.name] = list(field.value)\n else:\n dct[field.name] = field.value\n with open(fpath, 'w') as f:\n json.dump(dct, f, indent=2)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef find_data(self, subject_ids=None, visit_ids=None, **kwargs):\n all_filesets = []\n all_fields = []\n all_records = []\n for session_path, dirs, files in os.walk(self.root_dir):\n relpath = op.relpath(session_path, self.root_dir)\n path_parts = relpath.split(op.sep) if relpath != '.' else []\n ids = self._extract_ids_from_path(path_parts, dirs, files)\n if ids is None:\n continue\n subj_id, visit_id, from_study = ids\n # Check for summaries and filtered IDs\n if subj_id == self.SUMMARY_NAME:\n subj_id = None\n elif subject_ids is not None and subj_id not in subject_ids:\n continue\n if visit_id == self.SUMMARY_NAME:\n visit_id = None\n elif visit_ids is not None and visit_id not in visit_ids:\n continue\n # Map IDs into ID space of study\n subj_id = self.map_subject_id(subj_id)\n visit_id = self.map_visit_id(visit_id)\n # Determine frequency of session|summary\n if (subj_id, visit_id) == (None, None):\n frequency = 'per_study'\n elif subj_id is None:\n frequency = 'per_visit'\n elif visit_id is None:\n frequency = 'per_subject'\n else:\n frequency = 'per_session'\n filtered_files = self._filter_files(files, session_path)\n for fname in filtered_files:\n basename = split_extension(fname)[0]\n all_filesets.append(\n Fileset.from_path(\n op.join(session_path, fname),\n frequency=frequency,\n subject_id=subj_id, visit_id=visit_id,\n repository=self,\n from_study=from_study,\n potential_aux_files=[\n f for f in filtered_files\n if (split_extension(f)[0] == basename and\n f != fname)],\n **kwargs))\n for fname in self._filter_dirs(dirs, session_path):\n all_filesets.append(\n Fileset.from_path(\n op.join(session_path, fname),\n frequency=frequency,\n subject_id=subj_id, visit_id=visit_id,\n repository=self,\n from_study=from_study,\n **kwargs))\n if self.FIELDS_FNAME in files:\n with open(op.join(session_path,\n self.FIELDS_FNAME), 'r') as f:\n dct = json.load(f)\n all_fields.extend(\n Field(name=k, value=v, frequency=frequency,\n subject_id=subj_id, visit_id=visit_id,\n repository=self, from_study=from_study,\n **kwargs)\n for k, v in list(dct.items()))\n if self.PROV_DIR in dirs:\n if from_study is None:\n raise ArcanaRepositoryError(\n \"Found provenance directory in session directory (i.e.\"\n \" not in study-specific sub-directory)\")\n base_prov_dir = op.join(session_path, self.PROV_DIR)\n for fname in os.listdir(base_prov_dir):\n all_records.append(Record.load(\n split_extension(fname)[0],\n frequency, subj_id, visit_id, from_study,\n op.join(base_prov_dir, fname)))\n return all_filesets, all_fields, all_records", "response": "Find all data within a repository and return a list of all the filesets fields and provenance records."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntrying to guess the depth of a directory repository.", "response": "def guess_depth(self, root_dir):\n \"\"\"\n Try to guess the depth of a directory repository (i.e. whether it has\n sub-folders for multiple subjects or visits, depending on where files\n and/or derived label files are found in the hierarchy of\n sub-directories under the root dir.\n\n Parameters\n ----------\n root_dir : str\n Path to the root directory of the repository\n \"\"\"\n deepest = -1\n for path, dirs, files in os.walk(root_dir):\n depth = self.path_depth(path)\n filtered_files = self._filter_files(files, path)\n if filtered_files:\n logger.info(\"Guessing depth of directory repository at '{}' is\"\n \" {} due to unfiltered files ('{}') in '{}'\"\n .format(root_dir, depth,\n \"', '\".join(filtered_files), path))\n return depth\n if self.PROV_DIR in dirs:\n depth_to_return = max(depth - 1, 0)\n logger.info(\"Guessing depth of directory repository at '{}' is\"\n \"{} due to \\\"Derived label file\\\" in '{}'\"\n .format(root_dir, depth_to_return, path))\n return depth_to_return\n if depth >= self.MAX_DEPTH:\n logger.info(\"Guessing depth of directory repository at '{}' is\"\n \" {} as '{}' is already at maximum depth\"\n .format(root_dir, self.MAX_DEPTH, path))\n return self.MAX_DEPTH\n try:\n for fpath in chain(filtered_files,\n self._filter_dirs(dirs, path)):\n Fileset.from_path(fpath)\n except ArcanaError:\n pass\n else:\n if depth > deepest:\n deepest = depth\n if deepest == -1:\n raise ArcanaRepositoryError(\n \"Could not guess depth of '{}' repository as did not find \"\n \"a valid session directory within sub-directories.\"\n .format(root_dir))\n return deepest"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the data source from the database.", "response": "def get_data_source(query):\n \"\"\" Get data source connection metadata based on config. Prefer, in order:\n - Query file frontmatter\n - `db.yaml` file in query subdirectory\n - DATABASE_URL in environment\n\n To allow for possible non-SQL sources to be contributed, we only\n return a config value here, and leave connecting to concrete\n resultset module, as required values and connection method might change.\n (This is also why we return a dict, despite having a single value in the\n SQL case.)\n \"\"\"\n # *** GET CONNECTION PARAMS: ***\n # from frontmatter\n try:\n return query.metadata['data_source']\n except KeyError:\n try:\n return dict(url=query.metadata['database_url'])\n except KeyError:\n pass\n\n # from file in directory if present\n db_file = query.path.parent / 'db.yaml'\n if db_file.is_file():\n with open(db_file, 'r') as dbf:\n db_meta = yaml.safe_load(dbf.read())\n try:\n return db_meta['data_source']\n except KeyError:\n try:\n return dict(url=db_meta['database_url'])\n except KeyError:\n pass\n\n # Use env['DATABASE_URL']/sqlite if nothing else is configured\n return dict(\n url=os.getenv('DATABASE_URL', 'sqlite:///'))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a particular fileset|field in the collection corresponding to the given subject and visit ids.", "response": "def item(self, subject_id=None, visit_id=None):\n \"\"\"\n Returns a particular fileset|field in the collection corresponding to\n the given subject and visit_ids. subject_id and visit_id must be\n provided for relevant frequencies. Note that subject_id/visit_id can\n also be provided for non-relevant frequencies, they will just be\n ignored.\n\n Parameter\n ---------\n subject_id : str\n The subject id of the item to return\n visit_id : str\n The visit id of the item to return\n \"\"\"\n\n if self.frequency == 'per_session':\n if subject_id is None or visit_id is None:\n raise ArcanaError(\n \"The 'subject_id' ({}) and 'visit_id' ({}) must be \"\n \"provided to get an item from {}\".format(\n subject_id, visit_id, self))\n try:\n subj_dct = self._collection[subject_id]\n except KeyError:\n raise ArcanaIndexError(\n subject_id,\n \"{} not a subject ID in '{}' collection ({})\"\n .format(subject_id, self.name,\n ', '.join(self._collection.keys())))\n try:\n fileset = subj_dct[visit_id]\n except KeyError:\n raise ArcanaIndexError(\n visit_id,\n \"{} not a visit ID in subject {} of '{}' \"\n \"collection ({})\"\n .format(visit_id, subject_id, self.name,\n ', '.join(subj_dct.keys())))\n elif self.frequency == 'per_subject':\n if subject_id is None:\n raise ArcanaError(\n \"The 'subject_id' arg must be provided to get \"\n \"the match from {}\"\n .format(self))\n try:\n fileset = self._collection[subject_id]\n except KeyError:\n raise ArcanaIndexError(\n subject_id,\n \"{} not a subject ID in '{}' collection ({})\"\n .format(subject_id, self.name,\n ', '.join(self._collection.keys())))\n elif self.frequency == 'per_visit':\n if visit_id is None:\n raise ArcanaError(\n \"The 'visit_id' arg must be provided to get \"\n \"the match from {}\"\n .format(self))\n try:\n fileset = self._collection[visit_id]\n except KeyError:\n raise ArcanaIndexError(\n visit_id,\n \"{} not a visit ID in '{}' collection ({})\"\n .format(visit_id, self.name,\n ', '.join(self._collection.keys())))\n elif self.frequency == 'per_study':\n try:\n fileset = self._collection[0]\n except IndexError:\n raise ArcanaIndexError(\n \"'{}' Collection is empty so doesn't have a \"\n \"per_study node\".format(self.name))\n return fileset"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef bind(self, study, **kwargs): # @UnusedVariable\n if self.frequency == 'per_subject':\n tree_subject_ids = list(study.tree.subject_ids)\n subject_ids = list(self._collection.keys())\n if tree_subject_ids != subject_ids:\n raise ArcanaUsageError(\n \"Subject IDs in collection provided to '{}' ('{}') \"\n \"do not match Study tree ('{}')\".format(\n self.name, \"', '\".join(subject_ids),\n \"', '\".join(tree_subject_ids)))\n elif self.frequency == 'per_visit':\n tree_visit_ids = list(study.tree.visit_ids)\n visit_ids = list(self._collection.keys())\n if tree_visit_ids != visit_ids:\n raise ArcanaUsageError(\n \"Subject IDs in collection provided to '{}' ('{}') \"\n \"do not match Study tree ('{}')\".format(\n self.name, \"', '\".join(visit_ids),\n \"', '\".join(tree_visit_ids)))\n elif self.frequency == 'per_session':\n for subject in study.tree.subjects:\n if subject.id not in self._collection:\n raise ArcanaUsageError(\n \"Study subject ID '{}' was not found in colleciton \"\n \"provided to '{}' (found '{}')\".format(\n subject.id, self.name,\n \"', '\".join(self._collection.keys())))\n for session in subject.sessions:\n if session.visit_id not in self._collection[subject.id]:\n raise ArcanaUsageError(\n \"Study visit ID '{}' for subject '{}' was not \"\n \"found in colleciton provided to '{}' (found '{}')\"\n .format(subject.id, self.name,\n \"', '\".join(\n self._collection[subject.id].keys())))", "response": "Bind the study to the source and sink initiation."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef fileset(self, name, from_study=None, format=None): # @ReservedAssignment @IgnorePep8\n if isinstance(name, BaseFileset):\n if from_study is None and name.derived:\n from_study = name.study.name\n name = name.name\n try:\n format_dct = self._filesets[(name, from_study)]\n except KeyError:\n available = [\n ('{}(format={})'.format(f.name, f._resource_name)\n if f._resource_name is not None else f.name)\n for f in self.filesets if f.from_study == from_study]\n other_studies = [\n (f.from_study if f.from_study is not None else '')\n for f in self.filesets if f.name == name]\n if other_studies:\n msg = (\". NB: matching fileset(s) found for '{}' study(ies) \"\n \"('{}')\".format(name, \"', '\".join(other_studies)))\n else:\n msg = ''\n raise ArcanaNameError(\n name,\n (\"{} doesn't have a fileset named '{}'{} \"\n \"(available '{}'){}\"\n .format(self, name,\n (\" from study '{}'\".format(from_study)\n if from_study is not None else ''),\n \"', '\".join(available), msg)))\n else:\n if format is None:\n all_formats = list(format_dct.values())\n if len(all_formats) > 1:\n raise ArcanaNameError(\n \"Multiple filesets found for '{}'{} in {} with formats\"\n \" {}. Need to specify a format\"\n .format(name, (\"in '{}'\".format(from_study)\n if from_study is not None else ''),\n self, \"', '\".join(format_dct.keys())))\n fileset = all_formats[0]\n else:\n try:\n if isinstance(format, str):\n fileset = format_dct[format]\n else:\n try:\n fileset = format_dct[format.ext]\n except KeyError:\n fileset = None\n for rname, rfileset in format_dct.items():\n if rname in format.resource_names(\n self.tree.repository.type):\n fileset = rfileset\n break\n if fileset is None:\n raise\n except KeyError:\n raise ArcanaNameError(\n format,\n (\"{} doesn't have a fileset named '{}'{} with \"\n \"format '{}' (available '{}'){}\"\n .format(self, name,\n (\" from study '{}'\".format(from_study)\n if from_study is not None else ''),\n format,\n \"', '\".join(format_dct.keys()), msg)))\n\n return fileset", "response": "Returns the fileset with the given name produced by the given study or from the given spec."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets the field with the given name produced by the given study.", "response": "def field(self, name, from_study=None):\n \"\"\"\n Gets the field named 'name' produced by the Study named 'study' if\n provided. If a spec is passed instead of a str to the name argument,\n then the study will be set from the spec iff it is derived\n\n Parameters\n ----------\n name : str | BaseField\n The name of the field or a spec matching the given name\n study : str | None\n Name of the study that produced the field if derived. If None\n and a spec is passed instaed of string to the name argument then\n the study name will be taken from the spec instead.\n \"\"\"\n if isinstance(name, BaseField):\n if from_study is None and name.derived:\n from_study = name.study.name\n name = name.name\n try:\n return self._fields[(name, from_study)]\n except KeyError:\n available = [d.name for d in self.fields\n if d.from_study == from_study]\n other_studies = [(d.from_study if d.from_study is not None\n else '')\n for d in self.fields\n if d.name == name]\n if other_studies:\n msg = (\". NB: matching field(s) found for '{}' study(ies) \"\n \"('{}')\".format(name, \"', '\".join(other_studies)))\n else:\n msg = ''\n raise ArcanaNameError(\n name, (\"{} doesn't have a field named '{}'{} \"\n \"(available '{}')\"\n .format(\n self, name,\n (\" from study '{}'\".format(from_study)\n if from_study is not None else ''),\n \"', '\".join(available), msg)))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef record(self, pipeline_name, from_study):\n try:\n return self._records[(pipeline_name, from_study)]\n except KeyError:\n found = []\n for sname, pnames in groupby(sorted(self._records,\n key=itemgetter(1)),\n key=itemgetter(1)):\n found.append(\n \"'{}' for '{}'\".format(\"', '\".join(p for p, _ in pnames),\n sname))\n raise ArcanaNameError(\n (pipeline_name, from_study),\n (\"{} doesn't have a provenance record for pipeline '{}' \"\n \"for '{}' study (found {})\".format(\n self, pipeline_name, from_study,\n '; '.join(found))))", "response": "Returns the provenance record for a given pipeline and from_study."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef find_mismatch(self, other, indent=''):\n if self != other:\n mismatch = \"\\n{}{}\".format(indent, type(self).__name__)\n else:\n mismatch = ''\n sub_indent = indent + ' '\n if len(list(self.filesets)) != len(list(other.filesets)):\n mismatch += ('\\n{indent}mismatching summary fileset lengths '\n '(self={} vs other={}): '\n '\\n{indent} self={}\\n{indent} other={}'\n .format(len(list(self.filesets)),\n len(list(other.filesets)),\n list(self.filesets),\n list(other.filesets),\n indent=sub_indent))\n else:\n for s, o in zip(self.filesets, other.filesets):\n mismatch += s.find_mismatch(o, indent=sub_indent)\n if len(list(self.fields)) != len(list(other.fields)):\n mismatch += ('\\n{indent}mismatching summary field lengths '\n '(self={} vs other={}): '\n '\\n{indent} self={}\\n{indent} other={}'\n .format(len(list(self.fields)),\n len(list(other.fields)),\n list(self.fields),\n list(other.fields),\n indent=sub_indent))\n else:\n for s, o in zip(self.fields, other.fields):\n mismatch += s.find_mismatch(o, indent=sub_indent)\n return mismatch", "response": "Returns a string that contains the string that contains the mismatch of two TreeNode objects."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef nodes(self, frequency=None):\n if frequency is None:\n nodes = chain(*(self._nodes(f)\n for f in ('per_study', 'per_subject',\n 'per_visit', 'per_session')))\n else:\n nodes = self._nodes(frequency=frequency)\n return nodes", "response": "Returns an iterator over all nodes in the tree for the specified frequency."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef find_mismatch(self, other, indent=''):\n mismatch = super(Tree, self).find_mismatch(other, indent)\n sub_indent = indent + ' '\n if len(list(self.subjects)) != len(list(other.subjects)):\n mismatch += ('\\n{indent}mismatching subject lengths '\n '(self={} vs other={}): '\n '\\n{indent} self={}\\n{indent} other={}'\n .format(len(list(self.subjects)),\n len(list(other.subjects)),\n list(self.subjects),\n list(other.subjects),\n indent=sub_indent))\n else:\n for s, o in zip(self.subjects, other.subjects):\n mismatch += s.find_mismatch(o, indent=sub_indent)\n if len(list(self.visits)) != len(list(other.visits)):\n mismatch += ('\\n{indent}mismatching visit lengths '\n '(self={} vs other={}): '\n '\\n{indent} self={}\\n{indent} other={}'\n .format(len(list(self.visits)),\n len(list(other.visits)),\n list(self.visits),\n list(other.visits),\n indent=sub_indent))\n else:\n for s, o in zip(self.visits, other.visits):\n mismatch += s.find_mismatch(o, indent=sub_indent)\n return mismatch", "response": "Used in debugging unittests"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfill in tree with additional empty subjects and or visits to maintain order.", "response": "def _fill_empty_sessions(self, fill_subjects, fill_visits):\n \"\"\"\n Fill in tree with additional empty subjects and/or visits to\n allow the study to pull its inputs from external repositories\n \"\"\"\n if fill_subjects is None:\n fill_subjects = [s.id for s in self.subjects]\n if fill_visits is None:\n fill_visits = [v.id for v in self.complete_visits]\n for subject_id in fill_subjects:\n try:\n subject = self.subject(subject_id)\n except ArcanaNameError:\n subject = self._subjects[subject_id] = Subject(\n subject_id, [], [], [])\n for visit_id in fill_visits:\n try:\n subject.session(visit_id)\n except ArcanaNameError:\n session = Session(subject_id, visit_id, [], [])\n subject._sessions[visit_id] = session\n try:\n visit = self.visit(visit_id)\n except ArcanaNameError:\n visit = self._visits[visit_id] = Visit(\n visit_id, [], [], [])\n visit._sessions[subject_id] = session"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef construct(cls, repository, filesets=(), fields=(), records=(),\n file_formats=(), **kwargs):\n \"\"\"\n Return the hierarchical tree of the filesets and fields stored in a\n repository\n\n Parameters\n ----------\n respository : Repository\n The repository that the tree comes from\n filesets : list[Fileset]\n List of all filesets in the tree\n fields : list[Field]\n List of all fields in the tree\n records : list[Record]\n List of all records in the tree\n\n Returns\n -------\n tree : arcana.repository.Tree\n A hierarchical tree of subject, session and fileset\n information for the repository\n \"\"\"\n # Sort the data by subject and visit ID\n filesets_dict = defaultdict(list)\n for fset in filesets:\n if file_formats:\n fset.set_format(file_formats)\n filesets_dict[(fset.subject_id, fset.visit_id)].append(fset)\n fields_dict = defaultdict(list)\n for field in fields:\n fields_dict[(field.subject_id, field.visit_id)].append(field)\n records_dict = defaultdict(list)\n for record in records:\n records_dict[(record.subject_id, record.visit_id)].append(record)\n # Create all sessions\n subj_sessions = defaultdict(list)\n visit_sessions = defaultdict(list)\n for sess_id in set(chain(filesets_dict, fields_dict,\n records_dict)):\n if None in sess_id:\n continue # Save summaries for later\n subj_id, visit_id = sess_id\n session = Session(\n subject_id=subj_id, visit_id=visit_id,\n filesets=filesets_dict[sess_id],\n fields=fields_dict[sess_id],\n records=records_dict[sess_id])\n subj_sessions[subj_id].append(session)\n visit_sessions[visit_id].append(session)\n subjects = []\n for subj_id in subj_sessions:\n subjects.append(Subject(\n subj_id,\n sorted(subj_sessions[subj_id]),\n filesets_dict[(subj_id, None)],\n fields_dict[(subj_id, None)],\n records_dict[(subj_id, None)]))\n visits = []\n for visit_id in visit_sessions:\n visits.append(Visit(\n visit_id,\n sorted(visit_sessions[visit_id]),\n filesets_dict[(None, visit_id)],\n fields_dict[(None, visit_id)],\n records_dict[(None, visit_id)]))\n return Tree(sorted(subjects),\n sorted(visits),\n repository,\n filesets_dict[(None, None)],\n fields_dict[(None, None)],\n records_dict[(None, None)],\n **kwargs)", "response": "Constructs a new tree of all filesets fields and records and stores the summary for later tree of filesets and fields."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning all nodes related to the Session for the specified frequency", "response": "def nodes(self, frequency=None):\n \"\"\"\n Returns all nodes of the specified frequency that are related to\n the given Session\n\n Parameters\n ----------\n frequency : str | None\n The frequency of the nodes to return\n\n Returns\n -------\n nodes : iterable[TreeNode]\n All nodes related to the Session for the specified frequency\n \"\"\"\n if frequency is None:\n []\n elif frequency == 'per_session':\n return [self]\n elif frequency in ('per_visit', 'per_subject'):\n return [self.parent]\n elif frequency == 'per_study':\n return [self.parent.parent]"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nwriting data on the asyncio Protocol", "response": "def write_data(self, data, response_required=None, timeout=5.0, raw=False):\n \"\"\"Write data on the asyncio Protocol\"\"\"\n if self._transport is None:\n return\n\n if self._paused:\n return\n\n if self._waiting_for_response:\n LOG.debug(\"queueing write %s\", data)\n self._queued_writes.append((data, response_required, timeout))\n return\n\n if response_required:\n self._waiting_for_response = response_required\n if timeout > 0:\n self._timeout_task = self.loop.call_later(\n timeout, self._response_required_timeout)\n\n if not raw:\n cksum = 256 - reduce(lambda x, y: x+y, map(ord, data)) % 256\n data = data + '{:02X}'.format(cksum)\n if int(data[0:2], 16) != len(data)-2:\n LOG.debug(\"message length wrong: %s\", data)\n\n LOG.debug(\"write_data '%s'\", data)\n self._transport.write((data + '\\r\\n').encode())"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef unwrap_querystring_lists(obj):\n new_dict = {\n key: (obj[key][0]\n if len(obj[key]) == 1 else obj[key])\n for key in obj.keys()\n }\n return new_dict", "response": "Convert responder querystring params pulling values out of list\n "} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pipeline_getter(self):\n \"For duck-typing with *Spec types\"\n if not self.derivable:\n raise ArcanaUsageError(\n \"There is no pipeline getter for {} because it doesn't \"\n \"fallback to a derived spec\".format(self))\n return self._fallback.pipeline_getter", "response": "For duck - typing with *Spec types"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef prerequisites(self):\n # Loop through the inputs to the pipeline and add the instancemethods\n # for the pipelines to generate each of the processed inputs\n prereqs = defaultdict(set)\n for input in self.inputs: # @ReservedAssignment\n spec = self._study.spec(input)\n # Could be an input to the study or optional acquired spec\n if spec.is_spec and spec.derived:\n prereqs[spec.pipeline_getter].add(input.name)\n return prereqs", "response": "Returns a dictionary of all prerequisite names that are used in the pipeline."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef study_inputs(self):\n return set(chain(\n (i for i in self.inputs if not i.derived),\n *(self.study.pipeline(p, required_outputs=r).study_inputs\n for p, r in self.prerequisites.items())))", "response": "Returns all inputs of the study used by the pipeline including inputs\n of prerequisites and inputs\n of prerequisites recursively."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd a processing Node to the pipeline.", "response": "def add(self, name, interface, inputs=None, outputs=None,\n requirements=None, wall_time=None, annotations=None, **kwargs):\n \"\"\"\n Adds a processing Node to the pipeline\n\n Parameters\n ----------\n name : str\n Name for the node\n interface : nipype.Interface\n The interface to use for the node\n inputs : dict[str, (str, FileFormat) | (Node, str)]\n Connections from inputs of the pipeline and outputs of other nodes\n to inputs of node. The keys of the dictionary are the field names\n and the values are 2-tuple containing either the name of the data\n spec and the data format it is expected in for pipeline inputs or\n the sending Node and the the name of an output of the sending Node.\n Note that pipeline inputs can be specified outside this method\n using the 'connect_input' method and connections between nodes with\n the the 'connect' method.\n outputs : dict[str, (str, FileFormat)]\n Connections to outputs of the pipeline from fields of the\n interface. The keys of the dictionary are the names of the data\n specs that will be written to and the values are the interface\n field name and the data format it is produced in. Note that output\n connections can also be specified using the 'connect_output'\n method.\n requirements : list(Requirement)\n List of required packages need for the node to run (default: [])\n wall_time : float\n Time required to execute the node in minutes (default: 1)\n mem_gb : int\n Required memory for the node in GB\n n_procs : int\n Preferred number of threads to run the node on (default: 1)\n annotations : dict[str, *]\n Additional annotations to add to the node, which may be used by\n the Processor node to optimise execution (e.g. 'gpu': True)\n iterfield : str\n Name of field to be passed an iterable to iterator over.\n If present, a MapNode will be created instead of a regular node\n joinsource : str\n Name of iterator field to join. Typically one of the implicit\n iterators (i.e. Study.SUBJECT_ID or Study.VISIT_ID)\n to join over the subjects and/or visits\n joinfield : str\n Name of field to pass the joined list when creating a JoinNode\n\n Returns\n -------\n node : Node\n The Node object that has been added to the pipeline\n \"\"\"\n if annotations is None:\n annotations = {}\n if requirements is None:\n requirements = []\n if wall_time is None:\n wall_time = self.study.processor.default_wall_time\n if 'mem_gb' not in kwargs or kwargs['mem_gb'] is None:\n kwargs['mem_gb'] = self.study.processor.default_mem_gb\n if 'iterfield' in kwargs:\n if 'joinfield' in kwargs or 'joinsource' in kwargs:\n raise ArcanaDesignError(\n \"Cannot provide both joinsource and iterfield to when \"\n \"attempting to add '{}' node to {}\"\n .foramt(name, self._error_msg_loc))\n node_cls = self.study.environment.node_types['map']\n elif 'joinsource' in kwargs or 'joinfield' in kwargs:\n if not ('joinfield' in kwargs and 'joinsource' in kwargs):\n raise ArcanaDesignError(\n \"Both joinsource and joinfield kwargs are required to \"\n \"create a JoinNode (see {})\".format(name,\n self._error_msg_loc))\n joinsource = kwargs['joinsource']\n if joinsource in self.study.ITERFIELDS:\n self._iterator_joins.add(joinsource)\n node_cls = self.study.environment.node_types['join']\n # Prepend name of pipeline of joinsource to match name of nodes\n kwargs['joinsource'] = '{}_{}'.format(self.name, joinsource)\n else:\n node_cls = self.study.environment.node_types['base']\n # Create node\n node = node_cls(self.study.environment,\n interface,\n name=\"{}_{}\".format(self._name, name),\n requirements=requirements,\n wall_time=wall_time,\n annotations=annotations,\n **kwargs)\n # Ensure node is added to workflow\n self._workflow.add_nodes([node])\n # Connect inputs, outputs and internal connections\n if inputs is not None:\n assert isinstance(inputs, dict)\n for node_input, connect_from in inputs.items():\n if isinstance(connect_from[0], basestring):\n input_spec, input_format = connect_from\n self.connect_input(input_spec, node,\n node_input, input_format)\n else:\n conn_node, conn_field = connect_from\n self.connect(conn_node, conn_field, node, node_input)\n if outputs is not None:\n assert isinstance(outputs, dict)\n for output_spec, (node_output, output_format) in outputs.items():\n self.connect_output(output_spec, node, node_output,\n output_format)\n return node"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconnect a study fileset_spec as an input to the provided node.", "response": "def connect_input(self, spec_name, node, node_input, format=None, **kwargs): # @ReservedAssignment @IgnorePep8\n \"\"\"\n Connects a study fileset_spec as an input to the provided node\n\n Parameters\n ----------\n spec_name : str\n Name of the study data spec (or one of the IDs from the iterator\n nodes, 'subject_id' or 'visit_id') to connect to the node\n node : arcana.Node\n The node to connect the input to\n node_input : str\n Name of the input on the node to connect the fileset spec to\n format : FileFormat | None\n The file format the input is expected in. If it differs\n from the format in data spec or of study input then an implicit\n conversion is performed. If None the file format in the data spec\n is assumed\n \"\"\"\n if spec_name in self.study.ITERFIELDS:\n self._iterator_conns[spec_name].append((node, node_input, format))\n else:\n name = self._map_name(spec_name, self._input_map)\n if name not in self.study.data_spec_names():\n raise ArcanaDesignError(\n \"Proposed input '{}' to {} is not a valid spec name ('{}')\"\n .format(name, self._error_msg_loc,\n \"', '\".join(self.study.data_spec_names())))\n self._input_conns[name].append((node, node_input, format, kwargs))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef connect_output(self, spec_name, node, node_output, format=None, # @ReservedAssignment @IgnorePep8\n **kwargs):\n \"\"\"\n Connects an output to a study fileset spec\n\n Parameters\n ----------\n spec_name : str\n Name of the study fileset spec to connect to\n node : arcana.Node\n The node to connect the output from\n node_output : str\n Name of the output on the node to connect to the fileset\n format : FileFormat | None\n The file format the output is returned in. If it differs\n from the format in data spec then an implicit conversion is\n performed. If None the it is assumed to be returned in the file\n format of the entry the data spec\n \"\"\"\n name = self._map_name(spec_name, self._output_map)\n if name not in self.study.data_spec_names():\n raise ArcanaDesignError(\n \"Proposed output '{}' to {} is not a valid spec name ('{}')\"\n .format(name, self._error_msg_loc,\n \"', '\".join(self.study.data_spec_names())))\n if name in self._output_conns:\n prev_node, prev_node_output, _, _ = self._output_conns[name]\n logger.info(\n \"Reassigning '{}' output from {}:{} to {}:{} in {}\"\n .format(name, prev_node.name, prev_node_output,\n node.name, node_output, self._error_msg_loc))\n self._output_conns[name] = (node, node_output, format, kwargs)", "response": "Connects an output from a study fileset to a specific node."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _map_name(self, name, mapper):\n if mapper is not None:\n if isinstance(mapper, basestring):\n name = mapper + name\n try:\n name = mapper[name]\n except KeyError:\n pass\n return name", "response": "Maps a spec name to a new value based on the provided mapper"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck whether the fileset matches the requested file format", "response": "def requires_conversion(cls, fileset, file_format):\n \"\"\"Checks whether the fileset matches the requested file format\"\"\"\n if file_format is None:\n return False\n try:\n filset_format = fileset.format\n except AttributeError:\n return False # Field input\n else:\n return (file_format != filset_format)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef save_graph(self, fname, style='flat', format='png', **kwargs): # @ReservedAssignment @IgnorePep8\n fname = os.path.expanduser(fname)\n if not fname.endswith('.png'):\n fname += '.png'\n orig_dir = os.getcwd()\n tmpdir = tempfile.mkdtemp()\n os.chdir(tmpdir)\n workflow = self._workflow\n workflow.write_graph(graph2use=style, format=format, **kwargs)\n os.chdir(orig_dir)\n try:\n shutil.move(os.path.join(tmpdir, 'graph_detailed.{}'\n .format(format)), fname)\n except IOError as e:\n if e.errno == errno.ENOENT:\n shutil.move(os.path.join(tmpdir, 'graph.{}'.format(format)),\n fname)\n else:\n raise\n shutil.rmtree(tmpdir)", "response": "Saves a graph of the pipeline to file fname."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the iterators that the pipeline contains for each input frequency.", "response": "def iterators(self, frequency=None):\n \"\"\"\n Returns the iterators (i.e. subject_id, visit_id) that the pipeline\n iterates over\n\n Parameters\n ----------\n frequency : str | None\n A selected data frequency to use to determine which iterators are\n required. If None, all input frequencies of the pipeline are\n assumed\n \"\"\"\n iterators = set()\n if frequency is None:\n input_freqs = list(self.input_frequencies)\n else:\n input_freqs = [frequency]\n for freq in input_freqs:\n iterators.update(self.study.FREQUENCIES[freq])\n return iterators"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _unwrap_maps(self, name_maps, name, study=None, **inner_maps):\n # Set values of name and study\n name = name_maps.get('name', name)\n name = name_maps.get('prefix', '') + name\n study = name_maps.get('study', study)\n # Flatten input and output maps, combining maps from inner nests with\n # those in the \"mods\" dictionary\n maps = {}\n for mtype in ('input_map', 'output_map'):\n try:\n inner_map = inner_maps[mtype]\n except KeyError:\n try:\n maps[mtype] = name_maps[mtype] # Only outer map\n except KeyError:\n pass # No maps\n else:\n try:\n outer_map = name_maps[mtype]\n except KeyError:\n maps[mtype] = inner_map # Only inner map\n else:\n # Work through different combinations of inner and outer\n # map types (i.e. str & str, str & dict, dict & str, and\n # dict & dict) and combine into a single map\n if isinstance(outer_map, basestring):\n if isinstance(inner_map, basestring):\n # Concatenate prefixes\n maps[mtype] = outer_map + inner_map\n elif isinstance(inner_map, dict):\n # Add outer_map prefix to all values in inner map\n # dictionary\n maps[mtype] = {k: outer_map + v\n for k, v in inner_map.items()}\n else:\n raise ArcanaDesignError(\n \"Unrecognised type for name map in '{}' \"\n \"pipeline can be str or dict[str,str]: {}\"\n .format(name, inner_map))\n elif isinstance(outer_map, dict):\n if isinstance(inner_map, basestring):\n # Strip inner map prefix from outer dictionary\n # (which should have prefix included). This should\n # be an unlikely case I imagine\n maps[mtype] = {k[len(inner_map):]: v\n for k, v in outer_map.items()}\n elif isinstance(inner_map, dict):\n # Chain outer_map dictionary to inner map\n # dictionary\n maps[mtype] = deepcopy(outer_map)\n maps[mtype].update(\n {k: outer_map.get(v, v)\n for k, v in inner_map.items()})\n else:\n raise ArcanaDesignError(\n \"Unrecognised type for name map in '{}' \"\n \"pipeline can be str or dict[str,str]: {}\"\n .format(name, inner_map))\n else:\n raise ArcanaDesignError(\n \"Unrecognised type for name map in '{}' \"\n \"pipeline can be str or dict[str,str]: {}\"\n .format(name, outer_map))\n try:\n outer_maps = name_maps['name_maps']\n except KeyError:\n pass\n else:\n name, study, maps = self._unwrap_maps(\n outer_maps, name=name, study=study, **maps)\n return name, study, maps", "response": "Unwraps potentially nested name - mapping dictionaries to get values\n for name input_map output_map and study."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncaps the construction of the pipeline.", "response": "def cap(self):\n \"\"\"\n \"Caps\" the construction of the pipeline, signifying that no more inputs\n and outputs are expected to be added and therefore the input and output\n nodes can be created along with the provenance.\n \"\"\"\n to_cap = (self._inputnodes, self._outputnodes, self._prov)\n if to_cap == (None, None, None):\n self._inputnodes = {\n f: self._make_inputnode(f) for f in self.input_frequencies}\n self._outputnodes = {\n f: self._make_outputnode(f) for f in self.output_frequencies}\n self._prov = self._gen_prov()\n elif None in to_cap:\n raise ArcanaError(\n \"If one of _inputnodes, _outputnodes or _prov is not None then\"\n \" they all should be in {}\".format(self))"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates an input node for the given frequency and adds it to the pipeline.", "response": "def _make_inputnode(self, frequency):\n \"\"\"\n Generates an input node for the given frequency. It also adds implicit\n file format conversion nodes to the pipeline.\n\n Parameters\n ----------\n frequency : str\n The frequency (i.e. 'per_session', 'per_visit', 'per_subject' or\n 'per_study') of the input node to retrieve\n \"\"\"\n # Check to see whether there are any outputs for the given frequency\n inputs = list(self.frequency_inputs(frequency))\n # Get list of input names for the requested frequency, addding fields\n # to hold iterator IDs\n input_names = [i.name for i in inputs]\n input_names.extend(self.study.FREQUENCIES[frequency])\n if not input_names:\n raise ArcanaError(\n \"No inputs to '{}' pipeline for requested freqency '{}'\"\n .format(self.name, frequency))\n # Generate input node and connect it to appropriate nodes\n inputnode = self.add('{}_inputnode'.format(frequency),\n IdentityInterface(fields=input_names))\n # Loop through list of nodes connected to study data specs and\n # connect them to the newly created input node\n for input in inputs: # @ReservedAssignment\n # Keep track of previous conversion nodes to avoid replicating the\n # conversion for inputs that are used in multiple places\n prev_conv_nodes = {}\n for (node, node_in, format, # @ReservedAssignment @IgnorePep8\n conv_kwargs) in self._input_conns[input.name]:\n # If fileset formats differ between study and pipeline\n # inputs create converter node (if one hasn't been already)\n # and connect input to that before connecting to inputnode\n if self.requires_conversion(input, format):\n try:\n conv = format.converter_from(input.format,\n **conv_kwargs)\n except ArcanaNoConverterError as e:\n e.msg += (\n \"which is required to convert '{}' from {} to {} \"\n \"for '{}' input of '{}' node\".format(\n input.name, input.format, format, node_in,\n node.name))\n raise e\n try:\n in_node = prev_conv_nodes[format.name]\n except KeyError:\n in_node = prev_conv_nodes[format.name] = self.add(\n 'conv_{}_to_{}_format'.format(input.name,\n format.name),\n conv.interface,\n inputs={conv.input: (inputnode, input.name)},\n requirements=conv.requirements,\n mem_gb=conv.mem_gb,\n wall_time=conv.wall_time)\n in_node_out = conv.output\n else:\n in_node = inputnode\n in_node_out = input.name\n self.connect(in_node, in_node_out, node, node_in)\n # Connect iterator inputs\n for iterator, conns in self._iterator_conns.items():\n # Check to see if this is the right frequency for the iterator\n # input, i.e. if it is the only iterator for this frequency\n if self.study.FREQUENCIES[frequency] == (iterator,):\n for (node, node_in, format) in conns: # @ReservedAssignment\n self.connect(inputnode, iterator, node, node_in)\n return inputnode"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _make_outputnode(self, frequency):\n # Check to see whether there are any outputs for the given frequency\n outputs = list(self.frequency_outputs(frequency))\n if not outputs:\n raise ArcanaError(\n \"No outputs to '{}' pipeline for requested freqency '{}'\"\n .format(self.name, frequency))\n # Get list of output names for the requested frequency, addding fields\n # to hold iterator IDs\n output_names = [o.name for o in outputs]\n # Generate output node and connect it to appropriate nodes\n outputnode = self.add('{}_outputnode'.format(frequency),\n IdentityInterface(fields=output_names))\n # Loop through list of nodes connected to study data specs and\n # connect them to the newly created output node\n for output in outputs: # @ReservedAssignment\n (node, node_out, format, # @ReservedAssignment @IgnorePep8\n conv_kwargs) = self._output_conns[output.name]\n # If fileset formats differ between study and pipeline\n # outputs create converter node (if one hasn't been already)\n # and connect output to that before connecting to outputnode\n if self.requires_conversion(output, format):\n conv = output.format.converter_from(format, **conv_kwargs)\n node = self.add(\n 'conv_{}_from_{}_format'.format(output.name, format.name),\n conv.interface,\n inputs={conv.input: (node, node_out)},\n requirements=conv.requirements,\n mem_gb=conv.mem_gb,\n wall_time=conv.wall_time)\n node_out = conv.output\n self.connect(node, node_out, outputnode, output.name)\n return outputnode", "response": "Creates an output node for the given frequency."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ngenerate provenance information for the pipeline and returns a dictionary containing the provenance information for the pipeline and the pipeline itself.", "response": "def _gen_prov(self):\n \"\"\"\n Extracts provenance information from the pipeline into a PipelineProv\n object\n\n Returns\n -------\n prov : dict[str, *]\n A dictionary containing the provenance information to record\n for the pipeline\n \"\"\"\n # Export worfklow graph to node-link data format\n wf_dict = nx_json.node_link_data(self.workflow._graph)\n # Replace references to Node objects with the node's provenance\n # information and convert to a dict organised by node name to allow it\n # to be compared more easily. Also change link node-references from\n # node index to node ID so it is not dependent on the order the nodes\n # are written to the dictionary (which for Python < 3.7 is guaranteed\n # to be the same between identical runs)\n for link in wf_dict['links']:\n if int(networkx_version.split('.')[0]) < 2: # @UndefinedVariable\n link['source'] = wf_dict['nodes'][link['source']]['id'].name\n link['target'] = wf_dict['nodes'][link['target']]['id'].name\n else:\n link['source'] = link['source'].name\n link['target'] = link['target'].name\n wf_dict['nodes'] = {n['id'].name: n['id'].prov\n for n in wf_dict['nodes']}\n # Roundtrip to JSON to convert any tuples into lists so dictionaries\n # can be compared directly\n wf_dict = json.loads(json.dumps(wf_dict))\n dependency_versions = {d: extract_package_version(d)\n for d in ARCANA_DEPENDENCIES}\n pkg_versions = {'arcana': __version__}\n pkg_versions.update((k, v) for k, v in dependency_versions.items()\n if v is not None)\n prov = {\n '__prov_version__': PROVENANCE_VERSION,\n 'name': self.name,\n 'workflow': wf_dict,\n 'study': self.study.prov,\n 'pkg_versions': pkg_versions,\n 'python_version': sys.version,\n 'joined_ids': self._joined_ids()}\n return prov"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconstruct the provenance record that would be saved in the given node.", "response": "def expected_record(self, node):\n \"\"\"\n Constructs the provenance record that would be saved in the given node\n if the pipeline was run on the current state of the repository\n\n Parameters\n ----------\n node : arcana.repository.tree.TreeNode\n A node of the Tree representation of the study data stored in the\n repository (i.e. a Session, Visit, Subject or Tree node)\n\n Returns\n -------\n expected_record : arcana.provenance.Record\n The record that would be produced if the pipeline is run over the\n study tree.\n \"\"\"\n exp_inputs = {}\n # Get checksums/values of all inputs that would have been used in\n # previous runs of an equivalent pipeline to compare with that saved\n # in provenance to see if any have been updated.\n for inpt in self.inputs: # @ReservedAssignment\n # Get iterators present in the input that aren't in this node\n # and need to be joined\n iterators_to_join = (self.iterators(inpt.frequency) -\n self.iterators(node.frequency))\n if not iterators_to_join:\n # No iterators to join so we can just extract the checksums\n # of the corresponding input\n exp_inputs[inpt.name] = inpt.collection.item(\n node.subject_id, node.visit_id).checksums\n elif len(iterators_to_join) == 1:\n # Get list of checksums dicts for each node of the input\n # frequency that relates to the current node\n exp_inputs[inpt.name] = [\n inpt.collection.item(n.subject_id, n.visit_id).checksums\n for n in node.nodes(inpt.frequency)]\n else:\n # In the case where the node is the whole treee and the input\n # is per_seession, we need to create a list of lists to match\n # how the checksums are joined in the processor\n exp_inputs[inpt.name] = []\n for subj in node.subjects:\n exp_inputs[inpt.name].append([\n inpt.collection.item(s.subject_id,\n s.visit_id).checksums\n for s in subj.sessions])\n # Get checksums/value for all outputs of the pipeline. We are assuming\n # that they exist here (otherwise they will be None)\n exp_outputs = {\n o.name: o.collection.item(node.subject_id, node.visit_id).checksums\n for o in self.outputs}\n exp_prov = copy(self.prov)\n if PY2:\n # Need to convert to unicode strings for Python 2\n exp_inputs = json.loads(json.dumps(exp_inputs))\n exp_outputs = json.loads(json.dumps(exp_outputs))\n exp_prov['inputs'] = exp_inputs\n exp_prov['outputs'] = exp_outputs\n exp_prov['joined_ids'] = self._joined_ids()\n return Record(\n self.name, node.frequency, node.subject_id, node.visit_id,\n self.study.name, exp_prov)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd the subject and visit ids used to generate the derivatives iff there are any joins over them in the pipeline", "response": "def _joined_ids(self):\n \"\"\"\n Adds the subjects/visits used to generate the derivatives iff there are\n any joins over them in the pipeline\n \"\"\"\n joined_prov = {}\n if self.joins_subjects:\n joined_prov['subject_ids'] = list(self.study.subject_ids)\n if self.joins_visits:\n joined_prov['visit_ids'] = list(self.study.visit_ids)\n return joined_prov"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef tree(self, subject_ids=None, visit_ids=None, **kwargs):\n # Find all data present in the repository (filtered by the passed IDs)\n return Tree.construct(\n self, *self.find_data(subject_ids=subject_ids,\n visit_ids=visit_ids), **kwargs)", "response": "Return the hierarchical tree of subject and session and fileset related to the given subject and visit IDs."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the tree for the given subject and visit ID filters and the IDs that were actually returned by the tree.", "response": "def cached_tree(self, subject_ids=None, visit_ids=None, fill=False):\n \"\"\"\n Access the repository tree and caches it for subsequent\n accesses\n\n Parameters\n ----------\n subject_ids : list(str)\n List of subject IDs with which to filter the tree with. If\n None all are returned\n visit_ids : list(str)\n List of visit IDs with which to filter the tree with. If\n None all are returned\n fill : bool\n Create empty sessions for any that are missing in the\n subject_id x visit_id block. Typically only used if all\n the inputs to the study are coming from different repositories\n to the one that the derived products are stored in\n\n Returns\n -------\n tree : arcana.repository.Tree\n A hierarchical tree of subject, vist, session information and that\n of the filesets and fields they contain\n \"\"\"\n if subject_ids is not None:\n subject_ids = frozenset(subject_ids)\n if visit_ids is not None:\n visit_ids = frozenset(visit_ids)\n try:\n tree = self._cache[subject_ids][visit_ids]\n except KeyError:\n if fill:\n fill_subjects = subject_ids\n fill_visits = visit_ids\n else:\n fill_subjects = fill_visits = None\n tree = self.tree(\n subject_ids=subject_ids, visit_ids=visit_ids,\n fill_visits=fill_visits, fill_subjects=fill_subjects)\n # Save the tree within the cache under the given subject/\n # visit ID filters and the IDs that were actually returned\n self._cache[subject_ids][visit_ids] = self._cache[\n frozenset(tree.subject_ids)][frozenset(tree.visit_ids)] = tree\n return tree"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef resource_names(self, repo_type):\n try:\n names = self._resource_names[repo_type]\n except KeyError:\n names = [self.name, self.name.upper()]\n return names", "response": "Returns a list of resource names for this format on a given repository type."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef default_aux_file_paths(self, primary_path):\n return dict((n, primary_path[:-len(self.ext)] + ext)\n for n, ext in self.aux_files.items())", "response": "Get the default paths for auxiliary files relative to the path of the primary file"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef assort_files(self, candidates):\n by_ext = defaultdict(list)\n for path in candidates:\n by_ext[split_extension(path)[1].lower()].append(path)\n try:\n primary_file = by_ext[self.ext]\n except KeyError:\n raise ArcanaFileFormatError(\n \"No files match primary file extension of {} out of \"\n \"potential candidates of {}\"\n .format(self, \"', '\".join(candidates)))\n if not primary_file:\n raise ArcanaFileFormatError(\n \"No potential files for primary file of {}\".format(self))\n elif len(primary_file) > 1:\n raise ArcanaFileFormatError(\n \"Multiple potential files for '{}' primary file of {}\"\n .format(\"', '\".join(primary_file), self))\n else:\n primary_file = primary_file[0]\n aux_files = {}\n for aux_name, aux_ext in self.aux_files.items():\n try:\n aux = by_ext[aux_ext]\n except KeyError:\n raise ArcanaFileFormatError(\n \"No files match auxiliary file extension '{}' of {} out of\"\n \" potential candidates of {}\"\n .format(aux_ext, self, \"', '\".join(candidates)))\n if len(aux) > 1:\n raise ArcanaFileFormatError(\n \"Multiple potential files for '{}' auxiliary file ext. \"\n \"({}) of {}\".format(\"', '\".join(aux),\n self))\n aux_files[aux_name] = aux[0]\n return primary_file, aux_files", "response": "Assorts candidate files into primary and auxiliary files."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef matches(self, fileset):\n if fileset._resource_name is not None:\n return (fileset._resource_name in self.resource_names(\n fileset.repository.type))\n elif self.directory:\n if op.isdir(fileset.path):\n if self.within_dir_exts is None:\n return True\n else:\n # Get set of all extensions in the directory\n return self.within_dir_exts == frozenset(\n split_extension(f)[1] for f in os.listdir(fileset.path)\n if not f.startswith('.'))\n else:\n return False\n else:\n if op.isfile(fileset.path):\n all_paths = [fileset.path] + fileset._potential_aux_files\n try:\n primary_path = self.assort_files(all_paths)[0]\n except ArcanaFileFormatError:\n return False\n else:\n return primary_path == fileset.path\n else:\n return False", "response": "Checks to see if the format matches the given fileset."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_converter(self, file_format, converter):\n self._converters[file_format.name] = (file_format, converter)", "response": "Register a Converter and the FileFormat that it can convert from from\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse_url(url):\n scheme, dest = url.split('://')\n host = None\n ssl_context = None\n if scheme == 'elk':\n host, port = dest.split(':') if ':' in dest else (dest, 2101)\n elif scheme == 'elks':\n host, port = dest.split(':') if ':' in dest else (dest, 2601)\n ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)\n ssl_context.verify_mode = ssl.CERT_NONE\n elif scheme == 'serial':\n host, port = dest.split(':') if ':' in dest else (dest, 115200)\n else:\n raise ValueError(\"Invalid scheme '%s'\" % scheme)\n return (scheme, host, int(port), ssl_context)", "response": "Parse an Elk connection string into a tuple."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef pretty_const(value):\n words = value.split('_')\n pretty = words[0].capitalize()\n for word in words[1:]:\n pretty += ' ' + word.lower()\n return pretty", "response": "Make a constant pretty for printing in GUI"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn name of user.", "response": "def username(elk, user_number):\n \"\"\"Return name of user.\"\"\"\n if user_number >= 0 and user_number < elk.users.max_elements:\n return elk.users[user_number].name\n if user_number == 201:\n return \"*Program*\"\n if user_number == 202:\n return \"*Elk RP*\"\n if user_number == 203:\n return \"*Quick arm*\"\n return \"\""} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconnect to the ElkM1.", "response": "async def _connect(self, connection_lost_callbk=None):\n \"\"\"Asyncio connection to Elk.\"\"\"\n self.connection_lost_callbk = connection_lost_callbk\n url = self._config['url']\n LOG.info(\"Connecting to ElkM1 at %s\", url)\n scheme, dest, param, ssl_context = parse_url(url)\n conn = partial(Connection, self.loop, self._connected,\n self._disconnected, self._got_data, self._timeout)\n try:\n if scheme == 'serial':\n await serial_asyncio.create_serial_connection(\n self.loop, conn, dest, baudrate=param)\n else:\n await asyncio.wait_for(self.loop.create_connection(\n conn, host=dest, port=param, ssl=ssl_context), timeout=30)\n except (ValueError, OSError, asyncio.TimeoutError) as err:\n LOG.warning(\"Could not connect to ElkM1 (%s). Retrying in %d seconds\",\n err, self._connection_retry_timer)\n self.loop.call_later(self._connection_retry_timer, self.connect)\n self._connection_retry_timer = 2 * self._connection_retry_timer \\\n if self._connection_retry_timer < 32 else 60"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _connected(self, transport, conn):\n LOG.info(\"Connected to ElkM1\")\n self._conn = conn\n self._transport = transport\n self._connection_retry_timer = 1\n if url_scheme_is_secure(self._config['url']):\n self._conn.write_data(self._config['userid'], raw=True)\n self._conn.write_data(self._config['password'], raw=True)\n self.call_sync_handlers()\n if not self._config['url'].startswith('serial://'):\n self._heartbeat = self.loop.call_later(120, self._reset_connection)", "response": "Login and sync the ElkM1 panel to memory."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsend a message to the Elk panel.", "response": "def send(self, msg):\n \"\"\"Send a message to Elk panel.\"\"\"\n if self._conn:\n self._conn.write_data(msg.message, msg.response_command)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef sync(self):\n self.elk.send(az_encode())\n self.elk.send(zd_encode())\n self.elk.send(zp_encode())\n self.elk.send(zs_encode())\n self.get_descriptions(TextDescriptions.ZONE.value)", "response": "Retrieve zones from ElkM1"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nraising an exception if the model uses a history manager and historical model history_id is not a UUIDField.", "response": "def has_offline_historical_manager_or_raise(self):\n \"\"\"Raises an exception if model uses a history manager and\n historical model history_id is not a UUIDField.\n\n Note: expected to use edc_model.HistoricalRecords instead of\n simple_history.HistoricalRecords.\n \"\"\"\n try:\n model = self.instance.__class__.history.model\n except AttributeError:\n model = self.instance.__class__\n field = [field for field in model._meta.fields if field.name == \"history_id\"]\n if field and not isinstance(field[0], UUIDField):\n raise OfflineHistoricalManagerError(\n f\"Field 'history_id' of historical model \"\n f\"'{model._meta.app_label}.{model._meta.model_name}' \"\n \"must be an UUIDfield. \"\n \"For history = HistoricalRecords() use edc_model.HistoricalRecords instead of \"\n \"simple_history.HistoricalRecords(). \"\n f\"See '{self.instance._meta.app_label}.{self.instance._meta.model_name}'.\"\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef primary_key_field(self):\n return [field for field in self.instance._meta.fields if field.primary_key][0]", "response": "Return the primary key field."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nserialize the object to an OutgoingTransaction object and saves it to the OutgoingTransaction model.", "response": "def to_outgoing_transaction(self, using, created=None, deleted=None):\n \"\"\" Serialize the model instance to an AES encrypted json object\n and saves the json object to the OutgoingTransaction model.\n \"\"\"\n OutgoingTransaction = django_apps.get_model(\n \"django_collect_offline\", \"OutgoingTransaction\"\n )\n created = True if created is None else created\n action = INSERT if created else UPDATE\n timestamp_datetime = (\n self.instance.created if created else self.instance.modified\n )\n if not timestamp_datetime:\n timestamp_datetime = get_utcnow()\n if deleted:\n timestamp_datetime = get_utcnow()\n action = DELETE\n outgoing_transaction = None\n if self.is_serialized:\n hostname = socket.gethostname()\n outgoing_transaction = OutgoingTransaction.objects.using(using).create(\n tx_name=self.instance._meta.label_lower,\n tx_pk=getattr(self.instance, self.primary_key_field.name),\n tx=self.encrypted_json(),\n timestamp=timestamp_datetime.strftime(\"%Y%m%d%H%M%S%f\"),\n producer=f\"{hostname}-{using}\",\n action=action,\n using=using,\n )\n return outgoing_transaction"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef encrypted_json(self):\n json = serialize(objects=[self.instance])\n encrypted_json = Cryptor().aes_encrypt(json, LOCAL_MODE)\n return encrypted_json", "response": "Returns an encrypted json serialized from self.\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef map_req(self, requirement):\n if isinstance(self._packages_map, dict):\n local_name = self._packages_map.get(requirement, requirement.name)\n else:\n local_name = self._packages_map(requirement)\n return local_name", "response": "Maps the name of an Requirement class to the name of the corresponding\n module in the environment."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef map_version(self, requirement, local_version):\n if isinstance(self._versions_map, dict):\n version = self._versions_map.get(requirement, {}).get(\n local_version, local_version)\n else:\n version = self._versions_map(requirement, local_version)\n return version", "response": "Maps a local version name to one recognised by the Requirement class\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef init_tag_processors(self):\n '''Register the TagProcessors that are bundled with doxytag2zealdb.'''\n register = self.register_tag_processor\n\n register('class', classTagProcessor(**self.opts))\n register('file', fileTagProcessor(**self.opts))\n register('namespace', namespaceTagProcessor(**self.opts))\n register('struct', structTagProcessor(**self.opts))\n register('union', unionTagProcessor(**self.opts))\n register('function', functionTagProcessor(**self.opts))\n register('define', defineTagProcessor(**self.opts))\n register('enumeration', enumerationTagProcessor(**self.opts))\n register('enumvalue', enumvalueTagProcessor(**self.opts))\n register('typedef', typedefTagProcessor(**self.opts))\n register('variable', variableTagProcessor(**self.opts))", "response": "Register the TagProcessors that are bundled with doxytag2zealdb."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef process(self):\n '''Run all tag processors.'''\n for tag_proc in self.tag_procs:\n before_count = self.entry_count\n self.run_tag_processor(tag_proc)\n after_count = self.entry_count\n\n if self.verbose:\n print('Inserted %d entries for \"%s\" tag processor' % (\n after_count - before_count, tag_proc), file=sys.stderr)\n\n if self.verbose:\n print('Inserted %d entries overall' % self.entry_count,\n file=sys.stderr)", "response": "Run all tag processors."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef run_tag_processor(self, tag_proc_name):\n '''Run a tag processor.\n\n Args:\n tag_proc_name: A string key that maps to the TagProcessor to run.\n '''\n tag_processor = self.tag_procs[tag_proc_name]\n\n for tag in tag_processor.find(self.soup):\n self.process_tag(tag_proc_name, tag)", "response": "Run a tag processor."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef process_tag(self, tag_proc_name, tag):\n '''Process a tag with a tag processor and insert a DB entry.\n\n Args:\n tag_proc_name: A string key that maps to the TagProcessor to use.\n tag: A BeautifulSoup Tag to process.\n '''\n tag_processor = self.tag_procs[tag_proc_name]\n\n db_entry = (tag_processor.get_name(tag),\n tag_processor.get_entry_type(tag),\n tag_processor.get_filename(tag))\n\n self.zeal_db.insert(*db_entry)\n\n self.entry_count += 1", "response": "Process a tag with a tag processor and insert a DB entry."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nwrap to itertools. groupby that returns a list of each group and returns a list of each group", "response": "def groupby(iterable, key=0, filter=None):\n \"\"\"\n wrapper to itertools.groupby that returns a list of each group, rather\n than a generator and accepts integers or strings as the key and\n automatically converts them to callables with itemgetter(key)\n\n Arguments:\n iterable: iterable\n key: string, int or callable that tells how to group\n\n Returns:\n an iterable where each item is the key and a *list* of that\n group. (itertools.groupby returns a generator of that group).\n\n e.g. groupby(iterable, 0)\n \"\"\"\n if isinstance(key, (basestring, int)):\n key = itemgetter(key)\n elif isinstance(key, (tuple, list)):\n key = itemgetter(*key)\n for label, grp in igroupby(iterable, key):\n yield label, list(grp)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef groups_of(n, iterable):\n args = [iter(iterable)] * n\n for x in izip_longest(*args):\n yield [v for v in x if v is not None]", "response": "Generator that yields n elements from iterable."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef run(self, *pipelines, **kwargs):\n if not pipelines:\n raise ArcanaUsageError(\"No pipelines provided to {}.run\"\n .format(self))\n # Get filter kwargs (NB: in Python 3 they could be in the kwarg list)\n subject_ids = kwargs.pop('subject_ids', None)\n visit_ids = kwargs.pop('visit_ids', None)\n session_ids = kwargs.pop('session_ids', None)\n clean_work_dir = kwargs.pop('clean_work_dir',\n self._clean_work_dir_between_runs)\n required_outputs = kwargs.pop('required_outputs', repeat(None))\n # Create name by combining pipelines\n name = '_'.join(p.name for p in pipelines)\n # Clean work dir if required\n if clean_work_dir:\n workflow_work_dir = op.join(self.work_dir, name)\n if op.exists(workflow_work_dir):\n shutil.rmtree(workflow_work_dir)\n # Trim the end of very large names to avoid problems with\n # workflow names exceeding system limits.\n name = name[:WORKFLOW_MAX_NAME_LEN]\n workflow = pe.Workflow(name=name, base_dir=self.work_dir)\n\n # Generate filter array to optionally restrict the run to certain\n # subject and visit IDs.\n tree = self.study.tree\n # Create maps from the subject|visit IDs to an index used to represent\n # them in the filter array\n subject_inds = {s.id: i for i, s in enumerate(tree.subjects)}\n visit_inds = {v.id: i for i, v in enumerate(tree.visits)}\n if not subject_ids and not visit_ids and not session_ids:\n # No filters applied so create a full filter array\n filter_array = np.ones((len(subject_inds), len(visit_inds)),\n dtype=bool)\n else:\n # Filters applied so create an empty filter array and populate\n # from filter lists\n filter_array = np.zeros((len(subject_inds), len(visit_inds)),\n dtype=bool)\n if subject_ids is not None:\n for subj_id in subject_ids:\n filter_array[subject_inds[subj_id], :] = True\n if visit_ids is not None:\n for visit_id in visit_ids:\n filter_array[:, visit_inds[visit_id]] = True\n if session_ids is not None:\n for subj_id, visit_id in session_ids:\n filter_array[subject_inds[subj_id],\n visit_inds[visit_id]] = True\n if not filter_array.any():\n raise ArcanaUsageError(\n \"Provided filters:\\n\" +\n (\" subject_ids: {}\\n\".format(', '.join(subject_ids))\n if subject_ids is not None else '') +\n (\" visit_ids: {}\\n\".format(', '.join(visit_ids))\n if visit_ids is not None else '') +\n (\" session_ids: {}\\n\".format(', '.join(session_ids))\n if session_ids is not None else '') +\n \"Did not match any sessions in the project:\\n\" +\n \" subject_ids: {}\\n\".format(', '.join(subject_inds)) +\n \" visit_ids: {}\\n\".format(', '.join(visit_inds)))\n\n # Stack of pipelines to process in reverse order of required execution\n stack = OrderedDict()\n\n def push_on_stack(pipeline, filt_array, req_outputs):\n if req_outputs is None:\n req_outputs = pipeline.output_names\n if pipeline.name in stack:\n # Pop pipeline from stack in order to add it to the end of the\n # stack and ensure it is run before all downstream pipelines\n prev_pipeline, prev_req_outputs, prev_filt_array = stack.pop(\n pipeline.name)\n if pipeline is not prev_pipeline and pipeline != prev_pipeline:\n raise ArcanaDesignError(\n \"Attempting to run two different pipelines with the \"\n \"same name, {} and {}\".format(pipeline, prev_pipeline))\n # Combined required outputs and filter array with previous\n # references to the pipeline\n req_outputs = copy(req_outputs)\n req_outputs.update(prev_req_outputs)\n filt_array = filt_array | prev_filt_array\n # If the pipeline to process contains summary outputs (i.e. 'per-\n # subject|visit|study' frequency), then we need to \"dialate\" the\n # filter array to include IDs across the scope of the study, e.g.\n # all subjects for per-vist, or all visits for per-subject.\n output_freqs = set(pipeline.output_frequencies)\n dialated_filt_array = self._dialate_array(filt_array,\n pipeline.joins)\n added = dialated_filt_array ^ filt_array\n if added.any():\n filt_array = dialated_filt_array\n # Invert the index dictionaries to get index-to-ID maps\n inv_subject_inds = {v: k for k, v in subject_inds.items()}\n inv_visit_inds = {v: k for k, v in visit_inds.items()}\n logger.warning(\n \"Dialated filter array used to process '{}' pipeline to \"\n \"include {} subject/visit IDs due to its '{}' summary \"\n \"outputs \".format(\n pipeline.name,\n ', '.join('({},{})'.format(inv_subject_inds[s],\n inv_visit_inds[v])\n for s, v in zip(*np.nonzero(added))),\n \"' and '\".join(output_freqs)))\n\n stack[pipeline.name] = pipeline, req_outputs, filt_array\n # Recursively add all prerequisites to stack\n for prq_getter, prq_req_outputs in pipeline.prerequisites.items():\n try:\n prereq = pipeline.study.pipeline(\n prq_getter, prq_req_outputs)\n push_on_stack(prereq, filt_array, prq_req_outputs)\n except (ArcanaMissingDataException,\n ArcanaOutputNotProducedException) as e:\n e.msg += (\"{}, which are required as inputs to the '{}' \"\n \"pipeline to produce '{}'\".format(\n pipeline.name, \"', '\".join(req_outputs)))\n raise e\n\n # Add all primary pipelines to the stack along with their prereqs\n for pipeline, req_outputs in zip(pipelines, required_outputs):\n push_on_stack(pipeline, filter_array, req_outputs)\n\n # Iterate through stack of required pipelines from upstream to\n # downstream\n for pipeline, req_outputs, flt_array in reversed(list(stack.values())):\n try:\n self._connect_pipeline(\n pipeline, req_outputs, workflow, subject_inds, visit_inds,\n flt_array, **kwargs)\n except ArcanaNoRunRequiredException:\n logger.info(\"Not running '{}' pipeline as its outputs \"\n \"are already present in the repository\"\n .format(pipeline.name))\n # Save complete graph for debugging purposes\n# workflow.write_graph(graph2use='flat', format='svg')\n# print('Graph saved in {} directory'.format(os.getcwd()))\n # Actually run the generated workflow\n result = workflow.run(plugin=self._plugin)\n # Reset the cached tree of filesets in the repository as it will\n # change after the pipeline has run.\n self.study.clear_caches()\n return result", "response": "Runs the given pipelines and returns the final report node which can be used to generate the next set of nodes."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconnects a pipeline to an overarching workflow that sets up iterators over subjects and visits.", "response": "def _connect_pipeline(self, pipeline, required_outputs, workflow,\n subject_inds, visit_inds, filter_array, force=False):\n \"\"\"\n Connects a pipeline to a overarching workflow that sets up iterators\n over subjects|visits present in the repository (if required) and\n repository source and sink nodes\n\n Parameters\n ----------\n pipeline : Pipeline\n The pipeline to connect\n required_outputs : set[str] | None\n The outputs required to be produced by this pipeline. If None all\n are deemed to be required\n workflow : nipype.pipeline.engine.Workflow\n The overarching workflow to connect the pipeline to\n subject_inds : dct[str, int]\n A mapping of subject ID to row index in the filter array\n visit_inds : dct[str, int]\n A mapping of visit ID to column index in the filter array\n filter_array : 2-D numpy.array[bool]\n A two-dimensional boolean array, where rows correspond to\n subjects and columns correspond to visits in the repository. True\n values represent a combination of subject & visit ID to include\n in the current round of processing. Note that if the 'force'\n flag is not set, sessions won't be reprocessed unless the\n save provenance doesn't match that of the given pipeline.\n force : bool | 'all'\n A flag to force the processing of all sessions in the filter\n array, regardless of whether the parameters|pipeline used\n to generate existing data matches the given pipeline\n \"\"\"\n if self.reprocess == 'force':\n force = True\n # Close-off construction of the pipeline and created, input and output\n # nodes and provenance dictionary\n pipeline.cap()\n # Prepend prerequisite pipelines to complete workflow if they need\n # to be (re)processed\n final_nodes = []\n # The array that represents the subject/visit pairs for which any\n # prerequisite pipeline will be (re)processed, and which therefore\n # needs to be included in the processing of the current pipeline. Row\n # indices correspond to subjects and column indices visits\n prqs_to_process_array = np.zeros((len(subject_inds), len(visit_inds)),\n dtype=bool)\n # The array that represents the subject/visit pairs for which any\n # prerequisite pipeline will be skipped due to missing inputs. Row\n # indices correspond to subjects and column indices visits\n prqs_to_skip_array = np.zeros((len(subject_inds), len(visit_inds)),\n dtype=bool)\n for getter_name in pipeline.prerequisites:\n prereq = pipeline.study.pipeline(getter_name)\n if prereq.to_process_array.any():\n final_nodes.append(prereq.node('final'))\n prqs_to_process_array |= prereq.to_process_array\n prqs_to_skip_array |= prereq.to_skip_array\n # Get list of sessions that need to be processed (i.e. if\n # they don't contain the outputs of this pipeline)\n to_process_array, to_protect_array, to_skip_array = self._to_process(\n pipeline, required_outputs, prqs_to_process_array,\n prqs_to_skip_array, filter_array, subject_inds, visit_inds, force)\n # Store the arrays signifying which nodes to process, protect or skip\n # so they can be passed to downstream pipelines\n pipeline.to_process_array = to_process_array\n pipeline.to_protect_array = to_protect_array\n pipeline.to_skip_array = to_skip_array\n # Check to see if there are any sessions to process\n if not to_process_array.any():\n raise ArcanaNoRunRequiredException(\n \"No sessions to process for '{}' pipeline\"\n .format(pipeline.name))\n # Set up workflow to run the pipeline, loading and saving from the\n # repository\n workflow.add_nodes([pipeline._workflow])\n # If prerequisite pipelines need to be processed, connect their\n # \"final\" nodes to the initial node of this pipeline to ensure that\n # they are all processed before this pipeline is run.\n if final_nodes:\n prereqs = pipeline.add('prereqs', Merge(len(final_nodes)))\n for i, final_node in enumerate(final_nodes, start=1):\n workflow.connect(final_node, 'out', prereqs, 'in{}'.format(i))\n else:\n prereqs = None\n # Construct iterator structure over subjects and sessions to be\n # processed\n iter_nodes = self._iterate(pipeline, to_process_array, subject_inds,\n visit_inds)\n sources = {}\n # Loop through each frequency present in the pipeline inputs and\n # create a corresponding source node\n for freq in pipeline.input_frequencies:\n try:\n inputs = list(pipeline.frequency_inputs(freq))\n except ArcanaMissingDataException as e:\n raise ArcanaMissingDataException(\n str(e) + \", which is required for pipeline '{}'\".format(\n pipeline.name))\n inputnode = pipeline.inputnode(freq)\n sources[freq] = source = pipeline.add(\n '{}_source'.format(freq),\n RepositorySource(\n i.collection for i in inputs),\n inputs=({'prereqs': (prereqs, 'out')} if prereqs is not None\n else {}))\n # Connect iter_nodes to source and input nodes\n for iterator in pipeline.iterators(freq):\n pipeline.connect(iter_nodes[iterator], iterator, source,\n iterator)\n pipeline.connect(source, iterator, inputnode,\n iterator)\n for input in inputs: # @ReservedAssignment\n pipeline.connect(source, input.suffixed_name,\n inputnode, input.name)\n deiter_nodes = {}\n\n def deiter_node_sort_key(it):\n \"\"\"\n If there are two iter_nodes (i.e. both subject and visit ID) and\n one depends on the other (i.e. if the visit IDs per subject\n vary and vice-versa) we need to ensure that the dependent\n iterator is deiterated (joined) first.\n \"\"\"\n return iter_nodes[it].itersource is None\n\n # Connect all outputs to the repository sink, creating a new sink for\n # each frequency level (i.e 'per_session', 'per_subject', 'per_visit',\n # or 'per_study')\n for freq in pipeline.output_frequencies:\n outputs = list(pipeline.frequency_outputs(freq))\n if pipeline.iterators(freq) - pipeline.iterators():\n raise ArcanaDesignError(\n \"Doesn't make sense to output '{}', which are of '{}' \"\n \"frequency, when the pipeline only iterates over '{}'\"\n .format(\"', '\".join(o.name for o in outputs), freq,\n \"', '\".join(pipeline.iterators())))\n outputnode = pipeline.outputnode(freq)\n # Connect filesets/fields to sink to sink node, skipping outputs\n # that are study inputs\n to_connect = {o.suffixed_name: (outputnode, o.name)\n for o in outputs if o.is_spec}\n # Connect iterators to sink node\n to_connect.update(\n {i: (iter_nodes[i], i) for i in pipeline.iterators()})\n # Connect checksums/values from sources to sink node in order to\n # save in provenance, joining where necessary\n for input_freq in pipeline.input_frequencies:\n checksums_to_connect = [\n i.checksum_suffixed_name\n for i in pipeline.frequency_inputs(input_freq)]\n if not checksums_to_connect:\n # Rare case of a pipeline with no inputs only iter_nodes\n # that will only occur in unittests in all likelihood\n continue\n # Loop over iterators that need to be joined, i.e. that are\n # present in the input frequency but not the output frequency\n # and create join nodes\n source = sources[input_freq]\n for iterator in (pipeline.iterators(input_freq) -\n pipeline.iterators(freq)):\n join = pipeline.add(\n '{}_to_{}_{}_checksum_join'.format(\n input_freq, freq, iterator),\n IdentityInterface(\n checksums_to_connect),\n inputs={\n tc: (source, tc) for tc in checksums_to_connect},\n joinsource=iterator,\n joinfield=checksums_to_connect)\n source = join\n to_connect.update(\n {tc: (source, tc) for tc in checksums_to_connect})\n # Add sink node\n sink = pipeline.add(\n '{}_sink'.format(freq),\n RepositorySink(\n (o.collection for o in outputs), pipeline),\n inputs=to_connect)\n # \"De-iterate\" (join) over iterators to get back to single child\n # node by the time we connect to the final node of the pipeline Set\n # the sink and subject_id as the default deiterator if there are no\n # deiterates (i.e. per_study) or to use as the upstream node to\n # connect the first deiterator for every frequency\n deiter_nodes[freq] = sink # for per_study the \"deiterator\" == sink\n for iterator in sorted(pipeline.iterators(freq),\n key=deiter_node_sort_key):\n # Connect to previous deiterator or sink\n # NB: we only need to keep a reference to the last one in the\n # chain in order to connect with the \"final\" node, so we can\n # overwrite the entry in the 'deiter_nodes' dict\n deiter_nodes[freq] = pipeline.add(\n '{}_{}_deiter'.format(freq, iterator),\n IdentityInterface(\n ['checksums']),\n inputs={\n 'checksums': (deiter_nodes[freq], 'checksums')},\n joinsource=iterator,\n joinfield='checksums')\n # Create a final node, which is used to connect with downstream\n # pipelines\n pipeline.add(\n 'final',\n Merge(\n len(deiter_nodes)),\n inputs={\n 'in{}'.format(i): (di, 'checksums')\n for i, di in enumerate(deiter_nodes.values(), start=1)})"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _iterate(self, pipeline, to_process_array, subject_inds, visit_inds):\n # Check to see whether the subject/visit IDs to process (as specified\n # by the 'to_process' array) can be factorized into indepdent nodes,\n # i.e. all subjects to process have the same visits to process and\n # vice-versa.\n factorizable = True\n if len(list(pipeline.iterators())) == 2:\n nz_rows = to_process_array[to_process_array.any(axis=1), :]\n ref_row = nz_rows[0, :]\n factorizable = all((r == ref_row).all() for r in nz_rows)\n # If the subject/visit IDs to process cannot be factorized into\n # indepedent iterators, determine which to make make dependent on the\n # other in order to avoid/minimise duplicatation of download attempts\n dependent = None\n if not factorizable:\n input_freqs = list(pipeline.input_frequencies)\n # By default pick iterator the one with the most IDs to\n # iterate to be the dependent in order to reduce the number of\n # nodes created and any duplication of download attempts across\n # the nodes (if both 'per_visit' and 'per_subject' inputs are\n # required\n num_subjs, num_visits = nz_rows[:, nz_rows.any(axis=0)].shape\n if num_subjs > num_visits:\n dependent = self.study.SUBJECT_ID\n else:\n dependent = self.study.VISIT_ID\n if 'per_visit' in input_freqs:\n if 'per_subject' in input_freqs:\n logger.warning(\n \"Cannot factorize sessions to process into independent\"\n \" subject and visit iterators and both 'per_visit' and\"\n \" 'per_subject' inputs are used by pipeline therefore\"\n \" per_{} inputs may be cached twice\".format(\n dependent[:-3]))\n else:\n dependent = self.study.SUBJECT_ID\n elif 'per_subject' in input_freqs:\n dependent = self.study.VISIT_ID\n # Invert the index dictionaries to get index-to-ID maps\n inv_subj_inds = {v: k for k, v in subject_inds.items()}\n inv_visit_inds = {v: k for k, v in visit_inds.items()}\n # Create iterator for subjects\n iter_nodes = {}\n if self.study.SUBJECT_ID in pipeline.iterators():\n fields = [self.study.SUBJECT_ID]\n if dependent == self.study.SUBJECT_ID:\n fields.append(self.study.VISIT_ID)\n # Add iterator node named after subject iterator\n subj_it = pipeline.add(self.study.SUBJECT_ID,\n IdentityInterface(fields))\n if dependent == self.study.SUBJECT_ID:\n # Subjects iterator is dependent on visit iterator (because of\n # non-factorizable IDs)\n subj_it.itersource = ('{}_{}'.format(pipeline.name,\n self.study.VISIT_ID),\n self.study.VISIT_ID)\n subj_it.iterables = [(\n self.study.SUBJECT_ID,\n {inv_visit_inds[n]: [inv_subj_inds[m]\n for m in col.nonzero()[0]]\n for n, col in enumerate(to_process_array.T)})]\n else:\n subj_it.iterables = (\n self.study.SUBJECT_ID,\n [inv_subj_inds[n]\n for n in to_process_array.any(axis=1).nonzero()[0]])\n iter_nodes[self.study.SUBJECT_ID] = subj_it\n # Create iterator for visits\n if self.study.VISIT_ID in pipeline.iterators():\n fields = [self.study.VISIT_ID]\n if dependent == self.study.VISIT_ID:\n fields.append(self.study.SUBJECT_ID)\n # Add iterator node named after visit iterator\n visit_it = pipeline.add(self.study.VISIT_ID,\n IdentityInterface(fields))\n if dependent == self.study.VISIT_ID:\n visit_it.itersource = ('{}_{}'.format(pipeline.name,\n self.study.SUBJECT_ID),\n self.study.SUBJECT_ID)\n visit_it.iterables = [(\n self.study.VISIT_ID,\n {inv_subj_inds[m]:[inv_visit_inds[n]\n for n in row.nonzero()[0]]\n for m, row in enumerate(to_process_array)})]\n else:\n visit_it.iterables = (\n self.study.VISIT_ID,\n [inv_visit_inds[n]\n for n in to_process_array.any(axis=0).nonzero()[0]])\n iter_nodes[self.study.VISIT_ID] = visit_it\n if dependent == self.study.SUBJECT_ID:\n pipeline.connect(visit_it, self.study.VISIT_ID,\n subj_it, self.study.VISIT_ID)\n if dependent == self.study.VISIT_ID:\n pipeline.connect(subj_it, self.study.SUBJECT_ID,\n visit_it, self.study.SUBJECT_ID)\n return iter_nodes", "response": "Generate nodes that iterate over subjects and visits in the study that need to be processed by the pipeline."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck whether the outputs of the pipeline are present in all sessions in the project repository and were generated with matching provenance. Return an 2D boolean array (subjects: rows, visits: cols) with the sessions to process marked True. Parameters ---------- pipeline : Pipeline The pipeline to determine the sessions to process required_ouputs : set[str] The names of the pipeline outputs that are required. If None all are deemed to be required prqs_to_process_array : 2-D numpy.array[bool] A two-dimensional boolean array, where rows and columns correspond correspond to subjects and visits in the repository tree. True values represent a subject/visit ID pairs that will be (re)processed in prerequisite pipelines and therefore need to be included in the returned array. to_skip_array : 2-D numpy.array[bool] Similar to prqs_to_process_array, except denote the subject/visits that are to be skipped due to missing inputs filter_array : 2-D numpy.array[bool] A two-dimensional boolean array, where rows and columns correspond correspond to subjects and visits in the repository tree. True values represent a subject/visit ID pairs to include in the current round of processing. Note that if the 'force' flag is not set, sessions won't be reprocessed unless the parameters and pipeline version saved in the provenance doesn't match that of the given pipeline. subject_inds : dict[str,int] Mapping from subject ID to index in filter|to_process arrays visit_inds : dict[str,int] Mapping from visit ID to index in filter|to_process arrays force : bool Whether to force reprocessing of all (filtered) sessions or not. Note that if 'force' is true we can't just return the filter array as it might be dilated by summary outputs (i.e. of frequency 'per_visit', 'per_subject' or 'per_study'). So we still loop through all outputs and treat them like they don't exist Returns ------- to_process_array : 2-D numpy.array[bool] A two-dimensional boolean array, where rows correspond to subjects and columns correspond to visits in the repository. True values represent subject/visit ID pairs to run the pipeline for", "response": "def _to_process(self, pipeline, required_outputs, prqs_to_process_array,\n to_skip_array, filter_array, subject_inds, visit_inds,\n force):\n \"\"\"\n Check whether the outputs of the pipeline are present in all sessions\n in the project repository and were generated with matching provenance.\n Return an 2D boolean array (subjects: rows, visits: cols) with the\n sessions to process marked True.\n\n Parameters\n ----------\n pipeline : Pipeline\n The pipeline to determine the sessions to process\n required_ouputs : set[str]\n The names of the pipeline outputs that are required. If None all\n are deemed to be required\n prqs_to_process_array : 2-D numpy.array[bool]\n A two-dimensional boolean array, where rows and columns correspond\n correspond to subjects and visits in the repository tree. True\n values represent a subject/visit ID pairs that will be\n (re)processed in prerequisite pipelines and therefore need to be\n included in the returned array.\n to_skip_array : 2-D numpy.array[bool]\n Similar to prqs_to_process_array, except denote the subject/visits\n that are to be skipped due to missing inputs\n filter_array : 2-D numpy.array[bool]\n A two-dimensional boolean array, where rows and columns correspond\n correspond to subjects and visits in the repository tree. True\n values represent a subject/visit ID pairs to include\n in the current round of processing. Note that if the 'force'\n flag is not set, sessions won't be reprocessed unless the\n parameters and pipeline version saved in the provenance doesn't\n match that of the given pipeline.\n subject_inds : dict[str,int]\n Mapping from subject ID to index in filter|to_process arrays\n visit_inds : dict[str,int]\n Mapping from visit ID to index in filter|to_process arrays\n force : bool\n Whether to force reprocessing of all (filtered) sessions or not.\n Note that if 'force' is true we can't just return the filter array\n as it might be dilated by summary outputs (i.e. of frequency\n 'per_visit', 'per_subject' or 'per_study'). So we still loop\n through all outputs and treat them like they don't exist\n\n Returns\n -------\n to_process_array : 2-D numpy.array[bool]\n A two-dimensional boolean array, where rows correspond to\n subjects and columns correspond to visits in the repository. True\n values represent subject/visit ID pairs to run the pipeline for\n \"\"\"\n # Reference the study tree in local variable for convenience\n tree = self.study.tree\n # Check to see if the pipeline has any low frequency outputs, because\n # if not then each session can be processed indepdently. Otherwise,\n # the \"session matrix\" (as defined by subject_ids and visit_ids\n # passed to the Study class) needs to be complete, i.e. a session\n # exists (with the full complement of requird inputs) for each\n # subject/visit ID pair.\n summary_outputs = [\n o.name for o in pipeline.outputs if o.frequency != 'per_session']\n # Set of frequencies present in pipeline outputs\n output_freqs = pipeline.output_frequencies\n if summary_outputs:\n if list(tree.incomplete_subjects):\n raise ArcanaUsageError(\n \"Can't process '{}' pipeline as it has low frequency \"\n \" outputs (i.e. outputs that aren't of 'per_session' \"\n \"frequency) ({}) and subjects ({}) that are missing one \"\n \"or more visits ({}). Please restrict the subject/visit \"\n \"IDs in the study __init__ to continue the analysis\"\n .format(\n self.name,\n ', '.join(summary_outputs),\n ', '.join(s.id for s in tree.incomplete_subjects),\n ', '.join(v.id for v in tree.incomplete_visits)))\n\n def array_inds(x):\n \"\"\"\n Get the array index corresponding to an item or node. If it doesn't\n have a subject and/or visit ID (i.e. frequency != 'per_session')\n then the first index in the row/column is selected as it will be\n dialated to the whole row/column later\n \"\"\"\n return (subject_inds.get(x.subject_id, 0),\n visit_inds.get(x.visit_id, 0))\n\n # Initalise array to represent which sessions need to be reprocessed\n to_process_array = np.zeros((len(subject_inds), len(visit_inds)),\n dtype=bool)\n # An array to mark outputs that have been altered outside of Arcana\n # and therefore protect from over-writing\n to_protect_array = np.copy(to_process_array)\n # Mark the sessions that we should check to see if the configuration\n # saved in the provenance record matches that of the current pipeline\n to_check_array = np.copy(to_process_array)\n # As well as the the sessions that need to be protected or skipped,\n # keep track of the items in those sessions that are protected or\n # missing for more informative error messages\n to_protect = defaultdict(list)\n to_skip = defaultdict(list)\n # Check data tree for missing inputs\n for input in pipeline.inputs: # @ReservedAssignment\n # NB: Study inputs that don't have skip_missing set and have\n # missing data should raise an error before this point\n if input.skip_missing:\n for item in input.collection:\n if not item.exists:\n to_skip_array[array_inds(item)] = True\n to_skip[array_inds(item)].append(item)\n # Dialate array over all iterators that are joined by the pipeline\n to_skip_array = self._dialate_array(to_skip_array, pipeline.joins)\n # Check data tree for missing required outputs\n for output in pipeline.outputs:\n # Check to see if output is required by downstream processing\n required = (required_outputs is None or\n output.name in required_outputs)\n for item in output.collection:\n if item.exists:\n # Check to see if checksums recorded when derivative\n # was generated by previous run match those of current file\n # set. If not we assume they have been manually altered and\n # therefore should not be overridden\n if item.checksums != item.recorded_checksums:\n logger.warning(\n \"Checksums for {} do not match those recorded in \"\n \"provenance. Assuming it has been manually \"\n \"corrected outside of Arcana and will therefore \"\n \"not overwrite. Please delete manually if this \"\n \"is not intended\".format(repr(item)))\n to_protect_array[array_inds(item)] = True\n to_protect[array_inds(item)].append(item)\n elif required:\n if force:\n to_process_array[array_inds(item)] = True\n else:\n to_check_array[array_inds(item)] = True\n elif required:\n to_process_array[array_inds(item)] = True\n # Filter sessions to process by those requested\n to_process_array *= filter_array\n to_check_array *= (filter_array * np.invert(to_process_array))\n if to_check_array.any() and self.prov_check:\n # Get list of sessions, subjects, visits, tree objects to check\n # their provenance against that of the pipeline\n to_check = []\n if 'per_session' in output_freqs:\n to_check.extend(\n s for s in tree.sessions\n if to_check_array[subject_inds[s.subject_id],\n visit_inds[s.visit_id]])\n if 'per_subject' in output_freqs:\n # We can just test the first col of outputs_exist as rows\n # should be either all True or all False\n to_check.extend(s for s in tree.subjects\n if to_check_array[subject_inds[s.id], 0])\n if 'per_visit' in output_freqs:\n # We can just test the first row of outputs_exist as cols\n # should be either all True or all False\n to_check.extend(v for v in tree.visits\n if to_check_array[0, visit_inds[v.id]])\n if 'per_study' in output_freqs:\n to_check.append(tree)\n for node in to_check:\n # Generated expected record from current pipeline/repository-\n # state\n requires_reprocess = False\n try:\n # Retrieve record stored in tree node\n record = node.record(pipeline.name, pipeline.study.name)\n expected_record = pipeline.expected_record(node)\n\n # Compare record with expected\n mismatches = record.mismatches(expected_record,\n self.prov_check,\n self.prov_ignore)\n if mismatches:\n msg = (\"mismatch in provenance:\\n{}\\n Add mismatching \"\n \"paths (delimeted by '/') to 'prov_ignore' \"\n \"argument of Processor to ignore\"\n .format(\n pformat(mismatches)))\n requires_reprocess = True\n except ArcanaNameError as e:\n msg = \"missing provenance record\"\n requires_reprocess = False\n to_protect_array[array_inds(node)] = True\n except ArcanaDataNotDerivedYetError as e:\n msg = (\"missing input '{}' and therefore cannot check \"\n \"provenance\".format(e.name))\n requires_reprocess = True\n if requires_reprocess:\n if self.reprocess:\n to_process_array[array_inds(node)] = True\n logger.info(\n \"Reprocessing {} with '{}' due to {}\"\n .format(node, pipeline.name, msg))\n else:\n raise ArcanaReprocessException(\n \"Cannot use derivatives for '{}' pipeline stored \"\n \"in {} due to {}, set 'reprocess' \"\n \"flag of Processor to overwrite\".format(\n pipeline.name, node, msg))\n\n def inds_to_ids(inds):\n subject_id = next(k for k, v in subject_inds.items()\n if v == inds[0])\n visit_id = next(k for k, v in visit_inds.items() if v == inds[1])\n return (subject_id, visit_id)\n\n # Dialate to process array\n to_process_array = self._dialate_array(to_process_array,\n pipeline.joins)\n intersection = to_process_array * to_skip_array\n if intersection.any():\n missing_inputs_msg = ''\n missing_prq_inputs_msg = ''\n for sess_inds in zip(*np.nonzero(intersection)):\n subject_id, visit_id = inds_to_ids(sess_inds)\n if sess_inds in to_skip:\n missing_inputs_msg += (\n \"\\n(subject={}, visit={}): [{}]\".format(\n subject_id, visit_id,\n ', '.join(i.name for i in to_skip[sess_inds])))\n else:\n missing_prq_inputs_msg += (\n \"\\n(subject={}, visit={})\".format(subject_id,\n visit_id))\n warning_msg = (\"Skipping the following nodes for '{}' pipeline due\"\n \" to \".format(pipeline))\n if missing_inputs_msg:\n warning_msg += \"missing inputs:{}\".format(missing_inputs_msg)\n if missing_prq_inputs_msg:\n if missing_inputs_msg:\n warning_msg += \"\\nand the following due to \"\n warning_msg += (\"missing inputs to prerequisite pipelines:{}\"\n .format(missing_prq_inputs_msg))\n logger.warning(warning_msg)\n # Remove nodes that are to be skipped due to missing inputs\n to_process_array *= np.invert(to_skip_array)\n # Check for conflicts between nodes to process and nodes to protect\n conflicting = to_process_array * to_protect_array\n if conflicting.any():\n error_msg = ''\n for sess_inds in zip(*np.nonzero(conflicting)):\n subject_id, visit_id = inds_to_ids(sess_inds)\n if required_outputs is None:\n conflict_outputs = pipeline.outputs\n else:\n conflict_outputs = [pipeline.study.bound_spec(r)\n for r in required_outputs]\n items = [\n o.collection.item(subject_id=subject_id, visit_id=visit_id)\n for o in conflict_outputs]\n missing = [i for i in items if i not in to_protect[sess_inds]]\n error_msg += (\n \"\\n({}, {}): protected=[{}], missing=[{}]\"\n .format(\n subject_id, visit_id,\n ', '.join(i.name for i in to_protect[sess_inds]),\n ', '.join(i.name for i in missing)))\n raise ArcanaProtectedOutputConflictError(\n \"Cannot process {} as there are nodes with both protected \"\n \"outputs (ones modified externally to Arcana) and missing \"\n \"required outputs. Either delete protected outputs or provide \"\n \"missing required outputs to continue:{}\".format(pipeline,\n error_msg))\n # Add in any prerequisites to process that aren't explicitly protected\n to_process_array |= (prqs_to_process_array *\n filter_array *\n np.invert(to_protect_array))\n to_process_array = self._dialate_array(to_process_array,\n pipeline.joins)\n return to_process_array, to_protect_array, to_skip_array"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _dialate_array(self, array, iterators):\n if not iterators:\n return array\n dialated = np.copy(array)\n if self.study.SUBJECT_ID in iterators:\n # If we join over subjects we should include all subjects for every\n # visit we want to process\n dialated[:, dialated.any(axis=0)] = True\n if self.study.VISIT_ID in iterators:\n # If we join over visits we should include all visits for every\n # subject we want to process\n dialated[dialated.any(axis=1), :] = True\n return dialated", "response": "Dialates a to_process array to include all subject and or visit for every unique entry in the set iterators."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _get_args(self, node, keywords):\n args = super(ArcanaSlurmGraphPlugin, self)._get_args(\n node, keywords)\n # Substitute the template arg with the node-specific one\n new_args = []\n for name, arg in zip(keywords, args):\n if name == 'template':\n new_args.append(self._processor.slurm_template(node))\n else:\n new_args.append(arg)\n return tuple(new_args)", "response": "Intercept calls to get template and return our own node - specific oneCOOKIE - set - key - value pairs template\n - a node - id"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef wall_time_str(self, wall_time):\n days = int(wall_time // 1440)\n hours = int((wall_time - days * 1440) // 60)\n minutes = int(math.floor(wall_time - days * 1440 - hours * 60))\n seconds = int((wall_time - math.floor(wall_time)) * 60)\n return \"{}-{:0>2}:{:0>2}:{:0>2}\".format(days, hours, minutes, seconds)", "response": "Returns the wall time in the format required for the sbatch script."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef sync(self):\n self._elk.add_handler('VN', self._vn_handler)\n self._elk.add_handler('XK', self._xk_handler)\n self._elk.add_handler('RP', self._rp_handler)\n self._elk.add_handler('IE', self._elk.call_sync_handlers)\n self._elk.add_handler('SS', self._ss_handler)\n self._elk.send(vn_encode())\n self._elk.send(lw_encode())\n self._elk.send(ss_encode())", "response": "Retrieve panel information from ElkM1"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nset the thermostat to the given value.", "response": "def set(self, element_to_set, value):\n \"\"\"(Helper) Set thermostat\"\"\"\n self._elk.send(ts_encode(self.index, value, element_to_set))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef renamed(self, name):\n duplicate = copy(self)\n duplicate._name = name\n return duplicate", "response": "Duplicate the datum and rename it"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef compare(self, other):\n if self._req != other._req:\n raise ArcanaUsageError(\n \"Can't compare versions of different requirements {} and {}\"\n .format(self._req, other._req))\n # Compare main sequence\n if self._seq < other._seq:\n return -1\n elif self._seq > other._seq:\n return 1\n # If main sequence is equal check prerelease. If a prerelease is\n # None then it is a full release which is > then a prerelease so we\n # just assign it 'z' (which is greater than 'a', 'b' and 'rc')\n s = self._prerelease if self._prerelease is not None else ('z',)\n o = other._prerelease if other._prerelease is not None else ('z',)\n if s < o:\n return -1\n if s > o:\n return 1\n # If both main sequence and prereleases are equal, compare post release\n s = self._post if self._post is not None else 0\n o = other._post if other._post is not None else 0\n if s < o:\n return -1\n if s > o:\n return 1\n # If both main sequence and prereleases are equal, compare development\n # release\n s = self._dev if self._dev is not None else 0\n o = other._dev if other._dev is not None else 0\n if s < o:\n return -1\n if s > o:\n return 1\n assert self == other\n return 0", "response": "Compares the version with another version."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nparse a typical version string into a tuple that can be used to pick the version number and the version number of the version.", "response": "def parse(self, version):\n \"\"\"\n Splits a typical version string (e.g. ..)\n into a tuple that can be sorted properly. Ignores all leading\n and trailing characters by using a regex search (instead of match) so\n as to pick the version string out of a block of text.\n\n Parameters\n ----------\n version : str | int | float | tuple(int)\n The string containing the version numbers, or alternatively an\n integer, float (number after decimal is interpreted as minor ver),\n or tuple|list containing the version sequence.\n\n Returns\n -------\n sequence : tuple(int | str)\n A tuple containing the main sequence of the version,\n e.g. ..\n prerelease : 2-tuple(str, int) | None\n A 2-tuple containing the type of prerelease ('a' - alpha,\n 'b' - beta, or 'rc' - release-canditate) and the number of the\n prerelease\n post : int |None\n The number of the post version\n dev : int | None\n The number of the development version\n \"\"\"\n # Check to see if version is not a string but rather another type\n # that can be interpreted as a version\n if isinstance(version, int):\n return (version,), None, None, None\n elif isinstance(version, (tuple, list)):\n return tuple(int(i) for i in version), None, None, None\n elif isinstance(version, float):\n major = math.floor(version)\n minor = version - major\n return (major, minor), None, None, None\n match = self.regex.search(version)\n if match is None:\n raise ArcanaVersionNotDetectableError(\n \"Could not parse version string {} as {}. Regex ({}) did not \"\n \"match any sub-string\".format(version, type(self).__name__,\n self.regex.pattern))\n sequence = []\n prerelease = None\n dev = None\n post = None\n for part in match.group(1).split(self.delimeter):\n if part.startswith('dev'):\n dev = int(part[len('dev'):])\n elif part.startswith('post'):\n post = int(part[len('post'):])\n else:\n # Split on non-numeric parts of the version string so that we\n # can detect prerelease\n sub_parts = re.split('([^\\d]+)', part)\n if sub_parts[0]:\n try:\n seq_part = int(sub_parts[0])\n except ValueError:\n seq_part = sub_parts[0]\n sequence.append(seq_part)\n if len(sub_parts) > 1:\n stage = sub_parts[1]\n try:\n pr_ver = int(sub_parts[2])\n except ValueError:\n raise ArcanaVersionNotDetectableError(\n \"Could not parse version string {} as {}. \"\n \"Did not recognise pre-release version {}\"\n .format(version, type(self).__name__,\n sub_parts[2]))\n stage = stage.strip('-_').lower()\n if not stage: # No prerelease info, assume a dev version\n assert dev is None\n dev = pr_ver\n continue\n if 'alpha'.startswith(stage):\n stage = 'a'\n elif 'beta'.startswith(stage):\n stage = 'b'\n elif stage == 'rc' or stage == 'release-canditate':\n stage = 'rc'\n else:\n raise ArcanaVersionNotDetectableError(\n \"Could not parse version string {} as {}. \"\n \"Did not recognise pre-release stage {}\"\n .format(version, type(self).__name__, stage))\n prerelease = (stage, pr_ver)\n return tuple(sequence), prerelease, post, dev"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning True if the current version is within the given version.", "response": "def within(self, version):\n \"\"\"\n A single version can also be interpreted as an open range (i.e. no\n maximum version)\n \"\"\"\n if not isinstance(version, Version):\n version = type(self._min_ver)(self._req, version)\n return version >= self"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef v(self, version, max_version=None, **kwargs):\n if not isinstance(version, Version):\n version = self.version_cls(self, version, **kwargs)\n # Return a version range instead of version\n if max_version is not None:\n if not isinstance(max_version, Version):\n max_version = self.version_cls(self, max_version, **kwargs)\n version = VersionRange(version, max_version)\n return version", "response": "Returns either a single version or a version range depending on whether two arguments are supplied or one."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nopens a connection to the database.", "response": "def open(self):\n '''Open a connection to the database.\n\n If a connection appears to be open already, transactions are committed\n and it is closed before proceeding. After establishing the connection,\n the searchIndex table is prepared (and dropped if it already exists).\n '''\n if self.conn is not None:\n self.close()\n\n self.conn = sqlite3.connect(self.filename)\n self.cursor = self.conn.cursor()\n\n c = self.cursor\n c.execute('SELECT name FROM sqlite_master WHERE type=\"table\"')\n if (u'searchIndex',) in c:\n c.execute('DROP TABLE searchIndex')\n\n if self.verbose:\n print('Dropped existing table', file=sys.stderr)\n\n c.executescript(\n '''\n CREATE TABLE searchIndex\n (id INTEGER PRIMARY KEY, name TEXT, type TEXT, path TEXT);\n\n CREATE UNIQUE INDEX anchor ON searchIndex (name, type, path);\n '''\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef insert(self, name, entry_type, filename):\n '''Insert an entry into the Zeal database.\n\n Args:\n name: A string representing the name of the entry.\n entry_type: A string representing the entry type.\n filename: A string representing the filename of the documentation\n for the entry.\n\n Raises:\n RuntimeError: a database connection was not established before\n calling insert()\n '''\n if self.cursor is None:\n raise RuntimeError(\n 'Open DB connection before attempting to call insert!')\n\n db_entry = (name, entry_type, filename)\n\n if self.verbose:\n print('Inserting %s \"%s\" -> %s' % db_entry, file=sys.stderr)\n\n self.cursor.execute(\n '''INSERT OR IGNORE INTO searchIndex(name, type, path)\n VALUES (?, ?, ?)''', db_entry)", "response": "Insert an entry into the Zeal database."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef data(self, name, subject_ids=None, visit_ids=None, session_ids=None,\n **kwargs):\n \"\"\"\n Returns the Fileset(s) or Field(s) associated with the provided spec\n name(s), generating derived filesets as required. Multiple names in a\n list can be provided, to allow their workflows to be combined into a\n single workflow.\n\n Parameters\n ----------\n name : str | List[str]\n The name of the FilesetSpec|FieldSpec to retried the\n filesets for\n subject_id : str | None\n The subject ID of the data to return. If provided (including None\n values) the data will be return as a single item instead of a\n collection\n visit_id : str | None\n The visit ID of the data to return. If provided (including None\n values) the data will be return as a single item instead of a\n c ollection\n subject_ids : list[str]\n The subject IDs to include in the returned collection\n visit_ids : list[str]\n The visit IDs to include in the returned collection\n session_ids : list[str]\n The session IDs (i.e. 2-tuples of the form\n (, ) to include in the returned collection\n\n Returns\n -------\n data : BaseItem | BaseCollection | list[BaseItem | BaseCollection]\n If 'subject_id' or 'visit_id' is provided then the data returned is\n a single Fileset or Field. Otherwise a collection of Filesets or\n Fields are returned. If muliple spec names are provided then a\n list of items or collections corresponding to each spec name.\n \"\"\"\n if isinstance(name, basestring):\n single_name = True\n names = [name]\n else:\n names = name\n single_name = False\n single_item = 'subject_id' in kwargs or 'visit_id' in kwargs\n filter_items = (subject_ids, visit_ids, session_ids) != (None, None,\n None)\n specs = [self.spec(n) for n in names]\n if single_item:\n if filter_items:\n raise ArcanaUsageError(\n \"Cannot provide 'subject_id' and/or 'visit_id' in \"\n \"combination with 'subject_ids', 'visit_ids' or \"\n \"'session_ids'\")\n subject_id = kwargs.pop('subject_id', None)\n visit_id = kwargs.pop('visit_id', None)\n iterators = set(chain(self.FREQUENCIES[s.frequency]\n for s in specs))\n if subject_id is not None and visit_id is not None:\n session_ids = [(subject_id, visit_id)]\n elif subject_id is not None:\n if self.VISIT_ID in iterators:\n raise ArcanaUsageError(\n \"Non-None values for visit IDs need to be \"\n \"provided to select a single item for each of '{}'\"\n .format(\"', '\".join(names)))\n subject_ids = [subject_id]\n elif visit_id is not None:\n if self.SUBJECT_ID in iterators:\n raise ArcanaUsageError(\n \"Non-None values for subject IDs need to be \"\n \"provided to select a single item for each of '{}'\"\n .format(\"', '\".join(names)))\n visit_ids = [visit_id]\n elif iterators:\n raise ArcanaUsageError(\n \"Non-None values for subject and/or visit IDs need to be \"\n \"provided to select a single item for each of '{}'\"\n .format(\"', '\".join(names)))\n # Work out which pipelines need to be run\n pipeline_getters = defaultdict(set)\n for spec in specs:\n if spec.derived or spec.derivable: # Filter out Study inputs\n # Add name of spec to set of required outputs\n pipeline_getters[spec.pipeline_getter].add(spec.name)\n # Run required pipelines\n if pipeline_getters:\n kwargs = copy(kwargs)\n kwargs.update({'subject_ids': subject_ids,\n 'visit_ids': visit_ids,\n 'session_ids': session_ids})\n pipelines, required_outputs = zip(*(\n (self.pipeline(k), v) for k, v in pipeline_getters.items()))\n kwargs['required_outputs'] = required_outputs\n self.processor.run(*pipelines, **kwargs)\n # Find and return Item/Collection corresponding to requested spec\n # names\n all_data = []\n for name in names:\n spec = self.bound_spec(name)\n data = spec.collection\n if single_item:\n data = data.item(subject_id=subject_id, visit_id=visit_id)\n elif filter_items and spec.frequency != 'per_study':\n if subject_ids is None:\n subject_ids = []\n if visit_ids is None:\n visit_ids = []\n if session_ids is None:\n session_ids = []\n if spec.frequency == 'per_session':\n data = [d for d in data\n if (d.subject_id in subject_ids or\n d.visit_id in visit_ids or\n d.session_id in session_ids)]\n elif spec.frequency == 'per_subject':\n data = [d for d in data\n if (d.subject_id in subject_ids or\n d.subject_id in [s[0] for s in session_ids])]\n elif spec.frequency == 'per_visit':\n data = [d for d in data\n if (d.visit_id in visit_ids or\n d.visit_id in [s[1] for s in session_ids])]\n if not data:\n raise ArcanaUsageError(\n \"No matching data found (subject_ids={}, visit_ids={} \"\n \", session_ids={})\"\n .format(subject_ids, visit_ids, session_ids))\n data = spec.CollectionClass(spec.name, data)\n if single_name:\n return data\n else:\n all_data.append(data)\n return all_data", "response": "Returns the data for the given spec - related fileset or Field."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nchecks whether the given switch matches the given value.", "response": "def branch(self, name, values=None): # @UnusedVariable @IgnorePep8\n \"\"\"\n Checks whether the given switch matches the value provided\n\n Parameters\n ----------\n name : str\n The name of the parameter to retrieve\n value : str | None\n The value(s) of the switch to match if a non-boolean switch\n \"\"\"\n if isinstance(values, basestring):\n values = [values]\n spec = self.parameter_spec(name)\n if not isinstance(spec, SwitchSpec):\n raise ArcanaUsageError(\n \"{} is standard parameter not a switch\".format(spec))\n switch = self._get_parameter(name)\n if spec.is_boolean:\n if values is not None:\n raise ArcanaDesignError(\n \"Should not provide values ({}) to boolean switch \"\n \"'{}' in {}\".format(\n values, name, self._param_error_location))\n in_branch = switch.value\n else:\n if values is None:\n raise ArcanaDesignError(\n \"Value(s) need(s) to be provided non-boolean switch\"\n \" '{}' in {}\".format(\n name, self._param_error_location))\n # Register parameter as being used by the pipeline\n unrecognised_values = set(values) - set(spec.choices)\n if unrecognised_values:\n raise ArcanaDesignError(\n \"Provided value(s) ('{}') for switch '{}' in {} \"\n \"is not a valid option ('{}')\".format(\n \"', '\".join(unrecognised_values), name,\n self._param_error_location,\n \"', '\".join(spec.choices)))\n in_branch = switch.value in values\n return in_branch"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef unhandled_branch(self, name):\n raise ArcanaDesignError(\n \"'{}' value of '{}' switch in {} is not handled\"\n .format(self._get_parameter(name), name,\n self._param_error_location))", "response": "This method raises an exception if a pipeline doesn t handle a particular switch value"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef save_workflow_graph_for(self, spec_name, fname, full=False,\n style='flat', **kwargs):\n \"\"\"\n Saves a graph of the workflow to generate the requested spec_name\n\n Parameters\n ----------\n spec_name : str\n Name of the spec to generate the graph for\n fname : str\n The filename for the saved graph\n style : str\n The style of the graph, can be one of can be one of\n 'orig', 'flat', 'exec', 'hierarchical'\n \"\"\"\n pipeline = self.spec(spec_name).pipeline\n if full:\n workflow = pe.Workflow(name='{}_gen'.format(spec_name),\n base_dir=self.processor.work_dir)\n self.processor._connect_pipeline(\n pipeline, workflow, **kwargs)\n else:\n workflow = pipeline._workflow\n fname = op.expanduser(fname)\n if not fname.endswith('.png'):\n fname += '.png'\n dotfilename = fname[:-4] + '.dot'\n workflow.write_graph(graph2use=style,\n dotfilename=dotfilename)", "response": "Saves a graph of the workflow for the given spec_name to fname."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef spec(self, name):\n # If the provided \"name\" is actually a data item or parameter then\n # replace it with its name.\n if isinstance(name, (BaseData, Parameter)):\n name = name.name\n # If name is a parameter than return the parameter spec\n if name in self._param_specs:\n return self._param_specs[name]\n else:\n return self.bound_spec(name)", "response": "Returns either the input corresponding to a fileset or field or a spec or a parameter that has either been passed to the study as an input or can be derived."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn an input selector or derived spec bound to the study i. e. the input selector or derived spec bound to the study i. e. the repository tree is checked for existing outputs", "response": "def bound_spec(self, name):\n \"\"\"\n Returns an input selector or derived spec bound to the study, i.e.\n where the repository tree is checked for existing outputs\n\n Parameters\n ----------\n name : Str\n A name of a fileset or field\n \"\"\"\n # If the provided \"name\" is actually a data item or parameter then\n # replace it with its name.\n if isinstance(name, BaseData):\n name = name.name\n # Get the spec from the class\n spec = self.data_spec(name)\n try:\n bound = self._inputs[name]\n except KeyError:\n if not spec.derived and spec.default is None:\n raise ArcanaMissingDataException(\n \"Acquired (i.e. non-generated) fileset '{}' \"\n \"was not supplied when the study '{}' was \"\n \"initiated\".format(name, self.name))\n else:\n try:\n bound = self._bound_specs[name]\n except KeyError:\n bound = self._bound_specs[name] = spec.bind(self)\n return bound"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef data_spec(cls, name):\n # If the provided \"name\" is actually a data item or parameter then\n # replace it with its name.\n if isinstance(name, BaseData):\n name = name.name\n try:\n return cls._data_specs[name]\n except KeyError:\n raise ArcanaNameError(\n name,\n \"No fileset spec named '{}' in {}, available:\\n{}\"\n .format(name, cls.__name__,\n \"\\n\".join(list(cls._data_specs.keys()))))", "response": "Returns the fileset_spec i. e. the template of the fileset expected to\n be supplied or generated corresponding to the fileset_spec name."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef cache_inputs(self):\n workflow = pe.Workflow(name='cache_download',\n base_dir=self.processor.work_dir)\n subjects = pe.Node(IdentityInterface(['subject_id']), name='subjects',\n environment=self.environment)\n sessions = pe.Node(IdentityInterface(['subject_id', 'visit_id']),\n name='sessions', environment=self.environment)\n subjects.iterables = ('subject_id', tuple(self.subject_ids))\n sessions.iterables = ('visit_id', tuple(self.visit_ids))\n source = pe.Node(RepositorySource(\n self.bound_spec(i).collection for i in self.inputs), name='source')\n workflow.connect(subjects, 'subject_id', sessions, 'subject_id')\n workflow.connect(sessions, 'subject_id', source, 'subject_id')\n workflow.connect(sessions, 'visit_id', source, 'visit_id')\n workflow.run()", "response": "Runs the Study s repository source node for each of the inputs\n of the study."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef provided(self, spec_name, default_okay=True):\n try:\n spec = self.bound_spec(spec_name)\n except ArcanaMissingDataException:\n return False\n if isinstance(spec, BaseInputSpec):\n return spec.default is not None and default_okay\n else:\n return True", "response": "Checks to see if the corresponding data spec was provided an explicit input."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the frequency corresponding to the given iterators", "response": "def freq_from_iterators(cls, iterators):\n \"\"\"\n Returns the frequency corresponding to the given iterators\n \"\"\"\n return {\n set(it): f for f, it in cls.FREQUENCIES.items()}[set(iterators)]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef prov(self):\n # Get list of repositories where inputs to the study are stored\n input_repos = list(set((i.repository for i in self.inputs)))\n inputs = {}\n for input in self.inputs: # @ReservedAssignment\n inputs[input.name] = {\n 'repository_index': input_repos.index(input.repository)}\n if input.frequency == 'per_study':\n inputs[input.name]['names'] = next(input.collection).name\n elif input.frequency == 'per_subject':\n inputs[input.name]['names'] = {i.subject_id: i.name\n for i in input.collection}\n elif input.frequency == 'per_visit':\n inputs[input.name]['names'] = {i.visit_id: i.name\n for i in input.collection}\n elif input.frequency == 'per_session':\n names = defaultdict(dict)\n for item in input.collection:\n names[item.subject_id][item.visit_id] = item.name\n # Convert from defaultdict to dict\n inputs[input.name]['names'] = dict(names.items())\n return {\n 'name': self.name,\n 'type': get_class_info(type(self)),\n 'parameters': {p.name: p.value for p in self.parameters},\n 'inputs': inputs,\n 'environment': self.environment.prov,\n 'repositories': [r.prov for r in input_repos],\n 'processor': self.processor.prov,\n 'subject_ids': self.subject_ids,\n 'visit_ids': self.visit_ids}", "response": "Extracts the provenance information from the study for storage alongside\n "} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\narm system at specified level.", "response": "def arm(self, level, code):\n \"\"\"(Helper) Arm system at specified level (away, vacation, etc)\"\"\"\n self._elk.send(al_encode(level, self._index, code))"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndisplays a message on all of the keypads in this area.", "response": "def display_message(self, clear, beep, timeout, line1, line2):\n \"\"\"Display a message on all of the keypads in this area.\"\"\"\n self._elk.send(\n dm_encode(self._index, clear, beep, timeout, line1, line2)\n )"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef sync(self):\n self.elk.send(as_encode())\n self.get_descriptions(TextDescriptions.AREA.value)", "response": "Retrieve areas from ElkM1"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef level(self, level, time=0):\n if level <= 0:\n self._elk.send(pf_encode(self._index))\n elif level >= 98:\n self._elk.send(pn_encode(self._index))\n else:\n self._elk.send(pc_encode(self._index, 9, level, time))", "response": "(Helper) Set light to specified level"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sync(self):\n for i in range(4):\n self.elk.send(ps_encode(i))\n self.get_descriptions(TextDescriptions.LIGHT.value)", "response": "Retrieve lights from ElkM1"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef dir_modtime(dpath):\n return max(os.path.getmtime(d) for d, _, _ in os.walk(dpath))", "response": "Returns the modification time of all files in a directory."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse_single_value(value):\n if isinstance(value, basestring):\n try:\n if value.startswith('\"') and value.endswith('\"'):\n value = str(value[1:-1])\n elif '.' in value:\n value = float(value)\n else:\n value = int(value)\n except ValueError:\n value = str(value)\n elif not isinstance(value, (int, float)):\n raise ArcanaUsageError(\n \"Unrecognised type for single value {}\".format(value))\n return value", "response": "Parses a single value from the Arcana database into a single value."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef iscontainer(*items):\n return all(isinstance(i, Iterable) and not isinstance(i, basestring)\n for i in items)", "response": "Checks whether all the provided items are containers."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfinds where two objects differ in nested containers and returns a string that contains the mismatch.", "response": "def find_mismatch(first, second, indent=''):\n \"\"\"\n Finds where two objects differ, iterating down into nested containers\n (i.e. dicts, lists and tuples) They can be nested containers\n any combination of primary dtypes, str, int, float, dict and lists\n\n Parameters\n ----------\n first : dict | list | tuple | str | int | float\n The first object to compare\n second : dict | list | tuple | str | int | float\n The other object to compare with the first\n indent : str\n The amount newlines in the output string should be indented. Provide\n the actual indent, i.e. a string of spaces.\n\n Returns\n -------\n mismatch : str\n Human readable output highlighting where two container differ.\n \"\"\"\n\n # Basic case where we are dealing with non-containers\n if not (isinstance(first, type(second)) or\n isinstance(second, type(first))):\n mismatch = (' types: self={} v other={}'\n .format(type(first).__name__, type(second).__name__))\n elif not iscontainer(first, second):\n mismatch = ': self={} v other={}'.format(first, second)\n else:\n sub_indent = indent + ' '\n mismatch = ''\n if isinstance(first, dict):\n if sorted(first.keys()) != sorted(second.keys()):\n mismatch += (' keys: self={} v other={}'\n .format(sorted(first.keys()),\n sorted(second.keys())))\n else:\n mismatch += \":\"\n for k in first:\n if first[k] != second[k]:\n mismatch += (\"\\n{indent}'{}' values{}\"\n .format(k,\n find_mismatch(first[k], second[k],\n indent=sub_indent),\n indent=sub_indent))\n else:\n mismatch += \":\"\n for i, (f, s) in enumerate(zip_longest(first, second)):\n if f != s:\n mismatch += (\"\\n{indent}{} index{}\"\n .format(i,\n find_mismatch(f, s,\n indent=sub_indent),\n indent=sub_indent))\n return mismatch"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a generator of deserialized objects.", "response": "def deserialize(json_text=None):\n \"\"\"Returns a generator of deserialized objects.\n\n Wraps django deserialize with defaults for JSON\n and natural keys.\n\n See https://docs.djangoproject.com/en/2.1/topics/serialization/\n \"\"\"\n\n return serializers.deserialize(\n \"json\",\n json_text,\n ensure_ascii=True,\n use_natural_foreign_keys=True,\n use_natural_primary_keys=False,\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create_auth_token(sender, instance, raw, created, **kwargs):\n if not raw:\n if created:\n sender.objects.create(user=instance)", "response": "Create token when a user is created."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nserialize the m2m of the given object.", "response": "def serialize_m2m_on_save(sender, action, instance, using, **kwargs):\n \"\"\" Part of the serialize transaction process that ensures m2m are\n serialized correctly.\n\n Skip those not registered.\n \"\"\"\n if action == \"post_add\":\n try:\n wrapped_instance = site_offline_models.get_wrapped_instance(instance)\n except ModelNotRegistered:\n pass\n else:\n wrapped_instance.to_outgoing_transaction(using, created=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nserializes the model instance as an OutgoingTransaction.", "response": "def serialize_on_save(sender, instance, raw, created, using, **kwargs):\n \"\"\" Serialize the model instance as an OutgoingTransaction.\n\n Skip those not registered.\n \"\"\"\n if not raw:\n if \"historical\" not in instance._meta.label_lower:\n try:\n wrapped_instance = site_offline_models.get_wrapped_instance(instance)\n except ModelNotRegistered:\n pass\n else:\n wrapped_instance.to_outgoing_transaction(using, created=created)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nserializing the history instance as an OutgoingTransaction.", "response": "def serialize_history_on_post_create(history_instance, using, **kwargs):\n \"\"\" Serialize the history instance as an OutgoingTransaction.\n\n Skip those not registered.\n \"\"\"\n try:\n wrapped_instance = site_offline_models.get_wrapped_instance(history_instance)\n except ModelNotRegistered:\n pass\n else:\n wrapped_instance.to_outgoing_transaction(using, created=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nserializes a serialized OutgoingTransaction when a model instance is deleted.", "response": "def serialize_on_post_delete(sender, instance, using, **kwargs):\n \"\"\"Creates a serialized OutgoingTransaction when\n a model instance is deleted.\n\n Skip those not registered.\n \"\"\"\n try:\n wrapped_instance = site_offline_models.get_wrapped_instance(instance)\n except ModelNotRegistered:\n pass\n else:\n wrapped_instance.to_outgoing_transaction(using, created=False, deleted=True)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set(self, value):\n self._elk.send(cw_encode(self._index, value, self.value_format))", "response": "(Helper) Set custom value."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sync(self):\n self.elk.send(cp_encode())\n self.get_descriptions(TextDescriptions.SETTING.value)", "response": "Retrieve custom values from ElkM1"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef paths(self):\n if self.format is None:\n raise ArcanaFileFormatError(\n \"Cannot get paths of fileset ({}) that hasn't had its format \"\n \"set\".format(self))\n if self.format.directory:\n return chain(*((op.join(root, f) for f in files)\n for root, _, files in os.walk(self.path)))\n else:\n return chain([self.path], self.aux_files.values())", "response": "Iterates through all files in the set and returns a list of all paths in the set."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef detect_format(self, candidates):\n if self._format is not None:\n raise ArcanaFileFormatError(\n \"Format has already been set for {}\".format(self))\n matches = [c for c in candidates if c.matches(self)]\n if not matches:\n raise ArcanaFileFormatError(\n \"None of the candidate file formats ({}) match {}\"\n .format(', '.join(str(c) for c in candidates), self))\n return matches[0]", "response": "Detects the format of the fileset from a list of possible file - formats."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef contents_equal(self, other, **kwargs):\n if hasattr(self.format, 'contents_equal'):\n equal = self.format.contents_equal(self, other, **kwargs)\n else:\n equal = (self.checksums == other.checksums)\n return equal", "response": "Tests the equality of the contents of this fileset with another fileset."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a copy of the AcquiredSpec bound to the given study.", "response": "def bind(self, study, **kwargs): # @UnusedVariable\n \"\"\"\n Returns a copy of the AcquiredSpec bound to the given study\n\n Parameters\n ----------\n study : Study\n A study to bind the fileset spec to (should happen in the\n study __init__)\n \"\"\"\n if self.default is None:\n raise ArcanaError(\n \"Attempted to bind '{}' to {} but only acquired specs with \"\n \"a default value should be bound to studies{})\".format(\n self.name, study))\n if self._study is not None:\n # This avoids rebinding specs to sub-studies that have already\n # been bound to the multi-study\n bound = self\n else:\n bound = copy(self)\n bound._study = study\n bound._default = bound.default.bind(study)\n return bound"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a copy of the Spec bound to the given study", "response": "def bind(self, study, **kwargs): # @UnusedVariable\n \"\"\"\n Returns a copy of the Spec bound to the given study\n\n Parameters\n ----------\n study : Study\n A study to bind the fileset spec to (should happen in the\n study __init__)\n \"\"\"\n if self._study is not None:\n # Avoid rebinding specs in sub-studies that have already\n # been bound to MultiStudy\n bound = self\n else:\n bound = copy(self)\n bound._study = study\n if not hasattr(study, self.pipeline_getter):\n raise ArcanaError(\n \"{} does not have a method named '{}' required to \"\n \"derive {}\".format(study, self.pipeline_getter,\n self))\n bound._bind_tree(study.tree)\n return bound"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef nodes(self, tree):\n # Run the match against the tree\n if self.frequency == 'per_session':\n nodes = []\n for subject in tree.subjects:\n for sess in subject.sessions:\n nodes.append(sess)\n elif self.frequency == 'per_subject':\n nodes = tree.subjects\n elif self.frequency == 'per_visit':\n nodes = tree.visits\n elif self.frequency == 'per_study':\n nodes = [tree]\n else:\n assert False, \"Unrecognised frequency '{}'\".format(\n self.frequency)\n return nodes", "response": "Returns the relevant nodes for the spec s frequency"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns True if the spec can be derived from the study s base_specs and the study s inputs and switches provided to the study s base_specs.", "response": "def derivable(self):\n \"\"\"\n Whether the spec (only valid for derived specs) can be derived\n given the inputs and switches provided to the study\n \"\"\"\n try:\n # Just need to iterate all study inputs and catch relevant\n # exceptions\n list(self.pipeline.study_inputs)\n except (ArcanaOutputNotProducedException,\n ArcanaMissingDataException):\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef reconnect_all(self):\n for role in self.Instances.keys():\n for connection in self.Instances[role]:\n connection.reconnect()", "response": "Re - establish connection to all instances"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_instance(self,\n role,\n instance,\n username='root',\n key_filename=None,\n output_shell=False):\n \"\"\"\n Add instance to the setup\n\n @param role: instance's role\n @type role: str\n\n @param instance: host parameters we would like to establish connection\n to\n @type instance: dict\n\n @param username: user name for creating ssh connection\n @type username: str\n\n @param key_filename: file name with ssh private key\n @type key_filename: str\n\n @param output_shell: write output from this connection to standard\n output\n @type output_shell: bool\n \"\"\"\n if not role in self.Instances.keys():\n self.Instances[role] = []\n self.logger.debug('Adding ' + role + ' with private_hostname ' +\n instance['private_hostname'] +\n ', public_hostname ' + instance['public_hostname'])\n self.Instances[role].append(Connection(instance,\n username,\n key_filename,\n output_shell=output_shell))", "response": "Add instance to the list of instances that are in the list of hosts that are connected to the specified role."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef setup_from_yamlfile(self, yamlfile, output_shell=False):\n self.logger.debug('Loading config from ' + yamlfile)\n with open(yamlfile, 'r') as yamlfd:\n yamlconfig = yaml.load(yamlfd)\n for instance in yamlconfig['Instances']:\n self.add_instance(instance['role'].upper(),\n instance,\n output_shell=output_shell)\n if 'Config' in yamlconfig.keys():\n self.logger.debug('Config found: ' + str(yamlconfig['Config']))\n self.config = yamlconfig['Config'].copy()", "response": "Setup from yaml file"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef process_iter(proc, cmd=\"\"):\n try:\n for l in proc.stdout:\n yield l\n finally:\n if proc.poll() is None:\n # there was an exception\n return\n else:\n proc.wait()\n if proc.returncode not in (0, None, signal.SIGPIPE, signal.SIGPIPE + 128):\n sys.stderr.write(\"cmd was:%s\\n\" % cmd)\n sys.stderr.write(\"return code was:%s\\n\" % proc.returncode)\n raise ProcessException(cmd)", "response": "helper function to iterate over a process stdout and report error messages when done\n "} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nchecks that all b files have been modified more recently than a", "response": "def is_newer_b(a, bfiles):\n \"\"\"\n check that all b files have been modified more recently than a\n \"\"\"\n if isinstance(bfiles, basestring):\n bfiles = [bfiles]\n\n if not op.exists(a): return False\n if not all(op.exists(b) for b in bfiles): return False\n\n atime = os.stat(a).st_mtime # modification time\n for b in bfiles:\n # a has been modified since\n if atime > os.stat(b).st_mtime:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef expect_list(connection, regexp_list, timeout=10):\n '''\n Expect a list of expressions\n\n @param connection: Connection to the host\n @type connection: L{Connection}\n\n @param regexp_list: regular expressions and associated return values\n @type regexp_list: list of (regexp, return value)\n\n @param timeout: timeout for performing expect operation\n @type timeout: int\n\n @return: propper return value from regexp_list\n @rtype: return value\n\n @raises ExpectFailed\n '''\n result = \"\"\n count = 0\n while count < timeout:\n try:\n recv_part = connection.channel.recv(32768).decode()\n logging.getLogger('stitches.expect').debug(\"RCV: \" + recv_part)\n if connection.output_shell:\n sys.stdout.write(recv_part)\n result += recv_part\n except socket.timeout:\n # socket.timeout here means 'no more data'\n pass\n\n for (regexp, retvalue) in regexp_list:\n # search for the first matching regexp and return desired value\n if re.match(regexp, result):\n return retvalue\n time.sleep(1)\n count += 1\n raise ExpectFailed(result)", "response": "Expect a list of expressions and associated return values"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nexpect one expression @param connection: Connection to the host @type connection: L{Connection} @param strexp: string to convert to expression (.*string.*) @type strexp: str @param timeout: timeout for performing expect operation @type timeout: int @return: True if succeeded @rtype: bool @raises ExpectFailed", "response": "def expect(connection, strexp, timeout=10):\n '''\n Expect one expression\n\n @param connection: Connection to the host\n @type connection: L{Connection}\n\n @param strexp: string to convert to expression (.*string.*)\n @type strexp: str\n\n @param timeout: timeout for performing expect operation\n @type timeout: int\n\n @return: True if succeeded\n @rtype: bool\n\n @raises ExpectFailed\n '''\n return Expect.expect_list(connection,\n [(re.compile(\".*\" + strexp + \".*\",\n re.DOTALL), True)],\n timeout)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef match(connection, regexp, grouplist=[1], timeout=10):\n '''\n Match against an expression\n\n @param connection: Connection to the host\n @type connection: L{Connection}\n\n @param regexp: compiled regular expression\n @type regexp: L{SRE_Pattern}\n\n @param grouplist: list of groups to return\n @type group: list of int\n\n @param timeout: timeout for performing expect operation\n @type timeout: int\n\n @return: matched string\n @rtype: str\n\n @raises ExpectFailed\n '''\n logging.getLogger('stitches.expect').debug(\"MATCHING: \" + regexp.pattern)\n result = \"\"\n count = 0\n while count < timeout:\n try:\n recv_part = connection.channel.recv(32768).decode()\n logging.getLogger('stitches.expect').debug(\"RCV: \" + recv_part)\n if connection.output_shell:\n sys.stdout.write(recv_part)\n result += recv_part\n except socket.timeout:\n # socket.timeout here means 'no more data'\n pass\n\n match = regexp.match(result)\n if match:\n ret_list = []\n for group in grouplist:\n logging.getLogger('stitches.expect').debug(\"matched: \" + match.group(group))\n ret_list.append(match.group(group))\n return ret_list\n time.sleep(1)\n count += 1\n raise ExpectFailed(result)", "response": "Match against an expression and return the result."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef ping_pong(connection, command, strexp, timeout=10):\n '''\n Enter a command and wait for something to happen (enter + expect\n combined)\n\n @param connection: connection to the host\n @type connection: L{Connection}\n\n @param command: command to execute\n @type command: str\n\n @param strexp: string to convert to expression (.*string.*)\n @type strexp: str\n\n @param timeout: timeout for performing expect operation\n @type timeout: int\n\n @return: True if succeeded\n @rtype: bool\n\n @raises ExpectFailed\n '''\n Expect.enter(connection, command)\n return Expect.expect(connection, strexp, timeout)", "response": "Execute a command and wait for something to happen"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef expect_retval(connection, command, expected_status=0, timeout=10):\n '''\n Run command and expect specified return valud\n\n @param connection: connection to the host\n @type connection: L{Connection}\n\n @param command: command to execute\n @type command: str\n\n @param expected_status: expected return value\n @type expected_status: int\n\n @param timeout: timeout for performing expect operation\n @type timeout: int\n\n @return: return value\n @rtype: int\n\n @raises ExpectFailed\n '''\n retval = connection.recv_exit_status(command, timeout)\n if retval is None:\n raise ExpectFailed(\"Got timeout (%i seconds) while executing '%s'\"\n % (timeout, command))\n elif retval != expected_status:\n raise ExpectFailed(\"Got %s exit status (%s expected)\\ncmd: %s\\nstdout: %s\\nstderr: %s\"\n % (retval, expected_status, connection.last_command,\n connection.last_stdout, connection.last_stderr))\n if connection.output_shell:\n sys.stdout.write(\"Run '%s', got %i return value\\n\"\n % (command, retval))\n return retval", "response": "Run command and expect specified return value"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nretrieve areas from ElkM1", "response": "def sync(self):\n \"\"\"Retrieve areas from ElkM1\"\"\"\n self.elk.send(ka_encode())\n self.get_descriptions(TextDescriptions.KEYPAD.value)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef find(self, soup):\n '''Yield tags matching the tag criterion from a soup.\n\n There is no need to override this if you are satisfied with finding\n tags that match match_criterion.\n\n Args:\n soup: A BeautifulSoup to search through.\n\n Yields:\n BeautifulSoup Tags that match the criterion.\n '''\n for tag in soup.recursiveChildGenerator():\n if self.match_criterion(tag):\n yield tag", "response": "Yield tags matching the tag criterion from a soup."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nextract and return a representative name from a tag.", "response": "def get_name(self, tag):\n '''Extract and return a representative \"name\" from a tag.\n\n Override as necessary. get_name's output can be controlled through\n keyword arguments that are provided when initializing a\n TagProcessor. For instance, a member of a class or namespace can have\n its parent scope included in the name by passing\n include_parent_scopes=True to __init__().\n\n Args:\n tag: A BeautifulSoup Tag that satisfies match_criterion.\n\n Returns:\n A string that would be appropriate to use as an entry name in a\n Zeal database.\n '''\n name = tag.findChild('name').contents[0]\n\n if self.include_parent_scopes:\n # Include parent scope in returned name\n parent_tag = tag.findParent()\n if parent_tag.get('kind') in ['class', 'struct', 'namespace']:\n name = parent_tag.findChild('name').contents[0] + '::' + name\n\n return name"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nextracts and return a documentation filename from a tag.", "response": "def get_filename(self, tag):\n '''Extract and return a documentation filename from a tag.\n\n Override as necessary, though this default implementation probably\n covers all the cases of interest.\n\n Args:\n tag: A BeautifulSoup Tag that satisfies match_criterion.\n\n Returns:\n A string that would be appropriate to use as the documentation\n filename for an entry in a Zeal database.\n '''\n if tag.find('filename', recursive=False) is not None:\n return tag.filename.contents[0]\n elif tag.find('anchorfile', recursive=False) is not None:\n return tag.anchorfile.contents[0] + '#' + tag.anchor.contents[0]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef match_criterion(self, tag):\n '''Override. Determine if a tag has the desired name and kind attribute\n value.\n\n Args:\n tag: A BeautifulSoup Tag.\n\n Returns:\n True if tag has the desired name and kind, otherwise False.\n '''\n return tag.name == self.reference_tag_name and \\\n tag.attrs.get('kind', '') == self.reference_tag_kind", "response": "Override. Determine if a tag has the desired name and kind attribute\n value."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_name(self, tag):\n '''Override. Extract a representative \"name\" from a function tag.\n\n get_name's output can be controlled through keyword arguments that are\n provided when initializing a functionTagProcessor. For instance,\n function arguments and return types can be included by passing\n include_function_signatures=True to __init__().\n\n Args:\n tag: A BeautifulSoup Tag for a function.\n\n Returns:\n A string that would be appropriate to use as an entry name for a\n function in a Zeal database.\n '''\n name = super(functionTagProcessor, self).get_name(tag)\n\n if self.include_function_signatures:\n # Include complete function signature in returned name\n func_args = tag.findChild('arglist')\n if func_args and len(func_args.contents):\n name += func_args.contents[0]\n\n ret_type = tag.findChild('type')\n if ret_type and len(ret_type.contents):\n name += ' -> ' + ret_type.contents[0]\n\n return name", "response": "Override. Extract a representative name from a function tag."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\noverriding that returns uMethod for class or struct methods. Override as necessary.", "response": "def get_entry_type(self, tag):\n '''Override that returns u'Method' for class/struct methods.\n\n Override as necessary.\n\n Args:\n tag: A BeautifulSoup Tag for a function.\n\n Returns:\n If this is a class/struct method, returns u'Method', otherwise\n returns the value from the inherited implementation of\n get_entry_type (which should be u'Function').\n '''\n\n if tag.findParent().get('kind') in ['class', 'struct']:\n return u'Method'\n\n return super(functionTagProcessor, self).get_entry_type(tag)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef pool(n=None, dummy=False):\n\n if dummy:\n from multiprocessing.dummy import Pool\n else:\n from multiprocessing import Pool\n if n is None:\n import multiprocessing\n n = multiprocessing.cpu_count() - 1\n\n return Pool(n)", "response": "create a multiprocessing pool that responds to interrupts."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nparallel map of a function to an iterable if each item in iterable is itself an iterable, then automatically call f(*item) instead of f(item) Arguments: f: function iterable: any iterable where each item is sent to f n: number of cpus (default is number on machine) dummy: use dummy pool. p: existing pool to re-use", "response": "def pmap(f, iterable, n=None, dummy=False, p=None):\n \"\"\"\n parallel map of a function to an iterable\n if each item in iterable is itself an iterable, then\n automatically call f(*item) instead of f(item)\n\n Arguments:\n f: function\n iterable: any iterable where each item is sent to f\n n: number of cpus (default is number on machine)\n dummy: use dummy pool.\n p: existing pool to re-use\n \"\"\"\n\n # make it easier to debug.\n if n == 1:\n for r in it.starmap(f, iterable):\n yield r\n raise StopIteration\n\n\n if p is None:\n po = pool(n, dummy)\n else:\n po = p\n assert hasattr(po, 'imap')\n f = _func_star(f)\n\n try:\n for r in po.imap(f, iterable):\n yield r\n\n # explicitly clean up created pool\n finally:\n if p is None:\n try:\n po.close()\n po.join()\n except:\n pass"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncall all callbacks when attribute of element changes", "response": "def _call_callbacks(self):\n \"\"\"Callbacks when attribute of element changes\"\"\"\n for callback in self._callbacks:\n callback(self, self._changeset)\n self._changeset = {}"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting the attribute of the object to new_value. If the attribute value has changed then set it and call the callbacks.", "response": "def setattr(self, attr, new_value, close_the_changeset=True):\n \"\"\"If attribute value has changed then set it and call the callbacks\"\"\"\n existing_value = getattr(self, attr, None)\n if existing_value != new_value:\n setattr(self, attr, new_value)\n self._changeset[attr] = new_value\n\n if close_the_changeset and self._changeset:\n self._call_callbacks()"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a default name for based on class and index of element", "response": "def default_name(self, separator='-'):\n \"\"\"Return a default name for based on class and index of element\"\"\"\n return self.__class__.__name__ + '{}{:03d}'.format(\n separator, self._index+1)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\npackages up the public attributes as a dict.", "response": "def as_dict(self):\n \"\"\"Package up the public attributes as a dict.\"\"\"\n attrs = vars(self)\n return {key: attrs[key] for key in attrs if not key.startswith('_')}"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the descriptions for the specified type.", "response": "def get_descriptions(self, description_type):\n \"\"\"\n Gets the descriptions for specified type.\n When complete the callback is called with a list of descriptions\n \"\"\"\n (desc_type, max_units) = description_type\n results = [None] * max_units\n self.elk._descriptions_in_progress[desc_type] = (max_units,\n results,\n self._got_desc)\n self.elk.send(sd_encode(desc_type=desc_type, unit=0))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef page_url(self) -> str:\n url = self.attributes['canonicalurl']\n assert isinstance(url, str)\n return url", "response": "Returns the canonical url of the page."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef normalize_locale_code(locale: Union[Locale, str]) -> str:\n if not isinstance(locale, Locale):\n locale = Locale.parse(locale.replace('-', '_'))\n return str(locale)", "response": "Determine the normalized locale code string."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get(self, entity_id: EntityId, load: bool = False) -> Entity:\n try:\n entity = self.identity_map[entity_id]\n except KeyError:\n entity = Entity(entity_id, self)\n self.identity_map[entity_id] = entity\n if load:\n entity.load()\n return entity", "response": "Get a Wikidata entity by its entity id."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nguess the entity type from the given entity id.", "response": "def guess_entity_type(self, entity_id: EntityId) -> Optional[EntityType]:\n r\"\"\"Guess :class:`~.entity.EntityType` from the given\n :class:`~.entity.EntityId`. It could return :const:`None` when it\n fails to guess.\n\n .. note::\n\n It always fails to guess when :attr:`entity_type_guess`\n is configued to :const:`False`.\n\n :return: The guessed :class:`~.entity.EntityId`, or :const:`None`\n if it fails to guess.\n :rtype: :class:`~typing.Optional`\\ [:class:`~.entity.EntityType`]\n\n .. versionadded:: 0.2.0\n\n \"\"\"\n if not self.entity_type_guess:\n return None\n if entity_id[0] == 'Q':\n return EntityType.item\n elif entity_id[0] == 'P':\n return EntityType.property\n return None"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef decode_datavalue(self,\n datatype: str,\n datavalue: Mapping[str, object]) -> object:\n \"\"\"Decode the given ``datavalue`` using the configured\n :attr:`datavalue_decoder`.\n\n .. versionadded:: 0.3.0\n\n \"\"\"\n decode = cast(Callable[[Client, str, Mapping[str, object]], object],\n self.datavalue_decoder)\n return decode(self, datatype, datavalue)", "response": "Decode the given datavalue using the configured datavalue_decoder."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate or update a cache.", "response": "def set(self, key: CacheKey, value: Optional[CacheValue]) -> None:\n r\"\"\"Create or update a cache.\n\n :param key: A key string to create or update.\n :type key: :const:`CacheKey`\n :param value: A value to cache. :const:`None` to remove cache.\n :type value: :class:`~typing.Optional`\\ [:const:`CacheValue`]\n\n \"\"\"\n raise NotImplementedError(\n 'Concreate subclasses of {0.__module__}.{0.__qualname__} have to '\n 'override .set() method'.format(CachePolicy)\n )"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef getlist(self, key: 'Entity') -> Sequence[object]:\n if not (isinstance(key, type(self)) and\n key.type is EntityType.property):\n return []\n claims_map = self.attributes.get('claims') or {}\n assert isinstance(claims_map, collections.abc.Mapping)\n claims = claims_map.get(key.id, [])\n claims.sort(key=lambda claim: claim['rank'], # FIXME\n reverse=True)\n logger = logging.getLogger(__name__ + '.Entity.getitem')\n if logger.isEnabledFor(logging.DEBUG):\n logger.debug('claim data: %s',\n __import__('pprint').pformat(claims))\n decode = self.client.decode_datavalue\n return [decode(snak['datatype'], snak['datavalue'])\n for snak in (claim['mainsnak'] for claim in claims)]", "response": "r Returns all values associated to the given key property\n in sequence."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the type of the entity.", "response": "def type(self) -> EntityType:\n \"\"\"(:class:`EntityType`) The type of entity, :attr:`~EntityType.item`\n or :attr:`~EntityType.property`.\n\n .. versionadded:: 0.2.0\n\n \"\"\"\n if self.data is None:\n guessed_type = self.client.guess_entity_type(self.id)\n if guessed_type is not None:\n return guessed_type\n # If guessing was failed follow the straightforward way.\n return EntityType(self.attributes['type'])"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates the user - role association table so that the user - role table correctly references your own UserMixin subclass.", "response": "def make_user_role_table(table_name='user', id_column_name='id'):\n \"\"\"\n Create the user-role association table so that\n it correctly references your own UserMixin subclass.\n\n \"\"\"\n\n return db.Table('fp_user_role',\n db.Column(\n 'user_id', db.Integer, db.ForeignKey('{}.{}'.format(\n table_name, id_column_name))),\n db.Column(\n 'role_id', db.Integer, db.ForeignKey('fp_role.id')),\n extend_existing=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ntake an ability (a string name of either a role or an ability) and returns the function if the user has that ability", "response": "def user_has(ability, get_user=import_user):\n \"\"\"\n Takes an ability (a string name of either a role or an ability) and returns the function if the user has that ability\n \"\"\"\n def wrapper(func):\n @wraps(func)\n def inner(*args, **kwargs):\n from .models import Ability\n desired_ability = Ability.query.filter_by(\n name=ability).first()\n user_abilities = []\n current_user = get_user()\n for role in current_user._roles:\n user_abilities += role.abilities\n if desired_ability in user_abilities:\n return func(*args, **kwargs)\n else:\n raise Forbidden(\"You do not have access\")\n return inner\n return wrapper"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef user_is(role, get_user=import_user):\n def wrapper(func):\n @wraps(func)\n def inner(*args, **kwargs):\n from .models import Role\n current_user = get_user()\n if role in current_user.roles:\n return func(*args, **kwargs)\n raise Forbidden(\"You do not have access\")\n return inner\n return wrapper", "response": "Decorator that returns a function if the user has the specified role"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngenerate a combo file for the given path.", "response": "def combo_serve(request, path, client):\n \"\"\"\n Handles generating a 'combo' file for the given path. This is similar to\n what happens when we upload to S3. Processors are applied, and we get\n the value that we would if we were serving from S3. This is a good way\n to make sure combo files work as intended before rolling out\n to production.\n \"\"\"\n joinfile = path\n sourcefiles = msettings['JOINED'][path]\n # Generate the combo file as a string.\n combo_data, dirname = combine_files(joinfile, sourcefiles, client)\n \n if path.endswith('.css'):\n mime_type = 'text/css'\n elif joinfile.endswith('.js'):\n mime_type = 'application/javascript'\n\n return HttpResponse(combo_data, mimetype=mime_type)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _form_key_str(path):\n if path.endswith('.css'):\n media_path_prefix = msettings['CSS_PATH']\n elif path.endswith('.js'):\n media_path_prefix = msettings['JS_PATH']\n else:\n # This isn't a CSS/JS file, no combo for you.\n return None\n\n if media_path_prefix:\n # CS/JSS path prefix has been set. Factor that into the key lookup.\n if not media_path_prefix.endswith('/'):\n # We need to add this slash so we can lop it off the 'path'\n # variable, to match the value in the JOINED dict.\n media_path_prefix += '/'\n\n if path.startswith(media_path_prefix):\n # Given path starts with the CSS/JS media prefix. Lop this off\n # so we can perform a lookup in the JOINED dict.\n return path[len(media_path_prefix):]\n else:\n # Path is in a root dir, send along as-is.\n return path\n\n # No CSS/JS path prefix set. Keep it raw.\n return path", "response": "Given a URL path return a key that can be used for the URL path."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfind the combo file key for a given path.", "response": "def _find_combo_match(path):\n \"\"\"\n Calculate the key to check the MEDIASYNC['JOINED'] dict for, perform the\n lookup, and return the matching key string if a match is found. If no\n match is found, return None instead.\n \"\"\"\n key_str = _form_key_str(path)\n if not key_str:\n # _form_key_str() says this isn't even a CSS/JS file.\n return None\n\n if not msettings['JOINED'].has_key(key_str):\n # No combo file match found. Must be an single file.\n return None\n else:\n # Combo match found, return the JOINED key.\n return key_str"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef static_serve(request, path, client):\n \n if msettings['SERVE_REMOTE']:\n # We're serving from S3, redirect there.\n url = client.remote_media_url().strip('/') + '/%(path)s'\n return redirect(url, permanent=True)\n\n if not msettings['SERVE_REMOTE'] and msettings['EMULATE_COMBO']:\n # Combo emulation is on and we're serving media locally. Try to see if\n # the given path matches a combo file defined in the JOINED dict in\n # the MEDIASYNC settings dict.\n combo_match = _find_combo_match(path)\n if combo_match:\n # We found a combo file match. Combine it and serve the result.\n return combo_serve(request, combo_match, client)\n\n # No combo file, but we're serving locally. Use the standard (inefficient)\n # Django static serve view.\n \n resp = serve(request, path, document_root=client.media_root, show_indexes=True)\n try:\n resp.content = client.process(resp.content, resp['Content-Type'], path)\n except KeyError:\n # HTTPNotModifiedResponse lacks the \"Content-Type\" key.\n pass\n return resp", "response": "This function is used to serve a media asset."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a HTTP server and start it.", "response": "def start_server(self, port_number, number_of_procs=0):\n \"\"\"\n Create a HTTP server and start it.\n\n :param int port_number: the port number to bind the server to\n :param int number_of_procs: number of processes to pass to\n Tornado's ``httpserver.HTTPServer.start``.\n\n If the application's ``debug`` setting is ``True``, then we are\n going to run in a single-process mode; otherwise, we'll let\n tornado decide how many sub-processes to spawn.\n\n The following additional configuration parameters can be set on the\n ``httpserver.HTTPServer`` instance by setting them in the application\n settings: ``xheaders``, ``max_body_size``, ``max_buffer_size``.\n\n \"\"\"\n signal.signal(signal.SIGTERM, self._on_signal)\n signal.signal(signal.SIGINT, self._on_signal)\n xheaders = self.application.settings.get('xheaders', False)\n max_body_size = self.application.settings.get('max_body_size', None)\n max_buffer_size = self.application.settings.get('max_buffer_size',\n None)\n\n self.server = httpserver.HTTPServer(\n self.application.tornado_application,\n xheaders=xheaders,\n max_body_size=max_body_size,\n max_buffer_size=max_buffer_size)\n if self.application.settings.get('debug', False):\n self.logger.info('starting 1 process on port %d', port_number)\n self.server.listen(port_number)\n else:\n self.logger.info('starting processes on port %d', port_number)\n self.server.bind(port_number, reuse_port=True)\n self.server.start(number_of_procs)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates the server and run the IOLoop.", "response": "def run(self, port_number, number_of_procs=0):\n \"\"\"\n Create the server and run the IOLoop.\n\n :param int port_number: the port number to bind the server to\n :param int number_of_procs: number of processes to pass to\n Tornado's ``httpserver.HTTPServer.start``.\n\n If the application's ``debug`` setting is ``True``, then we are\n going to run in a single-process mode; otherwise, we'll let\n tornado decide how many sub-processes based on the value of the\n ``number_of_procs`` argument. In any case, the application's\n *before_run* callbacks are invoked. If a callback raises an exception,\n then the application is terminated by calling :func:`sys.exit`.\n\n If any ``on_start`` callbacks are registered, they will be added to\n the Tornado IOLoop for execution after the IOLoop is started.\n\n The following additional configuration parameters can be set on the\n ``httpserver.HTTPServer`` instance by setting them in the application\n settings: ``xheaders``, ``max_body_size``, ``max_buffer_size``.\n\n \"\"\"\n self.start_server(port_number, number_of_procs)\n iol = ioloop.IOLoop.instance()\n\n try:\n self.application.start(iol)\n except Exception:\n self.logger.exception('application terminated during start, '\n 'exiting')\n sys.exit(70)\n\n iol.start()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef short_description(description):\n def decorator(func):\n if isinstance(func, property):\n func = func.fget\n func.short_description = description\n return func\n return decorator", "response": "Decorator to set the short_description attribute of a resource."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef add_module(self, module):\n '''\n Adds configuration parameters from a Python module.\n '''\n for key, value in module.__dict__.iteritems():\n if key[0:2] != '__':\n self.__setattr__(attr=key, value=value)", "response": "Adds configuration parameters from a Python module."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef redirect_to_login(next, redirect_field_name=REDIRECT_FIELD_NAME):\n resolved_url = reverse('IS:login')\n\n login_url_parts = list(urlparse(resolved_url))\n if redirect_field_name:\n querystring = QueryDict(login_url_parts[4], mutable=True)\n querystring[redirect_field_name] = next\n login_url_parts[4] = querystring.urlencode(safe='/')\n\n raise HTTPRedirectResponseException(urlunparse(login_url_parts))", "response": "Redirects the user to the login page passing the given next page"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _check_permission(self, name, obj=None):\n def redirect_or_exception(ex):\n if not self.request.user or not self.request.user.is_authenticated:\n if self.auto_login_redirect:\n redirect_to_login(self.request.get_full_path())\n else:\n raise HTTPUnauthorizedResponseException\n else:\n raise ex\n\n try:\n if not self._has_permission(name, obj):\n redirect_or_exception(HTTPForbiddenResponseException)\n except Http404 as ex:\n redirect_or_exception(ex)", "response": "Check if object is authorized and if object is not found and if object is authorized redirect to login page."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nuse when the plot method defined does not create a figure nor calls save_plot Then the plot method has to use self. fig", "response": "def plot_and_save(self, **kwargs):\n \"\"\"Used when the plot method defined does not create a figure nor calls save_plot\n Then the plot method has to use self.fig\"\"\"\n self.fig = pyplot.figure()\n self.plot()\n self.axes = pyplot.gca()\n self.save_plot(self.fig, self.axes, **kwargs)\n pyplot.close(self.fig)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsave an animation to a file.", "response": "def save_anim(self, fig, animate, init, bitrate=10000, fps=30):\n \"\"\"Not functional -- TODO\"\"\"\n from matplotlib import animation\n anim = animation.FuncAnimation(fig, animate, init_func=init, frames=360, interval=20)\n FFMpegWriter = animation.writers['ffmpeg']\n writer = FFMpegWriter(bitrate= bitrate, fps=fps)\n # Save #\n self.avi_path = self.base_dir + self.short_name + '.avi'\n anim.save(self.avi_path, writer=writer, codec='x264')"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nguess the server name based on environment variables.", "response": "def guess_server_name():\n \"\"\"We often use the same servers, which one are we running on now ?\"\"\"\n if os.environ.get('CSCSERVICE') == 'sisu': return \"sisu\"\n elif os.environ.get('SLURM_JOB_PARTITION') == 'halvan': return \"halvan\"\n elif os.environ.get('SNIC_RESOURCE') == 'milou': return \"milou\"\n elif os.environ.get('LAPTOP') == 'macbook_air': return \"macbook_air\"\n else: return \"unknown\""} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_instance(cls, instance_or_pk):\n\n if isinstance(instance_or_pk, cls):\n if instance_or_pk in cls._flask_signalbus_sa.session:\n return instance_or_pk\n instance_or_pk = inspect(cls).primary_key_from_instance(instance_or_pk)\n return cls.query.get(instance_or_pk)", "response": "Return a model instance in db. session."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef lock_instance(cls, instance_or_pk, read=False):\n\n mapper = inspect(cls)\n pk_attrs = [mapper.get_property_by_column(c).class_attribute for c in mapper.primary_key]\n pk_values = cls.get_pk_values(instance_or_pk)\n clause = and_(*[attr == value for attr, value in zip(pk_attrs, pk_values)])\n return cls.query.filter(clause).with_for_update(read=read).one_or_none()", "response": "Return a locked model instance in db. session."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a tuple of values for the current primary key.", "response": "def get_pk_values(cls, instance_or_pk):\n \"\"\"Return a primary key as a tuple.\n\n :param instance_or_pk: An instance of this model class, or a\n primary key. A composite primary key can be passed as a\n tuple.\n\n \"\"\"\n\n if isinstance(instance_or_pk, cls):\n cls._flask_signalbus_sa.session.flush()\n instance_or_pk = inspect(cls).primary_key_from_instance(instance_or_pk)\n return instance_or_pk if isinstance(instance_or_pk, tuple) else (instance_or_pk,)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef retry_on_integrity_error(self):\n\n session = self.session\n assert session.info.get(_ATOMIC_FLAG_SESSION_INFO_KEY), \\\n 'Calls to \"retry_on_integrity_error\" must be wrapped in atomic block.'\n session.flush()\n try:\n yield\n session.flush()\n except IntegrityError:\n raise DBSerializationError", "response": "Re - raise DBSerializationError if an IntegrityError occurs."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting all the translations for this object.", "response": "def _get_fieldtranslations(instance, field=None, lang=None):\n\t\"\"\"\n\tGet all the translations for this object.\n\t\"\"\"\n\n\t# Basic filtering: filter translations by module, model an object_id\n\t_filter = {\"module\": instance.__module__, \"model\": instance.__class__.__name__, \"object_id\": instance.id}\n\n\tif lang:\n\t\t_filter[\"lang\"] = lang\n\n\t# If we specify a field we get ONE translation (this field translation)\n\tif field:\n\t\t_filter[\"field\"] = field\n\t\ttry:\n\t\t\treturn FieldTranslation.objects.get(**_filter)\n\t\texcept FieldTranslation.DoesNotExist:\n\t\t\treturn False\n\n\t# Otherwise, we get all translations\n\treturn FieldTranslation.objects.filter(**_filter)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _load_translations(instance, lang=None):\n\t# Gets field translations\n\ttranslations = _get_fieldtranslations(instance=instance, lang=lang)\n\tfor translation_i in translations:\n\t\t# Sets translated field lang for this language\n\t\tfield_lang = trans_attr(translation_i.field, translation_i.lang)\n\t\tsetattr(instance, field_lang, translation_i.translation)\n\t\t# Sets is_fuzzy value for this language\n\t\tfield_isfuzzy_lang = trans_is_fuzzy_attr(translation_i.field, translation_i.lang)\n\t\t#print \"{0} ({1}), attr_name={2} value={3}\".format(translation.field, translation.lang, isfuzzy_lang,translation.is_fuzzy)\n\t\tsetattr(instance, field_isfuzzy_lang, translation_i.is_fuzzy)\n\treturn True", "response": "Loads all translations as dynamic attributes"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _set_dict_translations(instance, dict_translations):\n\n\t# If class has no translatable fields get out\n\tif not hasattr(instance._meta, \"translatable_fields\"):\n\t\treturn False\n\n\t# If we are in a site with one language there is no need of saving translations\n\tif site_is_monolingual():\n\t\treturn False\n\n\t# Translatable fields\n\ttranslatable_fields = instance._meta.translatable_fields\n\n\t# For each translatable field and for each language (excluding default language), we have to see if there is\n\t# two dynamic fields:\n\t# - _: translated atribute in .\n\t# For example: name_fr, name_es, description_it, etc.\n\t# - _is_fuzzy_: is a provisional translation of for language .\n\t# For example: name_is_fuzzy_fr, name_is_fuzzy_es, description_is_fuzzy_it, etc.\n\tfor field in translatable_fields:\n\t\tfor lang in settings.LANGUAGES:\n\t\t\tlang = lang[0]\n\t\t\tif lang != settings.LANGUAGE_CODE:\n\t\t\t\t\t\n\t\t\t\t# Translated field name\n\t\t\t\ttrans_field = trans_attr(field,lang)\n\t\t\t\t# If translated field name is in the dict, we assign it to the object\n\t\t\t\tif dict_translations.has_key(trans_field):\n\t\t\t\t\tsetattr(instance,trans_field,dict_translations[trans_field])\n\t\t\t\t\t\n\t\t\t\t# Is fuzzy attribute\n\t\t\t\ttrans_isfuzzy = trans_is_fuzzy_attr(field,lang)\n\t\t\t\t# If \"is fuzzy\" name is in the dict, we assign it to the object\n\t\t\t\tif dict_translations.has_key(trans_isfuzzy):\n\t\t\t\t\tis_fuzzy_value = (dict_translations[trans_isfuzzy]==\"1\") or (dict_translations[trans_isfuzzy]==1)\n\t\t\t\t\tsetattr(instance,trans_isfuzzy, is_fuzzy_value)", "response": "This function sets the object s translatable fields to the values in the dict."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding the translations to the object.", "response": "def add_translation(sender):\n\t\"\"\"\n\tAdds the actions to a class.\n\t\"\"\"\n\t# 1. Execute _save_translations when saving an object\n\tsignals.post_save.connect(_save_translations, sender=sender)\n\t# 2. Adds get_fieldtranslations to class. Remember that this method get all the translations.\n\tsender.add_to_class(\"get_fieldtranslations\", _get_fieldtranslations)\n\t# 3. Adss load_translations. Remember that this method included all the translations as dynamic attributes.\n\tsender.add_to_class(\"load_translations\", _load_translations)\n\t# 4. Adds _set_dict_translations. This methods allows us setting all the translated fields form a dict.\n\t# Very useful when dealing with ModelForms.\n\tsender.add_to_class(\"set_translation_fields\", _set_dict_translations)\n\t# 5. This methods returns one translated attribute in Django.\n\t# Avoid using _ and use get_trans_attr because Django maketranslations parser is fooled believing that everything\n\t# inside _ methods is translatable.\n\tsender.add_to_class(\"_\", _get_translated_field)\n\tsender.add_to_class(\"get_trans_attr\", _get_translated_field)\n\tsender.add_to_class(\"_t\", _get_translated_field)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef encrypt_file_inline(filename, passphrase):\n key = key_generators.key_from_file(filename, passphrase)\n\n inline_transform(filename, key)\n\n return key", "response": "Encrypt a file inline with an optional passphrase."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef inline_transform(filename, key):\n pos = 0\n for chunk, fp in iter_transform(filename, key):\n fp.seek(pos)\n fp.write(chunk)\n fp.flush()\n pos = fp.tell()", "response": "Encrypt a given file with the given key and replaces it directly with any extra extra\n space requirement."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngenerating encrypted file with given key. This generator function reads the file in chunks and encrypts them using AES-CTR, with the specified key. :param filename: The name of the file to encrypt. :type filename: str :param key: The key used to encrypt the file. :type key: str :returns: A generator that produces encrypted file chunks. :rtype: generator", "response": "def iter_transform(filename, key):\n \"\"\"Generate encrypted file with given key.\n\n This generator function reads the file\n in chunks and encrypts them using AES-CTR,\n with the specified key.\n\n :param filename: The name of the file to encrypt.\n :type filename: str\n :param key: The key used to encrypt the file.\n :type key: str\n :returns: A generator that produces encrypted file chunks.\n :rtype: generator\n \"\"\"\n # We are not specifying the IV here.\n aes = AES.new(key, AES.MODE_CTR, counter=Counter.new(128))\n\n with open(filename, 'rb+') as f:\n for chunk in iter(lambda: f.read(CHUNK_SIZE), b''):\n yield aes.encrypt(chunk), f"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nadds a permission to the set.", "response": "def set(self, name, permission):\n \"\"\"\n Adds permission with the given name to the set. Permission with the same name will be overridden.\n Args:\n name: name of the permission\n permission: permission instance\n \"\"\"\n assert isinstance(permission, BasePermission), 'Only permission instances can be added to the set'\n\n self._permissions[name] = permission"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nrender the field using the passed widget.", "response": "def as_widget(self, widget=None, attrs=None, only_initial=False):\n \"\"\"\n Renders the field by rendering the passed widget, adding any HTML\n attributes passed as attrs. If no widget is specified, then the\n field's default widget will be used.\n \"\"\"\n if not widget:\n widget = self.field.widget\n\n if self.field.localize:\n widget.is_localized = True\n\n attrs = attrs or {}\n attrs = self.build_widget_attrs(attrs, widget)\n auto_id = self.auto_id\n if auto_id and 'id' not in attrs and 'id' not in widget.attrs:\n if not only_initial:\n attrs['id'] = auto_id\n else:\n attrs['id'] = self.html_initial_id\n\n if not only_initial:\n name = self.html_name\n else:\n name = self.html_initial_name\n if isinstance(widget, SmartWidgetMixin) and hasattr(self.form, '_request'):\n return force_text(widget.smart_render(self.form._request, name, self.value(), self.initial,\n self.form, attrs=attrs))\n else:\n return force_text(widget.render(name, self.value(), attrs=attrs))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _setup_schema(Base, session):\n\n def create_schema_class(m):\n if hasattr(m, 'send_signalbus_message'):\n # Signal models should not use the SQLAlchemy session.\n class Meta(object):\n model = m\n else:\n class Meta(object):\n model = m\n sqla_session = session\n\n schema_class_name = '%sSchema' % m.__name__\n return type(schema_class_name, (ModelSchema,), {'Meta': Meta})\n\n def setup_schema_fn():\n for model in Base._decl_class_registry.values():\n if hasattr(model, '__tablename__'):\n if model.__name__.endswith(\"Schema\"):\n raise ModelConversionError(\n 'Unexpected model name: \"{}\". '\n 'For safety, _setup_schema() can not be used when a '\n 'model class ends with \"Schema\".'.format(model.__name__)\n )\n schema_class = getattr(model, '__marshmallow__', None)\n if schema_class is None:\n schema_class = model.__marshmallow__ = create_schema_class(model)\n if hasattr(model, 'send_signalbus_message'):\n setattr(model, '__marshmallow_schema__', schema_class())\n\n return setup_schema_fn", "response": "Create a function which creates a schema class for all signal models."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning all signal types in a list.", "response": "def get_signal_models(self):\n \"\"\"Return all signal types in a list.\n\n :rtype: list(`signal-model`)\n\n \"\"\"\n\n base = self.db.Model\n return [\n cls for cls in base._decl_class_registry.values() if (\n isinstance(cls, type)\n and issubclass(cls, base)\n and hasattr(cls, 'send_signalbus_message')\n )\n ]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nflush all pending signals over the message bus.", "response": "def flush(self, models=None, wait=3.0):\n \"\"\"Send all pending signals over the message bus.\n\n :param models: If passed, flushes only signals of the specified types.\n :type models: list(`signal-model`) or `None`\n :param float wait: The number of seconds the method will wait\n after obtaining the list of pending signals, to allow\n concurrent senders to complete\n :return: The total number of signals that have been sent\n\n \"\"\"\n\n models_to_flush = self.get_signal_models() if models is None else models\n pks_to_flush = {}\n try:\n for model in models_to_flush:\n _raise_error_if_not_signal_model(model)\n m = inspect(model)\n pk_attrs = [m.get_property_by_column(c).class_attribute for c in m.primary_key]\n pks_to_flush[model] = self.signal_session.query(*pk_attrs).all()\n self.signal_session.rollback()\n time.sleep(wait)\n return sum(\n self._flush_signals_with_retry(model, pk_values_set=set(pks_to_flush[model]))\n for model in models_to_flush\n )\n finally:\n self.signal_session.remove()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsend a potentially huge number of pending signals over the message bus.", "response": "def flushmany(self):\n \"\"\"Send a potentially huge number of pending signals over the message bus.\n\n This method assumes that the number of pending signals might\n be huge, so that they might not fit into memory. However,\n `SignalBus.flushmany` is not very smart in handling concurrent\n senders. It is mostly useful when recovering from long periods\n of disconnectedness from the message bus.\n\n :return: The total number of signals that have been sent\n\n \"\"\"\n\n models_to_flush = self.get_signal_models()\n try:\n return sum(self._flushmany_signals(model) for model in models_to_flush)\n finally:\n self.signal_session.remove()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get(self, status_code):\n status_code = int(status_code)\n if status_code >= 400:\n kwargs = {'status_code': status_code}\n if self.get_query_argument('reason', None):\n kwargs['reason'] = self.get_query_argument('reason')\n if self.get_query_argument('log_message', None):\n kwargs['log_message'] = self.get_query_argument('log_message')\n self.send_error(**kwargs)\n else:\n self.set_status(status_code)", "response": "This method returns the requested status."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_absolute_path(self, path):\n '''\n Returns the absolute path of the ``path`` argument.\n\n If ``path`` is already absolute, nothing changes. If the ``path`` is\n relative, then the BASEDIR will be prepended.\n '''\n if os.path.isabs(path):\n return path\n else:\n return os.path.abspath(os.path.join(self.config.BASEDIR, path))", "response": "Returns the absolute path of the path argument."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning all absolute paths to the roles directories while considering the BASEDIR and ROLES config variables.", "response": "def get_roles_paths(self):\n '''\n Returns all absolute paths to the roles/ directories, while considering\n the ``BASEDIR`` and ``ROLES`` config variables.\n '''\n roles = []\n\n for path in self.config.ROLES:\n roles.append(self.get_absolute_path(path))\n\n return roles"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_roles(self):\n '''\n Returns a key-value dict with a roles, while the key is the role name\n and the value is the absolute role path.\n '''\n roles = {}\n paths = self.get_roles_paths()\n\n for path in paths:\n for entry in os.listdir(path):\n rolepath = os.path.join(path, entry)\n if os.path.isdir(rolepath):\n roles[entry] = rolepath\n\n return roles", "response": "Returns a key - value dict with a roles."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreading and parses a YAML file and returns the content.", "response": "def read_yaml(self, filename):\n '''\n Reads and parses a YAML file and returns the content.\n '''\n with open(filename, 'r') as f:\n d = re.sub(r'\\{\\{ *([^ ]+) *\\}\\}', r'\\1', f.read())\n y = yaml.safe_load(d)\n return y if y else {}"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreads all YAML files inside the dir_path and returns a list of all the items in the order they were found.", "response": "def get_yaml_items(self, dir_path, param=None):\n '''\n Loops through the dir_path and parses all YAML files inside the\n directory.\n\n If no param is defined, then all YAML items will be returned\n in a list. If a param is defined, then all items will be scanned for\n this param and a list of all those values will be returned.\n '''\n\n result = []\n\n if not os.path.isdir(dir_path):\n return []\n\n for filename in os.listdir(dir_path):\n\n path = os.path.join(dir_path, filename)\n items = self.read_yaml(path)\n\n for item in items:\n if param:\n if param in item:\n item = item[param]\n if isinstance(item, list):\n result.extend(item)\n else:\n result.append(item)\n else:\n result.append(item)\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef clean_response(response):\n response = re.sub(\"^['\\\"]\", \"\", response)\n response = re.sub(\"['\\\"]$\", \"\", response)\n return response", "response": "Cleans string quoting in response."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn decorator that retries the given function if the number of retries is greater than min_wait.", "response": "def retry_on_deadlock(session, retries=6, min_wait=0.1, max_wait=10.0):\n \"\"\"Return function decorator that executes the function again in case of a deadlock.\"\"\"\n\n def decorator(action):\n \"\"\"Function decorator that retries `action` in case of a deadlock.\"\"\"\n\n @wraps(action)\n def f(*args, **kwargs):\n num_failures = 0\n while True:\n try:\n return action(*args, **kwargs)\n except (DBAPIError, DBSerializationError) as e:\n num_failures += 1\n is_serialization_error = (\n isinstance(e, DBSerializationError)\n or get_db_error_code(e.orig) in DEADLOCK_ERROR_CODES\n )\n if num_failures > retries or not is_serialization_error:\n raise\n session.rollback()\n wait_seconds = min(max_wait, min_wait * 2 ** (num_failures - 1))\n time.sleep(wait_seconds)\n\n return f\n\n return decorator"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef cached(f):\n def memoized(*args, **kwargs):\n if hasattr(memoized, '__cache__'):\n return memoized.__cache__\n result = f(*args, **kwargs)\n memoized.__cache__ = result\n return result\n return memoized", "response": "Decorator for functions evaluated only once."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef smartformset_factory(form, formset=BaseFormSet, extra=1, can_order=False,\n can_delete=False, min_num=None, max_num=None, validate_min=False, validate_max=False):\n \"\"\"Return a FormSet for the given form class.\"\"\"\n if max_num is None:\n max_num = DEFAULT_MAX_NUM\n # hard limit on forms instantiated, to prevent memory-exhaustion attacks\n # limit is simply max_num + DEFAULT_MAX_NUM (which is 2*DEFAULT_MAX_NUM\n # if max_num is None in the first place)\n absolute_max = max_num + DEFAULT_MAX_NUM\n if min_num is None:\n min_num = 0\n\n attrs = {'form': form, 'extra': extra,\n 'can_order': can_order, 'can_delete': can_delete, 'min_num': min_num,\n 'max_num': max_num, 'absolute_max': absolute_max, 'validate_min': validate_min,\n 'validate_max': validate_max}\n return type(form.__name__ + str('FormSet'), (formset,), attrs)", "response": "Return a FormSet for the given form class."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef save_form(self, form, **kwargs):\n\n obj = form.save(commit=False)\n change = obj.pk is not None\n self.save_obj(obj, form, change)\n if hasattr(form, 'save_m2m'):\n form.save_m2m()\n return obj", "response": "Contains formset save prepare obj for saving"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting the value of the method returning the specified field.", "response": "def get_method_returning_field_value(self, field_name):\n \"\"\"\n Field values can be obtained from view or core.\n \"\"\"\n return (\n super().get_method_returning_field_value(field_name)\n or self.core.get_method_returning_field_value(field_name)\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_perm_obj_or_404(self, pk=None):\n if pk:\n obj = get_object_or_none(self.core.model, pk=pk)\n else:\n try:\n obj = self.get_obj(False)\n except Http404:\n obj = get_object_or_none(self.core.model, **self.get_obj_filters())\n if not obj:\n raise Http404\n return obj", "response": "Get object according to pk or get object from get_obj method."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nviews all translations that are in site.", "response": "def view_all(request, language, filter=None):\n\t\"\"\"\n\tView all translations that are in site.\n\t\"\"\"\n\n\t# Is there any filter?\n\tif request.method == \"POST\":\n\t\tdata = request.POST.dict()\n\n\t\tif not data[\"search\"] or data[\"search\"]==\"\":\n\t\t\treturn HttpResponseRedirect(reverse(\"modeltranslation:view_all_url\",args=(data[\"language\"],data[\"filter\"])))\n\n\t\treturn HttpResponseRedirect(reverse(\"modeltranslation:view_all_url\",args=(data[\"language\"],data[\"filter\"]))+\"?search=\"+data[\"search\"])\n\n\tLANGUAGES = dict(lang for lang in settings.LANGUAGES)\n\tif language not in LANGUAGES.keys():\n\t\traise Http404(u\"Language {0} does not exist\".format(language))\n\t\n\tif language == settings.LANGUAGE_CODE:\n\t\traise Http404(u\"El idioma {0} es el idioma por defecto\".format(language))\n\n\t# Translation filter\n\ttrans_filter = {\"lang\":language}\n\tif filter == \"all\" or filter is None:\n\t\tpass\n\telif filter == \"fuzzy\":\n\t\ttrans_filter[\"is_fuzzy\"] = True\n\telif filter == \"completed\":\n\t\ttrans_filter[\"is_fuzzy\"] = False\n\t\n\tsearch_query = \"\"\n\tif request.GET and \"search\" in request.GET and request.GET.get(\"search\")!=\"\":\n\t\tsearch_query = request.GET.get(\"search\")\n\t\ttrans_filter[\"source_text__icontains\"] = search_query\n\n\ttranslations = FieldTranslation.objects.filter(**trans_filter)\n\t\n\t# Update translations\n\tactive_translations = []\n\tfor translation in translations:\n\t\tsource_model = translation.get_source_model()\n\t\tif not translation.field in source_model._meta.translatable_fields:\n\t\t\ttranslation.delete()\n\t\telse:\n\t\t\tactive_translations.append(translation)\n\t\n\treplacements = {\"translations\":active_translations, \"filter\":filter, \"lang\":language, \"language\":LANGUAGES[language], \"search_query\":search_query}\n\treturn render_to_response('modeltranslation/admin/list.html',replacements, RequestContext(request))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nediting a translation. @param request: Django HttpRequest object. @param translation: Translation id @return Django HttpResponse object with the view or a redirection.", "response": "def edit(request, translation):\n\t\"\"\"\n\tEdit a translation.\n\t@param request: Django HttpRequest object.\n\t@param translation: Translation id\n\t@return Django HttpResponse object with the view or a redirection.\n\t\"\"\"\n\ttranslation = get_object_or_404(FieldTranslation, id=translation)\n\tif request.method == 'POST':\n\t\tif \"cancel\" in request.POST:\n\t\t\treturn HttpResponseRedirect(reverse(\"modeltranslation:view_all_url\",args=(translation.lang,\"all\")))\n\t\telif \"save\" in request.POST:\n\t\t\tform = FieldTranslationForm(request.POST, instance=translation)\n\t\t\tvalid_form = form.is_valid()\n\t\t\tif valid_form:\n\t\t\t\ttranslation = form.save(commit=False)\n\t\t\t\ttranslation.context = u\"Admin. Traducciones\"\n\t\t\t\ttranslation.save()\n\t\t\t\treturn HttpResponseRedirect(reverse(\"modeltranslation:view_all_url\",args=(translation.lang,\"all\")))\n\t\telse:\n\t\t\tform = FieldTranslationForm(instance=translation)\n\telse:\n\t\tform = FieldTranslationForm(instance=translation)\n\n\tLANGUAGES = dict(lang for lang in settings.LANGUAGES)\n\tlanguage = LANGUAGES[translation.lang]\n\t\n\treturn render_to_response('modeltranslation/admin/edit_translation.html',{\"translation\":translation, \"form\":form, \"lang\":translation.lang, \"language\":language}, RequestContext(request))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef export_translations(request, language):\n\tFieldTranslation.delete_orphan_translations()\n\ttranslations = FieldTranslation.objects.filter(lang=language)\n\tfor trans in translations:\n\t\ttrans.source_text = trans.source_text.replace(\"'\",\"\\'\").replace(\"\\\"\",\"\\\\\\\"\")\n\t\ttrans.translation = trans.translation.replace(\"'\",\"\\'\").replace(\"\\\"\",\"\\\\\\\"\")\n\treplacements = {\"translations\":translations, \"lang\":language}\n\tif len(settings.ADMINS)>0:\n\t\treplacements[\"last_translator\"] = settings.ADMINS[0][0]\n\t\treplacements[\"last_translator_email\"] = settings.ADMINS[0][1]\n\tif settings.WEBSITE_NAME:\n\t\treplacements[\"website_name\"] = settings.WEBSITE_NAME\n\tresponse = render(request=request, template_name='modeltranslation/admin/export_translations.po', dictionary=replacements, context_instance=RequestContext(request), content_type=\"text/x-gettext-translation\")\n\tresponse['Content-Disposition'] = 'attachment; filename=\"{0}.po\"'.format(language)\n\treturn response", "response": "Export translations to a file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nupdate translations for new objects in database.", "response": "def update_translations(request):\n\t\"\"\"\n\tUpdate translations: delete orphan translations and creates empty translations for new objects in database.\n\t\"\"\"\n\tFieldTranslation.delete_orphan_translations()\n\tnum_translations = FieldTranslation.update_translations()\n\treturn render_to_response('modeltranslation/admin/update_translations_ok.html',{\"num_translations\":num_translations}, RequestContext(request))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef client_get(self, url, **kwargs):\n response = requests.get(self.make_url(url), headers=self.headers)\n if not response.ok:\n raise Exception(\n '{status}: {reason}.\\nCircleCI Status NOT OK'.format(\n status=response.status_code, reason=response.reason))\n return response.json()", "response": "Send GET request with given url."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsending POST request with given url and keyword args.", "response": "def client_post(self, url, **kwargs):\n \"\"\"Send POST request with given url and keyword args.\"\"\"\n response = requests.post(self.make_url(url),\n data=json.dumps(kwargs),\n headers=self.headers)\n if not response.ok:\n raise Exception(\n '{status}: {reason}.\\nCircleCI Status NOT OK'.format(\n status=response.status_code, reason=response.reason))\n return response.json()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsends POST request with given url.", "response": "def client_delete(self, url, **kwargs):\n \"\"\"Send POST request with given url.\"\"\"\n response = requests.delete(self.make_url(url), headers=self.headers)\n if not response.ok:\n raise Exception(\n '{status}: {reason}.\\nCircleCI Status NOT OK'.format(\n status=response.status_code, reason=response.reason))\n return response.json()"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef list_projects(self):\n method = 'GET'\n url = '/projects?circle-token={token}'.format(\n token=self.client.api_token)\n json_data = self.client.request(method, url)\n return json_data", "response": "Return a list of all followed projects."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntriggering a new build and return a summary of the build.", "response": "def trigger(self, username, project, branch, **build_params):\n \"\"\"Trigger new build and return a summary of the build.\"\"\"\n method = 'POST'\n url = ('/project/{username}/{project}/tree/{branch}?'\n 'circle-token={token}'.format(\n username=username, project=project,\n branch=branch, token=self.client.api_token))\n\n if build_params is not None:\n json_data = self.client.request(method, url,\n build_parameters=build_params)\n else:\n json_data = self.client.request(method, url)\n return json_data"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef cancel(self, username, project, build_num):\n method = 'POST'\n url = ('/project/{username}/{project}/{build_num}/cancel?'\n 'circle-token={token}'.format(username=username,\n project=project,\n build_num=build_num,\n token=self.client.api_token))\n json_data = self.client.request(method, url)\n return json_data", "response": "Cancel the build and return its summary."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef recent_all_projects(self, limit=30, offset=0):\n method = 'GET'\n url = ('/recent-builds?circle-token={token}&limit={limit}&'\n 'offset={offset}'.format(token=self.client.api_token,\n limit=limit,\n offset=offset))\n json_data = self.client.request(method, url)\n return json_data", "response": "Return information about recent builds across all projects."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef recent(self, username, project, limit=1, offset=0, branch=None, status_filter=\"\"):\n method = 'GET'\n if branch is not None:\n url = ('/project/{username}/{project}/tree/{branch}?'\n 'circle-token={token}&limit={limit}&offset={offset}&filter={status_filter}'.format(\n username=username, project=project, branch=branch,\n token=self.client.api_token, limit=limit,\n offset=offset, status_filter=status_filter))\n else:\n url = ('/project/{username}/{project}?'\n 'circle-token={token}&limit={limit}&offset={offset}&filter={status_filter}'.format(\n username=username, project=project,\n token=self.client.api_token, limit=limit,\n offset=offset, status_filter=status_filter))\n json_data = self.client.request(method, url)\n return json_data", "response": "Returns status of recent builds for given project and branch."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef clear(self, username, project):\n method = 'DELETE'\n url = ('/project/{username}/{project}/build-cache?'\n 'circle-token={token}'.format(username=username,\n project=project,\n token=self.client.api_token))\n json_data = self.client.request(method, url)\n return json_data", "response": "Clear the cache for given project."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nadd a new Channel.", "response": "def post(self, request, format=None):\n \"\"\"\n Add a new Channel.\n \"\"\"\n data = request.data.copy()\n\n # Get chat type record\n try:\n ct = ChatType.objects.get(pk=data.pop(\"chat_type\"))\n data[\"chat_type\"] = ct\n except ChatType.DoesNotExist:\n return typeNotFound404\n\n if not self.is_path_unique(\n None, data[\"publish_path\"], ct.publish_path\n ):\n return notUnique400\n\n # Get user record\n try:\n u = User.objects.get(pk=data.pop(\"owner\"))\n data[\"owner\"] = u\n except User.DoesNotExist:\n return userNotFound404\n\n c = Channel(**data)\n c.save()\n\n self.handle_webhook(c)\n\n return Response(\n {\n \"text\": \"Channel saved.\",\n \"method\": \"POST\",\n \"saved\": ChannelCMSSerializer(c).data,\n },\n 200,\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nupdates an existing Channel record", "response": "def patch(self, request, format=None):\n \"\"\"\n Update an existing Channel\n \"\"\"\n data = request.data.copy()\n\n # Get chat type record\n try:\n ct = ChatType.objects.get(id=data.pop(\"chat_type\"))\n data[\"chat_type\"] = ct\n except ChatType.DoesNotExist:\n return typeNotFound404\n\n if not self.is_path_unique(\n data[\"id\"], data[\"publish_path\"], ct.publish_path\n ):\n return notUnique400\n\n # Get channel record\n try:\n c = Channel.objects.get(id=data.pop(\"id\"))\n except Channel.DoesNotExist:\n return channelNotFound404\n\n # Save new data\n for key, value in data.items():\n setattr(c, key, value)\n c.save()\n\n self.handle_webhook(c)\n\n return Response(\n {\n \"text\": \"Channel saved.\",\n \"method\": \"PATCH\",\n \"saved\": ChannelCMSSerializer(c).data,\n },\n 200,\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncalculates sha256 hex digest of a file.", "response": "def sha256_file(path):\n \"\"\"Calculate sha256 hex digest of a file.\n\n :param path: The path of the file you are calculating the digest of.\n :type path: str\n :returns: The sha256 hex digest of the specified file.\n :rtype: builtin_function_or_method\n \"\"\"\n h = hashlib.sha256()\n\n with open(path, 'rb') as f:\n for chunk in iter(lambda: f.read(CHUNK_SIZE), b''):\n h.update(chunk)\n\n return h.hexdigest()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef key_from_file(filename, passphrase):\n hexdigest = sha256_file(filename)\n\n if passphrase is None:\n passphrase = DEFAULT_HMAC_PASSPHRASE\n\n return keyed_hash(hexdigest, passphrase)", "response": "Calculate a convergent encryption key for a file."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncalculate a HMAC / keyed hash.", "response": "def keyed_hash(digest, passphrase):\n \"\"\"Calculate a HMAC/keyed hash.\n\n :param digest: Digest used to create hash.\n :type digest: str\n :param passphrase: Passphrase used to generate the hash.\n :type passphrase: str\n :returns: HMAC/keyed hash.\n :rtype: str\n \"\"\"\n encodedPassphrase = passphrase.encode()\n encodedDigest = digest.encode()\n\n return hmac.new(encodedPassphrase, encodedDigest, hashlib.sha256).digest()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef find_child(sexpr: Sexpr, *tags: str) -> Optional[Sexpr]:\n _assert_valid_sexpr(sexpr)\n\n for child in sexpr[1:]:\n if _is_sexpr(child) and child[0] in tags:\n return child\n\n return None", "response": "Search for a tag among direct children of the s - expression."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef build_attrs(self, *args, **kwargs):\n \"Helper function for building an attribute dictionary.\"\n self.attrs = self.widget.build_attrs(*args, **kwargs)\n return self.attrs", "response": "Helper function for building an attribute dictionary."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_template_substitution_values(self, value):\n return {\n 'initial': os.path.basename(conditional_escape(value)),\n 'initial_url': conditional_escape(value.url),\n }", "response": "Return value - related substitutions."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef is_restricted(self):\n return (\n not hasattr(self.choices, 'queryset') or\n self.choices.queryset.count() > settings.FOREIGN_KEY_MAX_SELECBOX_ENTRIES\n )", "response": "Returns True or False according to number of objects in queryset."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef city(name=None):\n info = find_info(name)\n os.environ['OPEN311_CITY_INFO'] = dumps(info)\n return Three(**info)", "response": "Returns a Three object that can be queried against."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nuse an endpoint and any additional keyword arguments rather than one .", "response": "def dev(endpoint, **kwargs):\n \"\"\"\n Use an endpoint and any additional keyword arguments rather than one\n of the pre-defined cities. Similar to the `city` function, but useful for\n development.\n \"\"\"\n kwargs['endpoint'] = endpoint\n os.environ['OPEN311_CITY_INFO'] = dumps(kwargs)\n return Three(**kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a list of field names from an admin fieldsets structure.", "response": "def flatten_fieldsets(fieldsets):\n \"\"\"Returns a list of field names from an admin fieldsets structure.\"\"\"\n field_names = []\n for _, opts in fieldsets or ():\n if 'fieldsets' in opts:\n field_names += flatten_fieldsets(opts.get('fieldsets'))\n else:\n for field in opts.get('fields', ()):\n if isinstance(field, (list, tuple)):\n field_names.extend(field)\n else:\n field_names.append(field)\n return field_names"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of field names from an admin fieldsets structure.", "response": "def get_inline_views_from_fieldsets(fieldsets):\n \"\"\"Returns a list of field names from an admin fieldsets structure.\"\"\"\n inline_views = []\n for _, opts in fieldsets or ():\n if 'fieldsets' in opts:\n inline_views += get_inline_views_from_fieldsets(opts.get('fieldsets'))\n elif 'inline_view' in opts:\n inline_views.append(opts.get('inline_view'))\n return inline_views"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a list of field names from an admin fieldsets structure.", "response": "def get_inline_views_opts_from_fieldsets(fieldsets):\n \"\"\"Returns a list of field names from an admin fieldsets structure.\"\"\"\n inline_views = []\n for _, opts in fieldsets or ():\n if 'fieldsets' in opts:\n inline_views += get_inline_views_opts_from_fieldsets(opts.get('fieldsets'))\n elif 'inline_view' in opts:\n inline_views.append(opts)\n return inline_views"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the readonly data for the given field or method.", "response": "def get_readonly_field_data(field_name, instance, view=None, fun_kwargs=None):\n \"\"\"\n Returns field humanized value, label and widget which are used to display of instance or view readonly data.\n Args:\n field_name: name of the field which will be displayed\n instance: model instance\n view: view instance\n fun_kwargs: kwargs that can be used inside method call\n\n Returns:\n field humanized value, label and widget which are used to display readonly data\n \"\"\"\n fun_kwargs = fun_kwargs or {}\n\n if view:\n view_readonly_data = _get_view_readonly_data(field_name, view, fun_kwargs)\n if view_readonly_data is not None:\n return view_readonly_data\n\n field_data = _get_model_readonly_data(field_name, instance, fun_kwargs)\n if field_data is not None:\n return field_data\n\n raise FieldOrMethodDoesNotExist('Field or method with name {} not found'.format(field_name))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef display_object_data(obj, field_name, request=None):\n from is_core.forms.utils import ReadonlyValue\n\n value, _, _ = get_readonly_field_data(field_name, obj)\n return display_for_value(value.humanized_value if isinstance(value, ReadonlyValue) else value, request=request)", "response": "Returns humanized value of object that can be rendered to HTML or returned as part of REST\n\n "} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef display_for_value(value, request=None):\n from is_core.utils.compatibility import admin_display_for_value\n\n if request and isinstance(value, Model):\n return render_model_object_with_link(request, value)\n else:\n return (\n (value and ugettext('Yes') or ugettext('No')) if isinstance(value, bool) else admin_display_for_value(value)\n )", "response": "Displays the value for the object in the order they are assigned."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_url_from_model_core(request, obj):\n from is_core.site import get_model_core\n model_core = get_model_core(obj.__class__)\n\n if model_core and hasattr(model_core, 'ui_patterns'):\n edit_pattern = model_core.ui_patterns.get('detail')\n return (\n edit_pattern.get_url_string(request, obj=obj)\n if edit_pattern and edit_pattern.has_permission('get', request, obj=obj) else None\n )\n else:\n return None", "response": "Returns object URL from model core."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_obj_url(request, obj):\n if (is_callable(getattr(obj, 'get_absolute_url', None)) and\n (not hasattr(obj, 'can_see_edit_link') or\n (is_callable(getattr(obj, 'can_see_edit_link', None)) and obj.can_see_edit_link(request)))):\n return call_method_with_unknown_input(obj.get_absolute_url, request=request)\n else:\n return get_url_from_model_core(request, obj)", "response": "Returns object URL if current logged user has permissions to see the object"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nretrieves object and check permissions.", "response": "def protect_api(uuid=None, **kwargs):\n \"\"\"Retrieve object and check permissions.\n\n Retrieve ObjectVersion of image being requested and check permission\n using the Invenio-Files-REST permission factory.\n \"\"\"\n bucket, version_id, key = uuid.split(':', 2)\n g.obj = ObjectResource.get_object(bucket, key, version_id)\n return g.obj"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a table as a dataframe.", "response": "def table_as_df(self, table_name):\n \"\"\"Return a table as a dataframe.\"\"\"\n self.table_must_exist(table_name)\n query = \"SELECT * FROM `%s`\" % table_name.lower()\n return pandas.read_sql(query, self.own_conn)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef insert_df(self, table_name, df):\n df.to_sql(table_name, con=self.own_conn)", "response": "Create a table and populate it with data from a dataframe."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef count_rows(self, table_name):\n self.table_must_exist(table_name)\n query = \"SELECT COUNT (*) FROM `%s`\" % table_name.lower()\n self.own_cursor.execute(query)\n return int(self.own_cursor.fetchone()[0])", "response": "Return the number of rows in a table by counting them."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef tables_with_counts(self):\n table_to_count = lambda t: self.count_rows(t)\n return zip(self.tables, map(table_to_count, self.tables))", "response": "Return the number of entries in all tables."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef convert_to_sqlite(self, destination=None, method=\"shell\", progress=False):\n # Display progress bar #\n if progress: progress = tqdm.tqdm\n else: progress = lambda x:x\n # Default path #\n if destination is None: destination = self.replace_extension('sqlite')\n # Delete if it exists #\n destination.remove()\n # Method with shell and a temp file #\n if method == 'shell': return self.sqlite_by_shell(destination)\n # Method without a temp file #\n if method == 'object': return self.sqlite_by_object(destination, progress)\n # Method with dataframe #\n if method == 'dataframe': return self.sqlite_by_df(destination, progress)", "response": "Convert the database to SQLite."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef sqlite_dump_shell(self, script_path):\n # First the schema #\n shell_output('mdb-schema \"%s\" sqlite >> \"%s\"' % (self.path, script_path))\n # Start a transaction, speeds things up when importing #\n script_path.append(\"\\n\\n\\nBEGIN TRANSACTION;\\n\")\n # Then export every table #\n for table in self.tables:\n command = 'mdb-export -I sqlite \"%s\" \"%s\" >> \"%s\"'\n shell_output(command % (self.path, table, script_path))\n # End the transaction\n script_path.append(\"\\n\\n\\nEND TRANSACTION;\\n\")", "response": "Generate a text dump compatible with SQLite by using\n shell commands. Place this script at script_path *."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngenerate a text dump compatible with SQLite.", "response": "def sqlite_dump_string(self, progress):\n \"\"\"Generate a text dump compatible with SQLite.\n By yielding every table one by one as a byte string.\"\"\"\n # First the schema #\n mdb_schema = sh.Command(\"mdb-schema\")\n yield mdb_schema(self.path, \"sqlite\").encode('utf8')\n # Start a transaction, speeds things up when importing #\n yield \"BEGIN TRANSACTION;\\n\"\n # Then export every table #\n mdb_export = sh.Command(\"mdb-export\")\n for table in progress(self.tables):\n yield mdb_export('-I', 'sqlite', self.path, table).encode('utf8')\n # End the transaction\n yield \"END TRANSACTION;\\n\""} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncopy a table from another Access database to this one.", "response": "def import_table(self, source, table_name):\n \"\"\"Copy a table from another Access database to this one.\n Requires that you have mdbtools command line executables installed\n in a Windows Subsystem for Linux environment.\"\"\"\n # Run commands #\n wsl = sh.Command(\"wsl.exe\")\n table_schema = wsl(\"-e\", \"mdb-schema\", \"-T\", table_name, source.wsl_style, \"access\")\n table_contents = wsl(\"-e\", \"mdb-export\", \"-I\", \"access\", source.wsl_style, table_name)\n # Filter #\n table_schema = ' '.join(l for l in table_schema.split('\\n') if not l.startswith(\"--\"))\n # Execute statements #\n self.cursor.execute(str(table_schema))\n self.cursor.execute(str(table_contents))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a new empty MDB at destination.", "response": "def create(cls, destination):\n \"\"\"Create a new empty MDB at destination.\"\"\"\n mdb_gz_b64 = \"\"\"\\\n H4sICIenn1gC/25ldzIwMDMubWRiAO2de2wcRx3Hf7O7Pt/d3u6eLyEtVaOaqg+EkjQvuVVDwa9a\n jWXHdZxQQlCJ7fOrfp3OTpqkhVxTItFWIhVQVFBRVNIKRaColVpAUKGKRwwFqUAhKiBIpUaoVWP+\n qKgIIHL8Znb39u72znWJiWP3+9l473fzm/nNY3cdf2fmbBJEPdO9E+nebLq+fWC6vrWZOImen9D7\n 9sR+vPPNE0PZxo/TE5879mj+yNc3/OzAD2bXv3DmV9/o/8PZnxxr+/fDL2w79ulzN7e+/sS/zvzz\n w3+N1z28p3PTfQ3nfn/m2YmeFS2no89uWnvqwO5HUvd/5Phr938tes3j/zm5+qT41J8/P/iZx87/\n +qHrjgyduubG1t/+7eWB2XztTNuT+1clZt9c2/e7HRGizevWEwAAAAAAAACAhUEIwvE+PoRIO8K7\n FzT6obPPwTMBAAAAAAAAAABcfpzPXwya+Ispo1xlEO2KEEX9eaGyWnrqyKQ60tQ0AcNZRcR1RYuy\n +XZCxoqRzmaMI6cKGRJuJVrIEZUOQ9UrHStUYpyzKkdNmSPFDkM6aguhXMdVHCMuHXE2Suu4IFQJ\n l6CErNWUDouDlbdKOZIcrKLD4S5WdNhqIEodqlVaofKgVTHpiBQ6uLG0uaKsuYbf3IS8BmV1qFAm\n j1Z5Hbp06GWDKC+DTS00SRN8DFA/TXNfW6mXX3upj7+mOHWllzLAObN8du0gdSdlKO3ZcWqjMbaH\n uOQqtidViRF+P0HbOH2c3xm0lfMb1EH7uHZ5vp32c+ks+5PqfSeXS9NejjTAvZQpd7J3kuuJFqLE\n qYvuVa3Ocqk7OVXWNMFxZPRVtJ1zSXuCBrlkh+rjEF1Zlt5Dw6qN0xx5Bx3gGgbowVo56EIjkc9T\n xX9Jdd+5PKDOD6q3VQvwv7qiZ8st419cdYHlo6iuriF8X4HA590AsodXhvrsj0yMDPnAuI+ZvOrq\n 1o7K51Hdy7a8cdXNm5AedbfG5W3j3lOybxFZKb6zAgAAAAAAsNzQxAlbvnYJV3VcUU3/S2luBIKF\n ha+IlWp+wxW4IiRXRSXxKeNU1eOxUuUbSOIINbEM7WT506ZE3LASgCOeYJWCMcnCsI/u8eSsFEYR\n lnlbWa6+u0jTYqSkvuQL9G5CLFwTRBMAAAAAAAAAgMtW/79lyVdLKxW7oqDF3bXOniib0UD/m/xq\n loWqvFwt3DX/mrLNALIu3V35NkpK1JDmL+2XOmr9pf1gKiFY4I672wc0mveaf6zaenyKmljPT6t5\n hT7a6y13y0XqjFpwneJjRC0oRwvL3eUL2fHCcuyGIntjhTkDuZCd5Vc5j+HNUMyx+myYcpHW5YG5\n ZijUdbg2VFu4ZzzcHFM3seQLAAAAAAAAAMtc//9S6cm1emX97ytK1v81rHelhtfVfAFnseZXRdV9\n Ad7+dhGS5kbl3eqe/K8pU/nnYwX5X2VeoLbCZwHi7txD6aTELabnoLJ5AfPFC8JmFd3Pun+MlfM4\n q/846/4s62i5+8Dmc7EvSVN0UG2tL00p1uPXqZTt/G5QqX+5lbufz+mSctVzFce6upBrTG3Fd+cn\n pmiYrUyw8+GNfL4hn8/k83qZrVlyGzgPeqbhjcOqx7KMEZRpU/MPQ+rsldEtuYm8vExkznoMS+6b\n KC5TZRt8wVf4xEkFX4V5D/X2vYz1/EcR8yMAAAAAAACAJY0Qf/d3vLPUlb//b4Nzzv6W3Wevtl+1\n vmxts2LWTxOHErcm3jGfMUfNG0yMGQAAAAAAeJ/8rLwAMXIYRgCARFv8IIaYtKpGqCdqlN/2kupD\n /ob67qXhsi0lDh2Vp6728faO9tHuUflfWJ1wE0e6724f35XuG71r16Dr0FwH573by6rKi0N7RveN\n tnd6aTVBWrpjd3fnuJtsBMnDk90ju7zckSA5XGGtdGrK2dWhUnRcMgAAAAAAAAD4v2CIV6vqf82I\n Jusbcwsy7wkWSf/n1JQNq/Oc+uQGq/ecmsphYZ6Tn6XwRLjwxb7mTxDoakLgURUFshwAAAAAAAAA\n ljpCrHZ8W/f2/2NUAAAAAAAAAAAAhXH5RLm4IIbotqot7hbW/0MGWCp46/+pgpHwjZS3IyAlfMPy\n tgakNN+wfcPxNgukdN9I+kadt30gZfhGjW+s8I2V3s6CVNTbWZCK+Eatb3zAN1Z5mw5SMd+I+wZ+\n +QQAAAAAAAAA/K8IcdT27Zqi3/+HkQEAAAAAAAAAsGgkMQQLjSHqbQPDAAAAAAAAAAAALGuw/g8A\n AAAAAAAA4DJUqwsQI7cQDWlcLiMq1/9rcGMBAAAAAAAAAADLGuh/AAAAAAAAAAAA+h8AAAAAAAAA\n AABLHyHusDTPjtLzTtoxnRftUftqe8YatDA+AAAAAAAAAPDeqJN/KVt+et0R9PYnzz7W8PrZRv+V\n HblO6qEDNEXbaYDGqJemaYQmaYJThtnK8Gvzb1opfDRTPZmUlxUY86qgm/ZyFVkOOqCC3kLhoyEI\n qs8raBO10O0q3EYKH+uDcNq8wnVRH93D7evnYZhHG5kkB3a0OYO2ctCWV9ZR+FhT0l2HCzl6xVBz\n XZyPUvi4taTjcwRuVUF7uYW9HMy9MJspfGwMAoo5A+5Qwca8UHN2WogeU/fu0ito1vmjM+M85zzp\n fNG5zxl2djrNzk3O9+0m+yWrx2q0fpH4buJ4Yk3ig4lvmkfxx9gBAAAAAAC4OAylQfJ5h5pfSVCc\n f853gqSmWPSZux6xjUznltH2HT/flNu7++0NZ7/07cg/vnPbVu30y6d/NLvlabPh+j81v/Xc5g9l\n 1h2f+epn9+VPdN90OHHvU50fm94y/ZXvWQ/tP/yJG/NH3llz8A79tlNPG72DHSePHdzz2s3XPzVj\n vzSUvSHjVys1Rv5CSUv8pEvcEqkbV/KX35JaQ+npikmRS9o4rtYIt8RYnJa4Ou6SV6stTm+l7rcX\n q9qSy+23pCVIcgV/SZKuJj5CSRc4Y/PpkiesLJcI53J37NvFuQzv4peGL0/SypP+C+45xVAAMAEA\n \"\"\"\n pristine = StringIO()\n pristine.write(base64.b64decode(mdb_gz_b64))\n pristine.seek(0)\n pristine = gzip.GzipFile(fileobj=pristine, mode='rb')\n with open(destination, 'wb') as handle: shutil.copyfileobj(pristine, handle)\n return cls(destination)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nmake a new connection.", "response": "def new_connection(self):\n \"\"\"Make a new connection.\"\"\"\n if not self.prepared: self.prepare()\n con = sqlite3.connect(self.path, isolation_level=self.isolation)\n con.row_factory = self.factory\n if self.text_fact: con.text_factory = self.text_fact\n return con"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef prepare(self):\n if not os.path.exists(self.path):\n if self.retrieve:\n print(\"Downloading SQLite3 database...\")\n download_from_url(self.retrieve, self.path, progress=True)\n else: raise Exception(\"The file '\" + self.path + \"' does not exist.\")\n self.check_format()\n if self.known_md5: assert self.known_md5 == self.md5\n self.prepared = True", "response": "Check that the file exists optionally downloads it."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a new database with a certain schema.", "response": "def create(self, columns=None, type_map=None, overwrite=False):\n \"\"\"Create a new database with a certain schema.\"\"\"\n # Check already exists #\n if self.count_bytes > 0:\n if overwrite: self.remove()\n else: raise Exception(\"File exists already at '%s'\" % self)\n # If we want it empty #\n if columns is None:\n self.touch()\n # Make the table #\n else:\n self.add_table(self.main_table, columns=columns, type_map=type_map)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd a new table to the database.", "response": "def add_table(self, name, columns, type_map=None, if_not_exists=False):\n \"\"\"Add add a new table to the database. For instance you could do this:\n self.add_table('data', {'id':'integer', 'source':'text', 'pubmed':'integer'})\"\"\"\n # Check types mapping #\n if type_map is None and isinstance(columns, dict): types = columns\n if type_map is None: types = {}\n # Safe or unsafe #\n if if_not_exists: query = 'CREATE TABLE IF NOT EXISTS \"%s\" (%s);'\n else: query = 'CREATE table \"%s\" (%s);'\n # Do it #\n cols = ','.join(['\"' + c + '\"' + ' ' + types.get(c, 'text') for c in columns])\n self.own_cursor.execute(query % (self.main_table, cols))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_columns_of_table(self, table=None):\n # Check the table exists #\n if table is None: table = self.main_table\n if not table in self.tables: return []\n # A PRAGMA statement will implicitly issue a commit, don't use #\n self.own_cursor.execute('SELECT * from \"%s\" LIMIT 1;' % table)\n columns = [x[0] for x in self.own_cursor.description]\n self.cursor.fetchall()\n return columns", "response": "Return the list of columns for a particular table by querying the SQL for the complete list of column names."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef add(self, entries, table=None, columns=None, ignore=False):\n # Default table and columns #\n if table is None: table = self.main_table\n if columns is None: columns = self.get_columns_of_table(table)\n # Default columns #\n question_marks = ','.join('?' for c in columns)\n cols = ','.join('\"' + c + '\"' for c in columns)\n ignore = \" OR IGNORE\" if ignore else \"\"\n sql_command = 'INSERT%s into \"%s\"(%s) VALUES (%s);'\n sql_command = sql_command % (ignore, table, cols, question_marks)\n # Possible errors we want to catch #\n errors = (sqlite3.OperationalError, sqlite3.ProgrammingError)\n errors += (sqlite3.IntegrityError, sqlite3.InterfaceError, ValueError)\n # Do it #\n try:\n new_cursor = self.own_connection.cursor()\n new_cursor.executemany(sql_command, entries)\n except errors as err:\n raise Exception(self.detailed_error(sql_command, columns, entries, err))\n except KeyboardInterrupt as err:\n print(\"You interrupted the data insertion.\")\n print(\"Committing everything done up to this point.\")\n self.own_connection.commit()\n raise err", "response": "Add entries to a table."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_by_steps(self, entries_by_step, table=None, columns=None):\n for entries in entries_by_step: self.add(entries, table=table, columns=columns)", "response": "Add entries to the main table."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_first(self, table=None):\n if table is None: table = self.main_table\n query = 'SELECT * FROM \"%s\" LIMIT 1;' % table\n return self.own_cursor.execute(query).fetchone()", "response": "Just the first entry."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets a specific entry by its number.", "response": "def get_number(self, num, table=None):\n \"\"\"Get a specific entry by its number.\"\"\"\n if table is None: table = self.main_table\n self.own_cursor.execute('SELECT * from \"%s\" LIMIT 1 OFFSET %i;' % (self.main_table, num))\n return self.own_cursor.fetchone()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_entry(self, key, column=None, table=None):\n if table is None: table = self.main_table\n if column is None: column = \"id\"\n if isinstance(key, basestring): key = key.replace(\"'\",\"''\")\n query = 'SELECT * from \"%s\" where \"%s\"==\"%s\" LIMIT 1;'\n query = query % (table, column, key)\n self.own_cursor.execute(query)\n return self.own_cursor.fetchone()", "response": "Get a specific entry."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef insert_df(self, table_name, df):\n df.to_sql(table_name, con=self.own_connection)", "response": "Create a table and populate it with data from a dataframe."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget specific entries and order them in the same way.", "response": "def get_and_order(self, ids, column=None, table=None):\n \"\"\"Get specific entries and order them in the same way.\"\"\"\n command = \"\"\"\n SELECT rowid, * from \"data\"\n WHERE rowid in (%s)\n ORDER BY CASE rowid\n %s\n END;\n \"\"\"\n ordered = ','.join(map(str,ids))\n rowids = '\\n'.join(\"WHEN '%s' THEN %s\" % (row,i) for i,row in enumerate(ids))\n command = command % (ordered, rowids)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef import_table(self, source, table_name):\n query = \"SELECT * FROM `%s`\" % table_name.lower()\n df = pandas.read_sql(query, source.connection)\n df.to_sql(table_name, con=self.own_connection)", "response": "Copy a table from another SQLite database to this one."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef jsonify(self, status_code=None, message=None, headers=None):\n if status_code is None:\n status_code = self.status_code\n if message is None:\n message = self.message\n if headers is None:\n headers = self.headers\n response = flask.jsonify({\n 'status_code': status_code,\n 'error': repr(self),\n 'message': message,\n })\n if status_code is not None:\n response.status_code = status_code\n if headers is not None:\n response.headers = headers\n return response", "response": "Returns a jsonified version of the exception."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef build_error_handler(*tasks):\n\n def _handler(error, tasks=[]):\n [t(error) for t in tasks]\n return error.jsonify(), error.status_code, error.headers\n\n return functools.partial(_handler, tasks=tasks)", "response": "Builds a generic error handler that packages a flask_buzz exception\n so that it can be handled nicely by flask s error handler."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef build_error_handler_for_flask_restplus(*tasks):\n def _handler(error, tasks=[]):\n [t(error) for t in tasks]\n response = error.jsonify()\n return flask.json.loads(response.get_data()), response.status_code\n\n return functools.partial(_handler, tasks=tasks)", "response": "Build a generic error handler that packages a flask_buzz exception\n so that it can be handled by the Flask - restplus error handler."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef register_error_handler_with_flask_restplus(cls, api, *tasks):\n\n for buzz_subclass in [cls] + cls.__subclasses__():\n api.errorhandler(buzz_subclass)(\n cls.build_error_handler_for_flask_restplus(*tasks)\n )", "response": "Registers an error handler for FlaskBuzz derived errors that are not currently imported."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a dict containing the humanized data in the given instance.", "response": "def humanized_model_to_dict(instance, readonly_fields, fields=None, exclude=None):\n \"\"\"\n Returns a dict containing the humanized data in ``instance`` suitable for passing as\n a Form's ``initial`` keyword argument.\n\n ``fields`` is an optional list of field names. If provided, only the named\n fields will be included in the returned dict.\n\n ``exclude`` is an optional list of field names. If provided, the named\n fields will be excluded from the returned dict, even if they are listed in\n the ``fields`` argument.\n \"\"\"\n opts = instance._meta\n data = {}\n for f in itertools.chain(opts.concrete_fields, opts.private_fields, opts.many_to_many):\n if not getattr(f, 'editable', False):\n continue\n if fields and f.name not in fields:\n continue\n if f.name not in readonly_fields:\n continue\n if exclude and f.name in exclude:\n continue\n\n if f.humanized:\n data[f.name] = f.humanized(getattr(instance, f.name), instance)\n return data"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfinds the needed city server information.", "response": "def find_info(name=None):\n \"\"\"Find the needed city server information.\"\"\"\n if not name:\n return list(servers.keys()) \n name = name.lower()\n if name in servers:\n info = servers[name]\n else:\n raise CityNotFound(\"Could not find the specified city: %s\" % name)\n return info"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns path of bin which returns None if no executable found", "response": "def which(\n exe=None, default_paths=['/bin', '/sbin', '/usr/bin', '/usr/sbin', '/usr/local/bin']\n):\n \"\"\"Return path of bin. Python clone of /usr/bin/which.\n\n from salt.util - https://www.github.com/saltstack/salt - license apache\n\n :param exe: Application to search PATHs for.\n :type exe: str\n :param default_path: Application to search PATHs for.\n :type default_path: list\n :rtype: str\n \"\"\"\n\n def _is_executable_file_or_link(exe):\n # check for os.X_OK doesn't suffice because directory may executable\n return os.access(exe, os.X_OK) and (os.path.isfile(exe) or os.path.islink(exe))\n\n if _is_executable_file_or_link(exe):\n # executable in cwd or fullpath\n return exe\n\n # Enhance POSIX path for the reliability at some environments, when\n # $PATH is changing. This also keeps order, where 'first came, first\n # win' for cases to find optional alternatives\n search_path = (\n os.environ.get('PATH') and os.environ['PATH'].split(os.pathsep) or list()\n )\n for default_path in default_paths:\n if default_path not in search_path:\n search_path.append(default_path)\n os.environ['PATH'] = os.pathsep.join(search_path)\n for path in search_path:\n full_path = os.path.join(path, exe)\n if _is_executable_file_or_link(full_path):\n return full_path\n logger.info(\n '\\'{0}\\' could not be found in the following search path: '\n '\\'{1}\\''.format(exe, search_path)\n )\n\n return None"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef run(\n cmd,\n shell=False,\n cwd=None,\n log_in_real_time=True,\n check_returncode=True,\n callback=None,\n):\n \"\"\" Run 'cmd' in a shell and return the combined contents of stdout and\n stderr (Blocking). Throws an exception if the command exits non-zero.\n\n :param cmd: list of str (or single str, if shell==True) indicating\n the command to run\n :param shell: boolean indicating whether we are using advanced shell\n features. Use only when absolutely necessary, since this allows a lot\n more freedom which could be exploited by malicious code. See the\n warning here:\n http://docs.python.org/library/subprocess.html#popen-constructor\n :param cwd: dir command is run from.\n :type cwd: str\n :param log_in_real_time: boolean indicating whether to read stdout from the\n subprocess in real time instead of when the process finishes.\n :param check_returncode: Indicate whether a :exc:`~exc.CommandError`\n should be raised if return code is different from 0.\n :type check_returncode: :class:`bool`\n :param cwd: dir command is run from, defaults :attr:`~.path`.\n :type cwd: str\n :param callback: callback to return output as a command executes, accepts\n a function signature of ``(output, timestamp)``. Example usage::\n\n def progress_cb(output, timestamp):\n sys.stdout.write(output)\n sys.stdout.flush()\n run(['git', 'pull'], callback=progrses_cb)\n :type callback: func\n \"\"\"\n proc = subprocess.Popen(\n cmd,\n shell=shell,\n stderr=subprocess.PIPE,\n stdout=subprocess.PIPE,\n creationflags=0,\n bufsize=1,\n cwd=cwd,\n )\n\n all_output = []\n code = None\n line = None\n while code is None:\n code = proc.poll()\n\n # output = console_to_str(proc.stdout.readline())\n # all_output.append(output)\n if callback and callable(callback):\n line = console_to_str(proc.stderr.read(128))\n if line:\n callback(output=line, timestamp=datetime.datetime.now())\n if callback and callable(callback):\n callback(output='\\r', timestamp=datetime.datetime.now())\n\n lines = filter(None, (line.strip() for line in proc.stdout.readlines()))\n all_output = console_to_str(b'\\n'.join(lines))\n if code:\n stderr_lines = filter(None, (line.strip() for line in proc.stderr.readlines()))\n all_output = console_to_str(b''.join(stderr_lines))\n output = ''.join(all_output)\n if code != 0 and check_returncode:\n raise exc.CommandError(output=output, returncode=code, cmd=cmd)\n return output", "response": "Runs a command in a shell and returns the combined contents of stdout and stderr."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nadd additional context information for loggers.", "response": "def process(self, msg, kwargs):\n \"\"\"Add additional context information for loggers.\"\"\"\n prefixed_dict = {}\n prefixed_dict['repo_vcs'] = self.bin_name\n prefixed_dict['repo_name'] = self.name\n\n kwargs[\"extra\"] = prefixed_dict\n\n return msg, kwargs"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef pop(self, sexp):\n '''\n Notes: Sequence works a bit different than other nodes.\n This method (like others) expectes a list. However, sequence matches\n against the list, whereas other nodes try to match against elements\n of the list.\n '''\n for t in self.terms:\n sexp = t.pop(sexp)\n return sexp", "response": "Removes the given sexp from the list of terms."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef re_clone(self, repo_dir):\n self.git('clone', self.remote_url, repo_dir)\n return GitRepo(repo_dir)", "response": "Clone again, somewhere else"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef path(self):\n yield self\n if not self.parent: return\n for node in self.parent.path: yield node", "response": "Iterate over all parent nodes and one - self."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef others(self):\n if not self.parent: return\n yield self.parent\n for sibling in self.parent.children.values():\n if sibling.name == self.name: continue\n for node in sibling: yield node\n for node in self.parent.others: yield node", "response": "Iterate over all nodes of the tree excluding one self and one s children."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_children(self, depth=1):\n yield self\n if depth == 0: return\n for child in self.children.values():\n for node in child.get_children(depth-1): yield node", "response": "Iterate over all children until a certain level and one - self."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget all nodes that are exactly this far away.", "response": "def get_level(self, level=2):\n \"\"\"Get all nodes that are exactly this far away.\"\"\"\n if level == 1:\n for child in self.children.values(): yield child\n else:\n for child in self.children.values():\n for node in child.get_level(level-1): yield node"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncuts all branches over a certain length making new leaves at a certain length.", "response": "def trim(self, length):\n \"\"\"Cut all branches over a certain length making new leaves at *length*.\"\"\"\n if length > 0:\n for child in self.children.values(): child.trim(length-1)\n else:\n if hasattr(self, 'count'): self.count = sum(map(lambda x: x.count, self))\n self.children = OrderedDict()"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncuts all branches from this node to its children and adopt all nodes at certain level.", "response": "def mend(self, length):\n \"\"\"Cut all branches from this node to its children and adopt\n all nodes at certain level.\"\"\"\n if length == 0: raise Exception(\"Can't mend the root !\")\n if length == 1: return\n self.children = OrderedDict((node.name, node) for node in self.get_level(length))\n for child in self.children.values(): child.parent = self"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets data from distroinfo instance.", "response": "def get_info(self, apply_tag=None, info_dicts=False):\n \"\"\"\n Get data from distroinfo instance.\n\n :param apply_tag: apply supplied tag to info\n :param info_dicts: return packages and releases as dicts\n :return: parsed info metadata\n \"\"\"\n raw_infos = self.fetcher.fetch(*self.info_files)\n raw_info = parse.merge_infos(*raw_infos, info_dicts=info_dicts)\n info = parse.parse_info(raw_info, apply_tag=apply_tag)\n return info"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef run(create_application, settings=None, log_config=None):\n from . import runner\n\n app_settings = {} if settings is None else settings.copy()\n debug_mode = bool(app_settings.get('debug',\n int(os.environ.get('DEBUG', 0)) != 0))\n app_settings['debug'] = debug_mode\n logging.config.dictConfig(_get_logging_config(debug_mode)\n if log_config is None else log_config)\n\n port_number = int(app_settings.pop('port', os.environ.get('PORT', 8000)))\n num_procs = int(app_settings.pop('number_of_procs', '0'))\n server = runner.Runner(create_application(**app_settings))\n\n server.run(port_number, num_procs)", "response": "Runs a Tornado create_application."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_url_and_revision_from_pip_url(cls, pip_url):\n if '://' not in pip_url:\n assert 'file:' not in pip_url\n pip_url = pip_url.replace('git+', 'git+ssh://')\n url, rev = super(GitRepo, cls).get_url_and_revision_from_pip_url(pip_url)\n url = url.replace('ssh://', '')\n elif 'github.com:' in pip_url:\n raise exc.LibVCSException(\n \"Repo %s is malformatted, please use the convention %s for\"\n \"ssh / private GitHub repositories.\"\n % (pip_url, \"git+https://github.com/username/repo.git\")\n )\n else:\n url, rev = super(GitRepo, cls).get_url_and_revision_from_pip_url(pip_url)\n\n return url, rev", "response": "This method is used to get the URL and revision from a pip URL."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nretrieves the repository clone if doesn t exist.", "response": "def obtain(self):\n \"\"\"Retrieve the repository, clone if doesn't exist.\"\"\"\n self.check_destination()\n\n url = self.url\n\n cmd = ['clone', '--progress']\n if self.git_shallow:\n cmd.extend(['--depth', '1'])\n if self.tls_verify:\n cmd.extend(['-c', 'http.sslVerify=false'])\n cmd.extend([url, self.path])\n\n self.info('Cloning.')\n self.run(cmd, log_in_real_time=True)\n\n if self.remotes:\n for r in self.remotes:\n self.error('Adding remote %s <%s>' % (r['remote_name'], r['url']))\n self.remote_set(name=r['remote_name'], url=r['url'])\n\n self.info('Initializing submodules.')\n self.run(['submodule', 'init'], log_in_real_time=True)\n cmd = ['submodule', 'update', '--recursive', '--init']\n cmd.extend(self.git_submodules)\n self.run(cmd, log_in_real_time=True)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef remotes_get(self):\n remotes = {}\n\n cmd = self.run(['remote'])\n ret = filter(None, cmd.split('\\n'))\n\n for remote_name in ret:\n remotes[remote_name] = self.remote_get(remote_name)\n return remotes", "response": "Return remotes like git remote - v."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget the fetch and push URL for a specified remote name.", "response": "def remote_get(self, remote='origin'):\n \"\"\"Get the fetch and push URL for a specified remote name.\n\n :param remote: the remote name used to define the fetch and push URL\n :type remote: str\n :returns: remote name and url in tuple form\n :rtype: tuple\n \"\"\"\n try:\n ret = self.run(['remote', 'show', '-n', remote])\n lines = ret.split('\\n')\n remote_fetch_url = lines[1].replace('Fetch URL: ', '').strip()\n remote_push_url = lines[2].replace('Push URL: ', '').strip()\n if remote_fetch_url != remote and remote_push_url != remote:\n res = (remote_fetch_url, remote_push_url)\n return res\n else:\n return None\n except exc.LibVCSException:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef remote_set(self, url, name='origin'):\n\n url = self.chomp_protocol(url)\n\n if self.remote_get(name):\n self.run(['remote', 'rm', 'name'])\n\n self.run(['remote', 'add', name, url])\n return self.remote_get(remote=name)", "response": "Set remote with name and URL like git remote add."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn url with VCS protocol removed", "response": "def chomp_protocol(url):\n \"\"\"Return clean VCS url from RFC-style url\n\n :param url: url\n :type url: str\n :rtype: str\n :returns: url as VCS software would accept it\n :seealso: #14\n \"\"\"\n if '+' in url:\n url = url.split('+', 1)[1]\n scheme, netloc, path, query, frag = urlparse.urlsplit(url)\n rev = None\n if '@' in path:\n path, rev = path.rsplit('@', 1)\n url = urlparse.urlunsplit((scheme, netloc, path, query, ''))\n if url.startswith('ssh://git@github.com/'):\n url = url.replace('ssh://', 'git+ssh://')\n elif '://' not in url:\n assert 'file:' not in url\n url = url.replace('git+', 'git+ssh://')\n url = url.replace('ssh://', '')\n return url"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nlog the message to the sys. stdout or sys. stderr stream.", "response": "def _log(self, message, stream, color=None, newline=False):\n '''\n Logs the message to the sys.stdout or sys.stderr stream.\n\n When color is defined and the TERM environemnt variable contains the\n string \"color\", then the output will be colored.\n '''\n\n if color and self.color_term:\n colorend = Logger.COLOR_END\n else:\n color = colorend = ''\n\n stream.write('{color}{message}{colorend}\\n'.format(\n color=color,\n message=message,\n colorend=colorend\n ))\n\n if newline:\n sys.stdout.write('\\n')\n\n stream.flush()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nlog an informational message to stdout.", "response": "def info(self, message):\n '''\n Logs an informational message to stdout.\n\n This method should only be used by the Runner.\n '''\n return self._log(\n message=message.upper(),\n stream=sys.stdout\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef passed(self, message):\n '''\n Logs as whole test result as PASSED.\n\n This method should only be used by the Runner.\n '''\n return self._log(\n message=message.upper(),\n stream=sys.stdout,\n color=Logger.COLOR_GREEN_BOLD,\n newline=True\n )", "response": "Logs as whole test result as PASSED."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nlogging as whole test result as FAILED.", "response": "def failed(self, message):\n '''\n Logs as whole test result as FAILED.\n\n This method should only be used by the Runner.\n '''\n return self._log(\n message=message.upper(),\n stream=sys.stderr,\n color=Logger.COLOR_RED_BOLD,\n newline=True\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef non_blocking(func):\n from functools import wraps\n @wraps(func)\n def non_blocking_version(*args, **kwargs):\n t = ReturnThread(target=func, args=args, kwargs=kwargs)\n t.start()\n return t\n return non_blocking_version", "response": "Decorator to run a function in a non - blocking way."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef media_url(self, with_ssl=False):\n if self.serve_remote:\n # Hand this off to whichever backend is being used.\n url = self.remote_media_url(with_ssl)\n else:\n # Serving locally, just use the value in settings.py.\n url = self.local_media_url\n return url.rstrip('/')", "response": "Returns a base media URL based on the current settings. py settings."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfinds the log directory and return all the logs sorted.", "response": "def logs(self):\n \"\"\"Find the log directory and return all the logs sorted.\"\"\"\n if not self.parent.loaded: self.parent.load()\n logs = self.parent.p.logs_dir.flat_directories\n logs.sort(key=lambda x: x.mod_time)\n return logs"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef run_locally(self, steps=None, **kwargs):\n self.slurm_job = LoggedJobSLURM(self.command(steps),\n base_dir = self.parent.p.logs_dir,\n modules = self.modules,\n **kwargs)\n self.slurm_job.run_locally()", "response": "A convenience method to run the same result as a SLURM job\n but locally in a non - blocking way."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nrun the steps via the SLURM queue.", "response": "def run_slurm(self, steps=None, **kwargs):\n \"\"\"Run the steps via the SLURM queue.\"\"\"\n # Optional extra SLURM parameters #\n params = self.extra_slurm_params\n params.update(kwargs)\n # Mandatory extra SLURM parameters #\n if 'time' not in params: params['time'] = self.default_time\n if 'job_name' not in params: params['job_name'] = self.job_name\n if 'email' not in params: params['email'] = None\n if 'dependency' not in params: params['dependency'] = 'singleton'\n # Send it #\n self.slurm_job = LoggedJobSLURM(self.command(steps),\n base_dir = self.parent.p.logs_dir,\n modules = self.modules,\n **params)\n # Return the Job ID #\n return self.slurm_job.run()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a ``GenericInlineFormSet`` for the given kwargs. You must provide ``ct_field`` and ``fk_field`` if they are different from the defaults ``content_type`` and ``object_id`` respectively.", "response": "def smart_generic_inlineformset_factory(model, request, form=ModelForm, formset=BaseGenericInlineFormSet,\n ct_field='content_type', fk_field='object_id', fields=None, exclude=None,\n extra=3, can_order=False, can_delete=True, min_num=None, max_num=None,\n formfield_callback=None, widgets=None, validate_min=False, validate_max=False,\n localized_fields=None, labels=None, help_texts=None, error_messages=None,\n formreadonlyfield_callback=None, readonly_fields=None, for_concrete_model=True,\n readonly=False):\n \"\"\"\n Returns a ``GenericInlineFormSet`` for the given kwargs.\n\n You must provide ``ct_field`` and ``fk_field`` if they are different from\n the defaults ``content_type`` and ``object_id`` respectively.\n \"\"\"\n opts = model._meta\n # if there is no field called `ct_field` let the exception propagate\n ct_field = opts.get_field(ct_field)\n if not isinstance(ct_field, models.ForeignKey) or ct_field.related_model != ContentType:\n raise Exception(\"fk_name '%s' is not a ForeignKey to ContentType\" % ct_field)\n fk_field = opts.get_field(fk_field) # let the exception propagate\n if exclude is not None:\n exclude = list(exclude)\n exclude.extend([ct_field.name, fk_field.name])\n else:\n exclude = [ct_field.name, fk_field.name]\n\n kwargs = {\n 'form': form,\n 'formfield_callback': formfield_callback,\n 'formset': formset,\n 'extra': extra,\n 'can_delete': can_delete,\n 'can_order': can_order,\n 'fields': fields,\n 'exclude': exclude,\n 'max_num': max_num,\n 'min_num': min_num,\n 'widgets': widgets,\n 'validate_min': validate_min,\n 'validate_max': validate_max,\n 'localized_fields': localized_fields,\n 'formreadonlyfield_callback': formreadonlyfield_callback,\n 'readonly_fields': readonly_fields,\n 'readonly': readonly,\n 'labels': labels,\n 'help_texts': help_texts,\n 'error_messages': error_messages,\n }\n\n FormSet = smartmodelformset_factory(model, request, **kwargs)\n FormSet.ct_field = ct_field\n FormSet.ct_fk_field = fk_field\n FormSet.for_concrete_model = for_concrete_model\n return FormSet"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset the name of the machine.", "response": "def rename(self, name):\n \"\"\"Set the name of the machine.\"\"\"\n self.ec2.create_tags(Resources = [self.instance_id],\n Tags = [{'Key': 'Name',\n 'Value': name}])\n self.refresh_info()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef update_ssh_config(self, path=\"~/.ssh/config\"):\n # Read the config file #\n import sshconf\n config = sshconf.read_ssh_config(os.path.expanduser(path))\n # In case it doesn't exist #\n if not config.host(self.instance_name): config.add(self.instance_name)\n # Add the new DNS #\n config.set(self.instance_name, Hostname=self.dns)\n # Write result #\n config.write(os.path.expanduser(path))", "response": "Update the SSH config file with the DNS."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nupdates widget. choices with special choice iterator that removes blank values and adds none values to clear filter data.", "response": "def _update_widget_choices(self, widget):\n \"\"\"\n Updates widget choices with special choice iterator that removes blank values and adds none value to clear\n filter data.\n :param widget: widget with choices\n :return: updated widget with filter choices\n \"\"\"\n\n widget.choices = FilterChoiceIterator(widget.choices, self.field)\n return widget"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_widget(self, request):\n return self._update_widget_choices(self.field.formfield(widget=RestrictedSelectWidget).widget)", "response": "Returns the widget to use for this field."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_widget(self, request):\n return self._update_widget_choices(\n forms.ModelChoiceField(\n widget=RestrictedSelectWidget, queryset=self.field.related_model._default_manager.all()\n ).widget\n )", "response": "Returns the widget for this field based on the current request."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nstrips proto:// | openstack/ prefixes and .git | -distgit suffixes", "response": "def strip_project_url(url):\n \"\"\"strip proto:// | openstack/ prefixes and .git | -distgit suffixes\"\"\"\n m = re.match(r'(?:[^:]+://)?(.*)', url)\n if m:\n url = m.group(1)\n if url.endswith('.git'):\n url, _, _ = url.rpartition('.')\n if url.endswith('-distgit'):\n url, _, _ = url.rpartition('-')\n if url.startswith('openstack/'):\n # openstack is always special :-p\n _, _, url = url.partition('/')\n return url"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn revision options. from pip pip. vcs. subversion.", "response": "def get_rev_options(url, rev):\n \"\"\"Return revision options.\n\n from pip pip.vcs.subversion.\n\n \"\"\"\n if rev:\n rev_options = ['-r', rev]\n else:\n rev_options = []\n\n r = urlparse.urlsplit(url)\n if hasattr(r, 'username'):\n # >= Python-2.5\n username, password = r.username, r.password\n else:\n netloc = r[1]\n if '@' in netloc:\n auth = netloc.split('@')[0]\n if ':' in auth:\n username, password = auth.split(':', 1)\n else:\n username, password = auth, None\n else:\n username, password = None, None\n\n if username:\n rev_options += ['--username', username]\n if password:\n rev_options += ['--password', password]\n return rev_options"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn revision for a file.", "response": "def get_revision_file(self, location):\n \"\"\"Return revision for a file.\"\"\"\n\n current_rev = self.run(['info', location])\n\n _INI_RE = re.compile(r\"^([^:]+):\\s+(\\S.*)$\", re.M)\n\n info_list = _INI_RE.findall(current_rev)\n return int(dict(info_list)['Revision'])"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the publish path for the given object.", "response": "def get_publish_path(self, obj):\n \"\"\"\n publish_path joins the publish_paths for the chat type and the channel.\n \"\"\"\n return os.path.join(\n obj.chat_type.publish_path, obj.publish_path.lstrip(\"/\")\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the name of the translated attribute of the object _ .", "response": "def trans_attr(attr, lang):\n\t\"\"\"\n\tReturns the name of the translated attribute of the object _.\n\tFor example: name_es (name attribute in Spanish)\n\t@param attr Attribute whose name will form the name translated attribute.\n\t@param lang ISO Language code that will be the suffix of the translated attribute.\n\t@return: string with the name of the translated attribute.\n\t\"\"\"\n\tlang = lang.replace(\"-\",\"_\").lower()\n\treturn \"{0}_{1}\".format(attr,lang)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ninitializes the module cache.", "response": "def _init_module_cache():\n\t\t\"\"\"\n\t\tModule caching, it helps with not having to import again and again same modules.\n\t\t@return: boolean, True if module caching has been done, False if module caching was already done.\n\t\t\"\"\"\n\n\t\t# While there are not loaded modules, load these ones\n\t\tif len(FieldTranslation._modules) < len(FieldTranslation._model_module_paths):\n\t\t\tfor module_path in FieldTranslation._model_module_paths:\n\t\t\t\tFieldTranslation._modules[module_path] = importlib.import_module(module_path)\n\t\t\treturn True\n\t\treturn False"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nloads the source model of the FieldTranslation object.", "response": "def _load_source_model(self):\n\t\t\"\"\"\n\t\tLoads and gets the source model of the FieldTranslation as a dynamic attribute. It is used only when deleting\n\t\torphan translations (translations without a parent object associated).\n\t\t\"\"\"\n\n\t\t# If source_model exists, return it\n\t\tif hasattr(self, \"source_model\"):\n\t\t\treturn self.source_model\n\n\t\t# Getting the source model\n\t\tmodule = self.get_python_module()\n\n\t\t# Test if module has inside the model we are looking for\n\t\tif hasattr(module, self.model):\n\t\t\t# Setting of source model\n\t\t\tself.source_model = getattr(module, self.model)\n\t\t\t# Setting of verbose name and its plural for later use\n\t\t\tself.source_model.meta__verbose_name = self.source_model._meta.verbose_name\n\t\t\tself.source_model.meta__verbose_name_plural = self.source_model._meta.verbose_name_plural\n\t\t\treturn self.source_model\n\t\t\n\t\traise ValueError(u\"Model {0} does not exist in module\".format(self.model, self.module))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _load_source_object(self):\n\t\tif hasattr(self, \"source_obj\"):\n\t\t\tself.source_text = getattr(self.source_obj, self.field)\n\t\t\treturn self.source_obj\n\n\t\tself._load_source_model()\n\t\tself.source_obj = self.source_model.objects.get(id=self.object_id)\n\t\treturn self.source_obj", "response": "Loads related object in a dynamic attribute and returns it."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndeleting orphan translations. This method needs refactoring to be improve its performance.", "response": "def delete_orphan_translations(condition=None):\n\t\t\"\"\"\n\t\tDelete orphan translations.\n\t\tThis method needs refactoring to be improve its performance.\n\t\t\"\"\"\n\t\tif condition is None:\n\t\t\tcondition = {}\n\t\t# TODO: optimize using one SQL sentence\n\t\ttranslations = FieldTranslation.objects.all()\n\t\tfor translation in translations:\n\t\t\ttranslation._load_source_model()\n\t\t\tcondition[\"id\"] = translation.object_id\n\t\t\tif not translation.source_model.objects.filter(**condition).exists():\n\t\t\t\ttranslation.delete()"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nupdate the FieldTranslations table with the given condition.", "response": "def update_translations(condition=None):\n\t\t\"\"\"\n\t\tUpdates FieldTranslations table\n\t\t\"\"\"\n\t\tif condition is None:\n\t\t\tcondition = {}\n\n\t\t# Number of updated translations\n\t\tnum_translations = 0\n\t\t\n\t\t# Module caching\n\t\tFieldTranslation._init_module_cache()\n\n\t\t# Current languages dict\n\t\tLANGUAGES = dict(lang for lang in MODELTRANSLATION_LANG_CHOICES)\n\t\tif settings.LANGUAGE_CODE in LANGUAGES:\n\t\t\tdel LANGUAGES[settings.LANGUAGE_CODE]\n\n\t\t# For each module, we are going to update the translations\n\t\tfor key in FieldTranslation._modules.keys():\n\t\t\tmodule = FieldTranslation._modules[key]\n\n\t\t\t# Class of the module\n\t\t\tclsmembers = inspect.getmembers(sys.modules[key], inspect.isclass)\n\t\t\tfor cls in clsmembers:\n\t\t\t\tcls = cls[1]\n\n\t\t\t\t# If the model has in Meta \"translatable_fields\", we insert this fields\n\t\t\t\tif hasattr(cls,\"_meta\") and not cls._meta.abstract and hasattr(cls._meta,\"translatable_fields\") and len(cls._meta.translatable_fields)>0:\n\t\t\t\t\tobjects = cls.objects.filter(**condition)\n\t\t\t\t\t\n\t\t\t\t\t# For each object, language and field are updated\n\t\t\t\t\tfor obj in objects:\n\t\t\t\t\t\tfor lang in LANGUAGES.keys():\n\t\t\t\t\t\t\tfor field in cls._meta.translatable_fields:\n\t\t\t\t\t\t\t\tif FieldTranslation.update(obj=obj, field=field, lang=lang, context=\"\"):\n\t\t\t\t\t\t\t\t\tnum_translations += 1\n\t\treturn num_translations"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsave the object in database updating the datetimes accordingly.", "response": "def save(self, *args, **kwargs):\n\t\t\"\"\"\n\t\tSave object in database, updating the datetimes accordingly.\n\t\t\n\t\t\"\"\"\n\t\t# Now in UTC\n\t\tnow_datetime = timezone.now()\n\n\t\t# If we are in a creation, assigns creation_datetime\n\t\tif not self.id:\n\t\t\tself.creation_datetime = now_datetime\n\n\t\t# Las update datetime is always updated\n\t\tself.last_update_datetime = now_datetime\n\t\t\n\t\t# Current user is creator\n\t\t# (get current user with django-cuser middleware)\n\t\tself.creator_user = None\n\t\tcurrent_user = CuserMiddleware.get_user()\n\t\tif not current_user is None and not current_user.is_anonymous():\n\t\t\tself.creator_user_id = current_user.id\n\t\t\n\t\t# Parent constructor call\n\t\tsuper(FieldTranslation, self).save(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef parse_info(raw_info, apply_tag=None):\n parse_releases(raw_info)\n parse_packages(raw_info, apply_tag=apply_tag)\n return raw_info", "response": "Parse raw rdoinfo metadata inplace."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts a dictionary of info to a list of dicts indexed by project and name.", "response": "def info2dicts(info, in_place=False):\n \"\"\"\n Return info with:\n\n 1) `packages` list replaced by a 'packages' dict indexed by 'project'\n 2) `releases` list replaced by a 'releases' dict indexed by 'name'\n \"\"\"\n if 'packages' not in info and 'releases' not in info:\n return info\n if in_place:\n info_dicts = info\n else:\n info_dicts = info.copy()\n packages = info.get('packages')\n if packages:\n info_dicts['packages'] = list2dict(packages, 'project')\n releases = info.get('releases')\n if releases:\n info_dicts['releases'] = list2dict(releases, 'name')\n return info_dicts"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef info2lists(info, in_place=False):\n\n \"\"\"\n Return info with:\n\n 1) `packages` dict replaced by a 'packages' list with indexes removed\n 2) `releases` dict replaced by a 'releases' list with indexes removed\n\n info2list(info2dicts(info)) == info\n \"\"\"\n if 'packages' not in info and 'releases' not in info:\n return info\n if in_place:\n info_lists = info\n else:\n info_lists = info.copy()\n packages = info.get('packages')\n if packages:\n info_lists['packages'] = list(packages.values())\n releases = info.get('releases')\n if releases:\n info_lists['releases'] = list(releases.values())\n return info_lists", "response": "Convert info dict to lists of dicts."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef alphanumeric(text):\n return \"\".join([c for c in text if re.match(r'\\w', c)])", "response": "Make an ASCII version a string.\n \\ w matches any alphanumeric character and the underscore."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nmake a safe representation of a string.", "response": "def sanitize_text(text):\n \"\"\"Make a safe representation of a string.\n Note: the `\\s` special character matches any whitespace character.\n This is equivalent to the set [\\t\\n\\r\\f\\v] as well as ` ` (whitespace).\"\"\"\n # First replace characters that have specific effects with their repr #\n text = re.sub(\"(\\s)\", lambda m: repr(m.group(0)).strip(\"'\"), text)\n # Make it a unicode string (the try supports python 2 and 3) #\n try: text = text.decode('utf-8')\n except AttributeError: pass\n # Normalize it \u201c\n text = unicodedata.normalize('NFC', text)\n return text"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef camel_to_snake(text):\n step_one = re.sub('(.)([A-Z][a-z]+)', r'\\1_\\2', text)\n step_two = re.sub('([a-z0-9])([A-Z])', r'\\1_\\2', step_one)\n return step_two.lower()", "response": "Converts a string from CamelCaseStrings to snake_case_strings."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconvert a boolean to a unicode string.", "response": "def bool_to_unicode(b):\n \"\"\"Different possibilities for True: \u2611\ufe0f\u2714\ufe0e\u2713\u2705\ud83d\udc4d\u2714\ufe0f\n Different possibilities for False: \u2715\u2716\ufe0e\u2717\u2718\u2716\ufe0f\u274c\u26d4\ufe0f\u274e\ud83d\udc4e\ud83d\uded1\ud83d\udd34\"\"\"\n b = bool(b)\n if b is True: return u\"\u2705\"\n if b is False: return u\"\u274e\""} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\naccessing a dictionary like if it was an object with properties.", "response": "def access_dict_like_obj(obj, prop, new_value=None):\n \"\"\"\n Access a dictionary like if it was an object with properties.\n If no \"new_value\", then it's a getter, otherwise it's a setter.\n >>> {'characters': {'cast': 'Jean-Luc Picard', 'featuring': 'Deanna Troi'}}\n >>> access_dict_like_obj(startrek, 'characters.cast', 'Pierce Brosnan')\n \"\"\"\n props = prop.split('.')\n if new_value:\n if props[0] not in obj: obj[props[0]] = {}\n if len(props)==1: obj[prop] = new_value\n else: return access_dict_like_obj(obj[props[0]], '.'.join(props[1:]), new_value)\n else:\n if len(props)==1: return obj[prop]\n else: return access_dict_like_obj(obj[props[0]], '.'.join(props[1:]))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngenerates all combinations of a given list of items.", "response": "def all_combinations(items):\n \"\"\"Generate all combinations of a given list of items.\"\"\"\n return (set(compress(items,mask)) for mask in product(*[[0,1]]*len(items)))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngiving a multiline string add whitespaces to every line of the multiline string so that every line has the same length.", "response": "def pad_equal_whitespace(string, pad=None):\n \"\"\"Given a multiline string, add whitespaces to every line\n so that every line has the same length.\"\"\"\n if pad is None: pad = max(map(len, string.split('\\n'))) + 1\n return '\\n'.join(('{0: <%i}' % pad).format(line) for line in string.split('\\n'))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nzipping two strings together line wise", "response": "def concatenate_by_line(first, second):\n \"\"\"Zip two strings together, line wise\"\"\"\n return '\\n'.join(x+y for x,y in zip(first.split('\\n'), second.split('\\n')))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef sort_string_by_pairs(strings):\n assert len(strings) % 2 == 0\n pairs = []\n strings = list(strings) # This shallow copies the list\n while strings:\n template = strings.pop()\n for i, candidate in enumerate(strings):\n if count_string_diff(template, candidate) == 1:\n pair = [template, strings.pop(i)]\n pair.sort()\n pairs.append(pair)\n break\n return pairs", "response": "Group a list of strings by pairs by matching those with only\n one character difference between each other together."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the number of characters in two strings that don t exactly match", "response": "def count_string_diff(a,b):\n \"\"\"Return the number of characters in two strings that don't exactly match\"\"\"\n shortest = min(len(a), len(b))\n return sum(a[i] != b[i] for i in range(shortest))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef uniquify_list(L):\n return [e for i, e in enumerate(L) if L.index(e) == i]", "response": "Same order unique list using only a list compression."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_next_item(iterable):\n try: x = iterable.next()\n except StopIteration: x = None\n except AttributeError: x = None\n return x", "response": "Gets the next item of an iterable."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef pretty_now():\n import datetime, tzlocal\n time_zone = tzlocal.get_localzone()\n now = datetime.datetime.now(time_zone)\n return now.strftime(\"%Y-%m-%d %H:%M:%S %Z%z\")", "response": "Returns some thing like 2011 - 02 - 15 - 58 CET + 0100"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef andify(list_of_strings):\n result = ', '.join(list_of_strings)\n comma_index = result.rfind(',')\n if comma_index > -1: result = result[:comma_index] + ' and' + result[comma_index+1:]\n return result", "response": "Given a list of strings will join them with commas and a final and word."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts a number to the th th.", "response": "def num_to_ith(num):\n \"\"\"1 becomes 1st, 2 becomes 2nd, etc.\"\"\"\n value = str(num)\n before_last_digit = value[-2]\n last_digit = value[-1]\n if len(value) > 1 and before_last_digit == '1': return value +'th'\n if last_digit == '1': return value + 'st'\n if last_digit == '2': return value + 'nd'\n if last_digit == '3': return value + 'rd'\n return value + 'th'"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndown - sample an enumerable list of things", "response": "def isubsample(full_sample, k, full_sample_len=None):\n \"\"\"Down-sample an enumerable list of things\"\"\"\n # Determine length #\n if not full_sample_len: full_sample_len = len(full_sample)\n # Check size coherence #\n if not 0 <= k <= full_sample_len:\n raise ValueError('Required that 0 <= k <= full_sample_length')\n # Do it #\n picked = 0\n for i, element in enumerate(full_sample):\n prob = (k-picked) / (full_sample_len-i)\n if random.random() < prob:\n yield element\n picked += 1\n # Did we pick the right amount #\n assert picked == k"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef wait(predicate, interval=1, message=lambda: \"Waiting...\"):\n ball, next_ball = u\"|/-\\\\\", \"|\"\n sys.stdout.write(\" \\033[K\")\n sys.stdout.flush()\n while not predicate():\n time.sleep(1)\n next_ball = ball[(ball.index(next_ball) + 1) % len(ball)]\n sys.stdout.write(\"\\r \" + str(message()) + \" \" + next_ball + \" \\033[K\")\n sys.stdout.flush()\n print(\"\\r Done. \\033[K\")\n sys.stdout.flush()", "response": "Wait until the predicate turns true and display a turning ball."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsort strings that contain numbers correctly. Works in Python 2 and 3.", "response": "def natural_sort(item):\n \"\"\"\n Sort strings that contain numbers correctly. Works in Python 2 and 3.\n\n >>> l = ['v1.3.12', 'v1.3.3', 'v1.2.5', 'v1.2.15', 'v1.2.3', 'v1.2.1']\n >>> l.sort(key=natural_sort)\n >>> l.__repr__()\n \"['v1.2.1', 'v1.2.3', 'v1.2.5', 'v1.2.15', 'v1.3.3', 'v1.3.12']\"\n \"\"\"\n dre = re.compile(r'(\\d+)')\n return [int(s) if s.isdigit() else s.lower() for s in re.split(dre, item)]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef split_thousands(s):\n # Check input #\n if s is None: return \"0\"\n # If it's a string #\n if isinstance(s, basestring): s = float(s)\n # If it's a float that should be an int #\n if isinstance(s, float) and s.is_integer(): s = int(s)\n # Use python built-in #\n result = \"{:,}\".format(s)\n # But we want single quotes #\n result = result.replace(',', \"'\")\n # Return #\n return result", "response": "Splits a number on thousands."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef reverse_compl_with_name(old_seq):\n new_seq = old_seq.reverse_complement()\n new_seq.id = old_seq.id\n new_seq.description = old_seq.description\n return new_seq", "response": "Reverse a SeqIO sequence but keep its name intact."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load_json_path(path):\n with open(path) as handle:\n try: return json.load(handle, object_pairs_hook=collections.OrderedDict)\n except ValueError as error:\n message = \"Could not decode JSON file '%s'.\" % path\n message = \"-\"*20 + \"\\n\" + message + \"\\n\" + str(error) + \"\\n\" + \"-\"*20 + \"\\n\"\n sys.stderr.write(message)\n raise error", "response": "Load a file with the json module and return a dict."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncompute the md5 of a file. Pretty fast.", "response": "def md5sum(file_path, blocksize=65536):\n \"\"\"Compute the md5 of a file. Pretty fast.\"\"\"\n md5 = hashlib.md5()\n with open(file_path, \"rb\") as f:\n for block in iter(lambda: f.read(blocksize), \"\"):\n md5.update(block)\n return md5.hexdigest()"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef download_from_url(source, destination, progress=False, uncompress=False):\n # Modules #\n from tqdm import tqdm\n import requests\n from autopaths.file_path import FilePath\n # Check destination exists #\n destination = FilePath(destination)\n destination.directory.create_if_not_exists()\n # Over HTTP #\n response = requests.get(source, stream=True)\n total_size = int(response.headers.get('content-length'))\n block_size = int(total_size/1024)\n # Do it #\n with open(destination, \"wb\") as handle:\n if progress:\n for data in tqdm(response.iter_content(chunk_size=block_size), total=1024): handle.write(data)\n else:\n for data in response.iter_content(chunk_size=block_size): handle.write(data)\n # Uncompress #\n if uncompress:\n with open(destination) as f: header = f.read(4)\n if header == \"PK\\x03\\x04\": unzip(destination, inplace=True)\n # Add other compression formats here\n # Return #\n return destination", "response": "Download a file from an URL and place it somewhere."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef reversed_lines(path):\n with open(path, 'r') as handle:\n part = ''\n for block in reversed_blocks(handle):\n for c in reversed(block):\n if c == '\\n' and part:\n yield part[::-1]\n part = ''\n part += c\n if part: yield part[::-1]", "response": "Generate the lines of file in reverse order."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef reversed_blocks(handle, blocksize=4096):\n handle.seek(0, os.SEEK_END)\n here = handle.tell()\n while 0 < here:\n delta = min(blocksize, here)\n here -= delta\n handle.seek(here, os.SEEK_SET)\n yield handle.read(delta)", "response": "Generate blocks of file s contents in reverse order."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef which(cmd, safe=False):\n from autopaths.file_path import FilePath\n def is_executable(path):\n return os.path.exists(path) and os.access(path, os.X_OK) and not os.path.isdir(path)\n path, name = os.path.split(cmd)\n if path:\n if is_executable(cmd): return FilePath(cmd)\n else:\n for path in os.environ['PATH'].split(os.pathsep):\n candidate = os.path.join(path, cmd)\n if is_executable(candidate): return FilePath(candidate)\n if not safe: raise Exception('which failed to locate a proper command path \"%s\"' % cmd)", "response": "Find the command path."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a tag that includes the stylesheet.", "response": "def css_tag(parser, token):\n \"\"\"\n Renders a tag to include the stylesheet. It takes an optional second \n parameter for the media attribute; the default media is \"screen, projector\".\n \n Usage::\n\n {% css \".css\" [\"\"] %}\n\n Examples::\n\n {% css \"myfile.css\" %}\n {% css \"myfile.css\" \"screen, projection\"%}\n \"\"\"\n path = get_path_from_tokens(token)\n\n tokens = token.split_contents()\n if len(tokens) > 2:\n # Get the media types from the tag call provided by the user.\n media_type = tokens[2][1:-1]\n else:\n # Default values.\n media_type = \"screen, projection\"\n\n return CssTagNode(path, media_type=media_type)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef css_print_tag(parser, token):\n path = get_path_from_tokens(token)\n # Hard wired media type, since this is for media type of 'print'.\n media_type = \"print\"\n\n return CssTagNode(path, media_type=media_type)", "response": "Shortcut to render CSS as a print stylesheet."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef supports_gzip(self, context):\n if 'request' in context and client.supports_gzip():\n enc = context['request'].META.get('HTTP_ACCEPT_ENCODING', '')\n return 'gzip' in enc and msettings['SERVE_REMOTE']\n return False", "response": "Returns True if the client supports gzip encoded content."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the URL of the current media source for the given context.", "response": "def get_media_url(self, context):\n \"\"\"\n Checks to see whether to use the normal or the secure media source,\n depending on whether the current page view is being sent over SSL.\n The USE_SSL setting can be used to force HTTPS (True) or HTTP (False).\n \n NOTE: Not all backends implement SSL media. In this case, they'll just\n return an unencrypted URL.\n \"\"\"\n use_ssl = msettings['USE_SSL']\n is_secure = use_ssl if use_ssl is not None else self.is_secure(context)\n return client.media_url(with_ssl=True) if is_secure else client.media_url()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nassemble various components to form a complete resource URL.", "response": "def mkpath(self, url, path, filename=None, gzip=False):\n \"\"\"\n Assembles various components to form a complete resource URL.\n \n args:\n url: (str) A base media URL.\n path: (str) The path on the host (specified in 'url') leading up\n to the file.\n filename: (str) The file name to serve.\n gzip: (bool) True if client should receive *.gzt version of file.\n \"\"\"\n if path:\n url = \"%s/%s\" % (url.rstrip('/'), path.strip('/'))\n\n if filename:\n url = \"%s/%s\" % (url, filename.lstrip('/'))\n \n content_type = mimetypes.guess_type(url)[0]\n if gzip and content_type in mediasync.TYPES_TO_COMPRESS:\n url = \"%s.gzt\" % url\n\n cb = msettings['CACHE_BUSTER']\n if cb:\n # Cache busters help tell the client to re-download the file after\n # a change. This can either be a callable or a constant defined\n # in settings.py.\n cb_val = cb(url) if callable(cb) else cb\n url = \"%s?%s\" % (url, cb_val)\n\n return msettings['URL_PROCESSOR'](url)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef linktag(self, url, path, filename, media, context):\n if msettings['DOCTYPE'] == 'xhtml':\n markup = \"\"\"\"\"\"\n elif msettings['DOCTYPE'] == 'html5':\n markup = \"\"\"\"\"\"\n else:\n markup = \"\"\"\"\"\"\n return markup % (self.mkpath(url, path, filename, gzip=self.supports_gzip(context)), media)", "response": "Renders a link tag for the stylesheet."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nrendering a script tag for the JS file.", "response": "def scripttag(self, url, path, filename, context):\n \"\"\"\n Renders a '%(''\\\n if eq['is_unreferenceable'] \\\n else 'id=\"%s\"'%label))\n innerspan = RawInline('html',\n '' %\n ('0' if text.startswith('$') and\n text.endswith('$') else '50%',))\n num = Math({\"t\":\"InlineMath\"}, '(%s)' % text[1:-1]) \\\n if text.startswith('$') and text.endswith('$') \\\n else Str('(%s)' % text)\n endspans = RawInline('html', '')\n return [outerspan, AttrMath(*value), innerspan, num, endspans]\n elif fmt == 'docx':\n # As per http://officeopenxml.com/WPhyperlink.php\n bookmarkstart = \\\n RawInline('openxml',\n ''\n %label)\n bookmarkend = \\\n RawInline('openxml',\n '')\n return [bookmarkstart, AttrMath(*value), bookmarkend]\n\n return None", "response": "Processes the attributed equations."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef process(meta):\n\n # pylint: disable=global-statement\n global capitalize\n global use_cleveref_default\n global plusname\n global starname\n global numbersections\n\n # Read in the metadata fields and do some checking\n\n for name in ['eqnos-cleveref', 'xnos-cleveref', 'cleveref']:\n # 'xnos-cleveref' enables cleveref in all 3 of fignos/eqnos/tablenos\n # 'cleveref' is deprecated\n if name in meta:\n use_cleveref_default = check_bool(get_meta(meta, name))\n break\n\n for name in ['eqnos-capitalize', 'eqnos-capitalise',\n 'xnos-capitalize', 'xnos-capitalise']:\n # 'eqnos-capitalise' is an alternative spelling\n # 'xnos-capitalise' enables capitalise in all 3 of fignos/eqnos/tablenos\n # 'xnos-capitalize' is an alternative spelling\n if name in meta:\n capitalize = check_bool(get_meta(meta, name))\n break\n\n if 'eqnos-plus-name' in meta:\n tmp = get_meta(meta, 'eqnos-plus-name')\n if isinstance(tmp, list):\n plusname = tmp\n else:\n plusname[0] = tmp\n assert len(plusname) == 2\n for name in plusname:\n assert isinstance(name, STRTYPES)\n\n if 'eqnos-star-name' in meta:\n tmp = get_meta(meta, 'eqnos-star-name')\n if isinstance(tmp, list):\n starname = tmp\n else:\n starname[0] = tmp\n assert len(starname) == 2\n for name in starname:\n assert isinstance(name, STRTYPES)\n\n if 'xnos-number-sections' in meta:\n numbersections = check_bool(get_meta(meta, 'xnos-number-sections'))", "response": "Saves metadata fields in global variables and returns a few\n computed fields."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfilter the document AST.", "response": "def main():\n \"\"\"Filters the document AST.\"\"\"\n\n # pylint: disable=global-statement\n global PANDOCVERSION\n global AttrMath\n\n # Get the output format and document\n fmt = args.fmt\n doc = json.loads(STDIN.read())\n\n # Initialize pandocxnos\n # pylint: disable=too-many-function-args\n PANDOCVERSION = pandocxnos.init(args.pandocversion, doc)\n\n # Element primitives\n AttrMath = elt('Math', 3)\n\n # Chop up the doc\n meta = doc['meta'] if PANDOCVERSION >= '1.18' else doc[0]['unMeta']\n blocks = doc['blocks'] if PANDOCVERSION >= '1.18' else doc[1:]\n\n # Process the metadata variables\n process(meta)\n\n # First pass\n attach_attrs_math = attach_attrs_factory(Math, allow_space=True)\n detach_attrs_math = detach_attrs_factory(Math)\n insert_secnos = insert_secnos_factory(Math)\n delete_secnos = delete_secnos_factory(Math)\n altered = functools.reduce(lambda x, action: walk(x, action, fmt, meta),\n [attach_attrs_math, insert_secnos,\n process_equations, delete_secnos,\n detach_attrs_math], blocks)\n\n # Second pass\n process_refs = process_refs_factory(references.keys())\n replace_refs = replace_refs_factory(references,\n use_cleveref_default, use_eqref,\n plusname if not capitalize else\n [name.title() for name in plusname],\n starname, 'equation')\n altered = functools.reduce(lambda x, action: walk(x, action, fmt, meta),\n [repair_refs, process_refs, replace_refs],\n altered)\n\n # Update the doc\n if PANDOCVERSION >= '1.18':\n doc['blocks'] = altered\n else:\n doc = doc[:1] + altered\n\n # Dump the results\n json.dump(doc, STDOUT)\n\n # Flush stdout\n STDOUT.flush()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef check_debug():\n from django.conf import settings\n\n if not settings.configured:\n return False\n\n # I _think_ this check is all that's needed and the 3 \"hasattr\" checks\n # below can be removed, but it's not clear how to verify that\n from django.apps import apps\n if not apps.ready:\n return False\n\n # django.template.backends.django gets loaded lazily, so return false\n # until they've been loaded\n if not hasattr(django.template, \"backends\"):\n return False\n if not hasattr(django.template.backends, \"django\"):\n return False\n if not hasattr(django.template.backends.django, \"DjangoTemplates\"):\n raise DjangoTemplatePluginException(\"Can't use non-Django templates.\")\n\n for engine in django.template.engines.all():\n if not isinstance(engine, django.template.backends.django.DjangoTemplates):\n raise DjangoTemplatePluginException(\n \"Can't use non-Django templates.\"\n )\n if not engine.engine.debug:\n raise DjangoTemplatePluginException(\n \"Template debugging must be enabled in settings.\"\n )\n\n return True", "response": "Checks that Django s template debugging is enabled and raises an exception if it is not."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef read_template_source(filename):\n # Import this late to be sure we don't trigger settings machinery too\n # early.\n from django.conf import settings\n\n if not settings.configured:\n settings.configure()\n\n with open(filename, \"rb\") as f:\n text = f.read().decode(settings.FILE_CHARSET)\n\n return text", "response": "Read the source of a Django template returning the Unicode text."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_line_number(line_map, offset):\n for lineno, line_offset in enumerate(line_map, start=1):\n if line_offset > offset:\n return lineno\n return -1", "response": "Find a line number given a line map and a character offset."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndumping interesting information about this frame.", "response": "def dump_frame(frame, label=\"\"):\n \"\"\"Dump interesting information about this frame.\"\"\"\n locals = dict(frame.f_locals)\n self = locals.get('self', None)\n context = locals.get('context', None)\n if \"__builtins__\" in locals:\n del locals[\"__builtins__\"]\n\n if label:\n label = \" ( %s ) \" % label\n print(\"-- frame --%s---------------------\" % label)\n print(\"{}:{}:{}\".format(\n os.path.basename(frame.f_code.co_filename),\n frame.f_lineno,\n type(self),\n ))\n print(locals)\n if self:\n print(\"self:\", self.__dict__)\n if context:\n print(\"context:\", context.__dict__)\n print(\"\\\\--\")"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_line_map(self, filename):\n if filename not in self.source_map:\n template_source = read_template_source(filename)\n if 0: # change to see the template text\n for i in range(0, len(template_source), 10):\n print(\"%3d: %r\" % (i, template_source[i:i+10]))\n self.source_map[filename] = make_line_map(template_source)\n return self.source_map[filename]", "response": "Returns a line map for filename."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsaves bytes to a file.", "response": "def bytes_to_file(input_data, output_file):\n \"\"\"Save bytes to a file.\"\"\"\n pathlib.Path(output_file.parent).mkdir(parents=True, exist_ok=True)\n\n with open(output_file, \"wb\") as file:\n file.write(input_data)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_art(cache_dir, size, client):\n song = client.currentsong()\n\n if len(song) < 2:\n print(\"album: Nothing currently playing.\")\n return\n\n file_name = f\"{song['artist']}_{song['album']}_{size}.jpg\".replace(\"/\", \"\")\n file_name = cache_dir / file_name\n\n if file_name.is_file():\n shutil.copy(file_name, cache_dir / \"current.jpg\")\n print(\"album: Found cached art.\")\n\n else:\n print(\"album: Downloading album art...\")\n\n brainz.init()\n album_art = brainz.get_cover(song, size)\n\n if album_art:\n util.bytes_to_file(album_art, cache_dir / file_name)\n util.bytes_to_file(album_art, cache_dir / \"current.jpg\")\n\n print(f\"album: Swapped art to {song['artist']}, {song['album']}.\")", "response": "Get the album art."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndownload the cover art.", "response": "def get_cover(song, size=250):\n \"\"\"Download the cover art.\"\"\"\n try:\n data = mus.search_releases(artist=song[\"artist\"],\n release=song[\"album\"],\n limit=1)\n release_id = data[\"release-list\"][0][\"release-group\"][\"id\"]\n print(f\"album: Using release-id: {data['release-list'][0]['id']}\")\n\n return mus.get_release_group_image_front(release_id, size=size)\n\n except mus.NetworkError:\n get_cover(song, size)\n\n except mus.ResponseError:\n print(\"error: Couldn't find album art for\",\n f\"{song['artist']} - {song['album']}\")"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_args():\n description = \"bum - Download and display album art \\\n for mpd tracks.\"\n arg = argparse.ArgumentParser(description=description)\n\n arg.add_argument(\"--size\", metavar=\"\\\"px\\\"\",\n help=\"what size to display the album art in.\",\n default=250)\n\n arg.add_argument(\"--cache_dir\", metavar=\"\\\"/path/to/dir\\\"\",\n help=\"Where to store the downloaded cover art.\",\n default=pathlib.Path.home() / \".cache/bum\")\n\n arg.add_argument(\"--version\", action=\"store_true\",\n help=\"Print \\\"bum\\\" version.\")\n\n arg.add_argument(\"--port\",\n help=\"Use a custom mpd port.\",\n default=6600)\n\n arg.add_argument(\"--server\",\n help=\"Use a remote server instead of localhost.\",\n default=\"localhost\")\n\n arg.add_argument(\"--no_display\",\n action=\"store_true\",\n help=\"Only download album art, don't display.\")\n\n return arg.parse_args()", "response": "Get the script arguments."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts from source format into target format. Relevant if converters are used in InputTemplates.", "response": "def convertforinput(self,filepath, metadata):\n \"\"\"Convert from target format into one of the source formats. Relevant if converters are used in InputTemplates. Metadata already is metadata for the to-be-generated file. 'filepath' is both the source and the target file, the source file will be erased and overwritten with the conversion result!\"\"\"\n assert isinstance(metadata, CLAMMetaData) #metadata of the destination file (file to be generated here)\n if not metadata.__class__ in self.acceptforinput:\n raise Exception(\"Convertor \" + self.__class__.__name__ + \" can not convert input files to \" + metadata.__class__.__name__ + \"!\")\n return False"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef convertforoutput(self,outputfile):\n assert isinstance(outputfile, CLAMOutputFile) #metadata of the destination file (file to be generated here)\n if not outputfile.metadata.__class__ in self.acceptforoutput:\n raise Exception(\"Convertor \" + self.__class__.__name__ + \" can not convert input files to \" + outputfile.metadata.__class__.__name__ + \"!\")\n return []", "response": "Convert from one of the source formats into one of the target formats. Relevant if converters are used in OutputTemplates. Sourcefile is a CLAMOutputFile instance."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef convertforinput(self,filepath, metadata=None):\n super(CharEncodingConverter,self).convertforinput(filepath, metadata)\n\n shutil.copy(filepath, filepath + '.convertsource')\n\n try:\n fsource = io.open(filepath + '.convertsource','r',encoding=self.charset)\n ftarget = io.open(filepath,'w',encoding=metadata['encoding'])\n for line in fsource:\n ftarget.write(line + \"\\n\")\n success = True\n except:\n ftarget.close()\n fsource.fclose()\n success = False\n finally:\n os.unlink(filepath + '.convertsource')\n\n return success", "response": "Convert from source format into one of the source formats. Relevant if converters are used in InputTemplates. Metadata already is metadata for the to - be - generated file."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef convertforoutput(self,outputfile):\n super(CharEncodingConverter,self).convertforoutput(outputfile)\n\n return withheaders( flask.make_response( ( line.encode(self.charset) for line in outputfile ) ) , 'text/plain; charset=' + self.charset)", "response": "Convert from one of the source formats into one of the target formats. Relevant in OutputTemplates. Outputfile is a CLAMOutputFile instance."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef getclamdata(filename, custom_formats=None):\n global CUSTOM_FORMATS #pylint: disable=global-statement\n \"\"\"This function reads the CLAM Data from an XML file. Use this to read\n the clam.xml file from your system wrapper. It returns a CLAMData instance.\n\n If you make use of CUSTOM_FORMATS, you need to pass the CUSTOM_FORMATS list as 2nd argument.\n \"\"\"\n f = io.open(filename,'r',encoding='utf-8')\n xml = f.read(os.path.getsize(filename))\n f.close()\n if custom_formats:\n CUSTOM_FORMATS = custom_formats #dependency injection for CUSTOM_FORMATS\n return CLAMData(xml, None, True)", "response": "This function reads the CLAM Data from an XML file."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsanitize the parameters for internal use only", "response": "def sanitizeparameters(parameters):\n \"\"\"Construct a dictionary of parameters, for internal use only\"\"\"\n if not isinstance(parameters,dict):\n d = {}\n for x in parameters:\n if isinstance(x,tuple) and len(x) == 2:\n for parameter in x[1]:\n d[parameter.id] = parameter\n elif isinstance(x, clam.common.parameters.AbstractParameter):\n d[x.id] = x\n return d\n else:\n return parameters"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngiving input files and parameters produce metadata for outputfiles. Returns a list of matched profiles and a program.", "response": "def profiler(profiles, projectpath,parameters,serviceid,servicename,serviceurl,printdebug=None):\n \"\"\"Given input files and parameters, produce metadata for outputfiles. Returns a list of matched profiles (empty if none match), and a program.\"\"\"\n\n parameters = sanitizeparameters(parameters)\n\n matched = []\n program = Program(projectpath)\n for profile in profiles:\n if profile.match(projectpath, parameters)[0]:\n matched.append(profile)\n program.update( profile.generate(projectpath,parameters,serviceid,servicename,serviceurl) )\n\n return matched, program"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the value string wrapped in the specified quotes if not empty", "response": "def shellsafe(s, quote='', doescape=True):\n \"\"\"Returns the value string, wrapped in the specified quotes (if not empty), but checks and raises an Exception if the string is at risk of causing code injection\"\"\"\n if sys.version[0] == '2' and not isinstance(s,unicode): #pylint: disable=undefined-variable\n s = unicode(s,'utf-8') #pylint: disable=undefined-variable\n if len(s) > 1024:\n raise ValueError(\"Variable value rejected for security reasons: too long\")\n if quote:\n if quote in s:\n if doescape:\n s = escape(s,quote)\n else:\n raise ValueError(\"Variable value rejected for security reasons: \" + s)\n return quote + s + quote\n else:\n for c in s:\n if c in DISALLOWINSHELLSAFE:\n raise ValueError(\"Variable value rejected for security reasons: \" + s)\n return s"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nattach viewers and converters to file automatically scan all profiles for outputtemplate or inputtemplate", "response": "def attachviewers(self, profiles):\n \"\"\"Attach viewers *and converters* to file, automatically scan all profiles for outputtemplate or inputtemplate\"\"\"\n if self.metadata:\n template = None\n for profile in profiles:\n if isinstance(self, CLAMInputFile):\n for t in profile.input:\n if self.metadata.inputtemplate == t.id:\n template = t\n break\n elif isinstance(self, CLAMOutputFile) and self.metadata and self.metadata.provenance:\n for t in profile.outputtemplates():\n if self.metadata.provenance.outputtemplate_id == t.id:\n template = t\n break\n else:\n raise NotImplementedError #Is ok, nothing to implement for now\n if template:\n break\n if template and template.viewers:\n for viewer in template.viewers:\n self.viewers.append(viewer)\n if template and template.converters:\n for converter in template.converters:\n self.converters.append(converter)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the filename for the metadata file. Only used for local files. Only used for local files.", "response": "def metafilename(self):\n \"\"\"Returns the filename for the metadata file (not full path). Only used for local files.\"\"\"\n metafilename = os.path.dirname(self.filename)\n if metafilename: metafilename += '/'\n metafilename += '.' + os.path.basename(self.filename) + '.METADATA'\n return metafilename"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef loadmetadata(self):\n if not self.remote:\n metafile = self.projectpath + self.basedir + '/' + self.metafilename()\n if os.path.exists(metafile):\n f = io.open(metafile, 'r',encoding='utf-8')\n xml = \"\".join(f.readlines())\n f.close()\n else:\n raise IOError(2, \"No metadata found, expected \" + metafile )\n else:\n if self.client:\n requestparams = self.client.initrequest()\n else:\n requestparams = {}\n response = requests.get(self.projectpath + self.basedir + '/' + self.filename + '/metadata', **requestparams)\n if response.status_code != 200:\n extramsg = \"\"\n if not self.client: extramsg = \"No client was associated with this CLAMFile, associating a client is necessary when authentication is needed\"\n raise HTTPError(2, \"Can't download metadata for \" + self.filename + \". \" + extramsg)\n xml = response.text\n\n\n #parse metadata\n try:\n self.metadata = CLAMMetaData.fromxml(xml, self) #returns CLAMMetaData object (or child thereof)\n except ElementTree.XMLSyntaxError:\n raise ValueError(\"Metadata is not XML! Contents: \" + xml)", "response": "Load metadata for this file. This is usually called automatically upon instantiation except if explicitly disabled. Works both locally as well as for clients connecting to a CLAM service."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nloading all lines in memory", "response": "def read(self):\n \"\"\"Loads all lines in memory\"\"\"\n lines = self.readlines()\n if self.metadata and 'encoding' in self.metadata:\n encoding = self.metadata['encoding']\n else:\n encoding = 'utf-8'\n if sys.version < '3':\n return \"\\n\".join( unicode(line, 'utf-8') if isinstance(line, str) else line for line in lines)\n else:\n return \"\\n\".join( str(line, 'utf-8') if isinstance(line, bytes) else line for line in lines)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef copy(self, target, timeout=500):\n\n if self.metadata and 'encoding' in self.metadata:\n with io.open(target,'w', encoding=self.metadata['encoding']) as f:\n for line in self:\n f.write(line)\n else:\n with io.open(target,'wb') as f:\n for line in self:\n if sys.version < '3' and isinstance(line,unicode): #pylint: disable=undefined-variable\n f.write(line.encode('utf-8'))\n elif sys.version >= '3' and isinstance(line,str):\n f.write(line.encode('utf-8'))\n else:\n f.write(line)", "response": "Copy or download this file to a new local file"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nparse CLAM XML and sets attributes of this object.", "response": "def parseresponse(self, xml, localroot = False):\n \"\"\"Parses CLAM XML, there's usually no need to call this directly\"\"\"\n root = parsexmlstring(xml)\n if root.tag != 'clam':\n raise FormatError(\"CLAM root tag not found\")\n\n self.system_id = root.attrib['id']\n self.system_name = root.attrib['name']\n if 'project' in root.attrib:\n self.project = root.attrib['project']\n else:\n self.project = None\n if 'baseurl' in root.attrib:\n self.baseurl = root.attrib['baseurl']\n if self.project:\n if localroot is True: #implicit, assume CWD\n self.remote = False\n self.projecturl = '' #relative directory (assume CWD is project directory, as is the case when wrapper scripts are called)\n elif localroot: #explicit\n self.remote = False\n self.projecturl = localroot + '/' + self.project + '/' #explicit directory\n else:\n self.remote = True #no directory: remote URL\n self.projecturl = self.baseurl + '/' + self.project + '/'\n\n if 'user' in root.attrib:\n self.user = root.attrib['user']\n else:\n self.user = None\n\n self.oauth_access_token = \"\"\n if 'oauth_access_token' in root.attrib:\n self.oauth_access_token = root.attrib['oauth_access_token']\n\n for node in root: #pylint: disable=too-many-nested-blocks\n if node.tag == \"description\":\n self.description = node.text\n elif node.tag == \"version\":\n self.system_version = node.text\n elif node.tag == \"email\":\n self.system_email = node.text\n elif node.tag == 'status':\n self.status = int(node.attrib['code'])\n self.statusmessage = node.attrib['message']\n self.completion = node.attrib['completion']\n if 'errors' in node.attrib:\n self.errors = ((node.attrib['errors'] == 'yes') or (node.attrib['errors'] == '1'))\n if 'errormsg' in node.attrib:\n self.errormsg = node.attrib['errormsg']\n elif node.tag == 'parameters':\n for parametergroupnode in node:\n if parametergroupnode.tag == 'parametergroup':\n parameterlist = []\n for parameternode in parametergroupnode:\n if parameternode.tag in vars(clam.common.parameters):\n parameterlist.append(vars(clam.common.parameters)[parameternode.tag].fromxml(parameternode))\n else:\n raise Exception(\"Expected parameter class '\" + parameternode.tag + \"', but not defined!\")\n self.parameters.append( (parametergroupnode.attrib['name'], parameterlist) )\n elif node.tag == 'corpora':\n for corpusnode in node:\n if corpusnode.tag == 'corpus':\n self.corpora.append(corpusnode.value)\n elif node.tag == 'profiles':\n for subnode in node:\n if subnode.tag == 'profile':\n self.profiles.append(Profile.fromxml(subnode))\n elif node.tag == 'input':\n for filenode in node:\n if filenode.tag == 'file':\n for n in filenode:\n if n.tag == 'name':\n self.input.append( CLAMInputFile( self.projecturl, n.text, self.loadmetadata, self.client,True) )\n elif node.tag == 'output':\n for filenode in node:\n if filenode.tag == 'file':\n for n in filenode:\n if n.tag == 'name':\n self.output.append( CLAMOutputFile( self.projecturl, n.text, self.loadmetadata, self.client ) )\n elif node.tag == 'projects':\n self.projects = []\n for projectnode in node:\n if projectnode.tag == 'project':\n self.projects.append(projectnode.text)\n elif node.tag == 'program':\n self.program = Program(self.projecturl, [ int(i) for i in node.attrib['matchedprofiles'].split(',') ] )\n for outputfilenode in node:\n if outputfilenode.tag == 'outputfile':\n inputfound = False\n for inputfilenode in outputfilenode:\n if inputfilenode.tag == 'inputfile':\n inputfound = True\n self.program.add(outputfilenode.attrib['name'],outputfilenode.attrib['template'],inputfilenode.attrib['name'], inputfilenode.attrib['template'])\n if not inputfound:\n self.program.add(outputfilenode.attrib['name'],outputfilenode.attrib['template'])"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting an output template by ID", "response": "def outputtemplate(self, template_id):\n \"\"\"Get an output template by ID\"\"\"\n for profile in self.profiles:\n for outputtemplate in profile.outputtemplates():\n if outputtemplate.id == template_id:\n return outputtemplate\n return KeyError(\"Outputtemplate \" + template_id + \" not found\")"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nobtaining a string of all parameters using the paramater flags they were defined with in order to pass to an external command. This is shell - safe by definition.", "response": "def commandlineargs(self):\n \"\"\"Obtain a string of all parameters, using the paramater flags they were defined with, in order to pass to an external command. This is shell-safe by definition.\"\"\"\n commandlineargs = []\n for parametergroup, parameters in self.parameters: #pylint: disable=unused-variable\n for parameter in parameters:\n p = parameter.compilearg()\n if p:\n commandlineargs.append(p)\n return \" \".join(commandlineargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the specified global parameter", "response": "def parameter(self, parameter_id):\n \"\"\"Return the specified global parameter (the entire object, not just the value)\"\"\"\n for parametergroup, parameters in self.parameters: #pylint: disable=unused-variable\n for parameter in parameters:\n if parameter.id == parameter_id:\n return parameter\n raise KeyError(\"No such parameter exists: \" + parameter_id )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef parametererror(self):\n for parametergroup, parameters in self.parameters: #pylint: disable=unused-variable\n for parameter in parameters:\n if parameter.error:\n return parameter.error\n return False", "response": "Return the first parameter error"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns all parameters as a dictionary", "response": "def passparameters(self):\n \"\"\"Return all parameters as {id: value} dictionary\"\"\"\n paramdict = {}\n for parametergroup, params in self.parameters: #pylint: disable=unused-variable\n for parameter in params:\n if parameter.value:\n if isinstance(parameter.value, list):\n paramdict[parameter.id] = \",\".join(parameter.value)\n else:\n paramdict[parameter.id] = parameter.value\n return paramdict"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef inputtemplates(self):\n l = []\n for profile in self.profiles:\n l += profile.input\n return l", "response": "Return all input templates as a list of InputTemplate instances"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the inputtemplate with the specified ID. This is used to resolve a inputtemplate ID to an InputTemplate object instance instance.", "response": "def inputtemplate(self,template_id):\n \"\"\"Return the inputtemplate with the specified ID. This is used to resolve a inputtemplate ID to an InputTemplate object instance\"\"\"\n for profile in self.profiles:\n for inputtemplate in profile.input:\n if inputtemplate.id == template_id:\n return inputtemplate\n raise Exception(\"No such input template: \" + repr(template_id))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef inputfile(self, inputtemplate=None):\n inputfiles = list(self.inputfiles(inputtemplate))\n if len(inputfiles) < 1:\n raise Exception(\"No such input file\")\n elif len(inputfiles) > 1:\n raise Exception(\"Multiple input files matched. Use inputfiles() instead!\")\n return inputfiles[0]", "response": "Return the inputfile for the specified inputtemplate if inputtemplate = None inputfile is returned regardless of inputtemplate."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef inputfiles(self, inputtemplate=None):\n if isinstance(inputtemplate, InputTemplate):\n #ID suffices:\n inputtemplate = inputtemplate.id\n for inputfile in self.input:\n if not inputtemplate or inputfile.metadata.inputtemplate == inputtemplate:\n yield inputfile", "response": "Generator yielding all inputfiles for the specified inputtemplate."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef match(self, projectpath, parameters):\n parameters = sanitizeparameters(parameters)\n\n mandatory_absent = [] #list of input templates that are missing but mandatory\n optional_absent = [] #list of absent but optional input templates\n\n #check if profile matches inputdata (if there are no inputtemplate, this always matches intentionally!)\n for inputtemplate in self.input:\n if not inputtemplate.matchingfiles(projectpath):\n if inputtemplate.optional:\n optional_absent.append(inputtemplate)\n else:\n mandatory_absent.append(inputtemplate)\n\n if mandatory_absent:\n return False, mandatory_absent\n\n #check if output is produced\n if not self.output: return False, mandatory_absent\n for outputtemplate in self.output:\n if isinstance(outputtemplate, ParameterCondition):\n outputtemplate = outputtemplate.evaluate(parameters)\n if not outputtemplate:\n continue\n assert isinstance(outputtemplate, OutputTemplate)\n if outputtemplate.parent:\n if outputtemplate.getparent(self) not in optional_absent: #pylint: disable=protected-access\n return True, optional_absent\n else:\n return True, optional_absent\n\n return False, optional_absent", "response": "Check if the profile matches all inputdata and produces output given the set parameters. Returns a boolean"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a list of all inputfiles matching the profile", "response": "def matchingfiles(self, projectpath):\n \"\"\"Return a list of all inputfiles matching the profile (filenames)\"\"\"\n l = []\n for inputtemplate in self.input:\n l += inputtemplate.matchingfiles(projectpath)\n return l"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns all outputtemplates resolving ParameterConditions to all possibilities", "response": "def outputtemplates(self):\n \"\"\"Returns all outputtemplates, resolving ParameterConditions to all possibilities\"\"\"\n outputtemplates = []\n for o in self.output:\n if isinstance(o, ParameterCondition):\n outputtemplates += o.allpossibilities()\n else:\n assert isinstance(o, OutputTemplate)\n outputtemplates.append(o)\n return outputtemplates"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef generate(self, projectpath, parameters, serviceid, servicename,serviceurl):\n\n #Make dictionary of parameters\n parameters = sanitizeparameters(parameters)\n\n program = Program(projectpath, [self])\n\n match, optional_absent = self.match(projectpath, parameters) #Does the profile match?\n if match: #pylint: disable=too-many-nested-blocks\n\n #gather all input files that match\n inputfiles = self.matchingfiles(projectpath) #list of (seqnr, filename,inputtemplate) tuples\n\n inputfiles_full = [] #We need the full CLAMInputFiles for generating provenance data\n for seqnr, filename, inputtemplate in inputfiles: #pylint: disable=unused-variable\n inputfiles_full.append(CLAMInputFile(projectpath, filename))\n\n for outputtemplate in self.output:\n if isinstance(outputtemplate, ParameterCondition):\n outputtemplate = outputtemplate.evaluate(parameters)\n\n #generate output files\n if outputtemplate:\n if isinstance(outputtemplate, OutputTemplate):\n #generate provenance data\n provenancedata = CLAMProvenanceData(serviceid,servicename,serviceurl,outputtemplate.id, outputtemplate.label, inputfiles_full, parameters)\n\n create = True\n if outputtemplate.parent:\n if outputtemplate.getparent(self) in optional_absent:\n create = False\n\n if create:\n for inputtemplate, inputfilename, outputfilename, metadata in outputtemplate.generate(self, parameters, projectpath, inputfiles, provenancedata):\n clam.common.util.printdebug(\"Writing metadata for outputfile \" + outputfilename)\n metafilename = os.path.dirname(outputfilename)\n if metafilename: metafilename += '/'\n metafilename += '.' + os.path.basename(outputfilename) + '.METADATA'\n f = io.open(projectpath + '/output/' + metafilename,'w',encoding='utf-8')\n f.write(metadata.xml())\n f.close()\n program.add(outputfilename, outputtemplate, inputfilename, inputtemplate)\n else:\n raise TypeError(\"OutputTemplate expected, but got \" + outputtemplate.__class__.__name__)\n\n return program", "response": "Generate output metadata on the basis of input files and parameters. Projectpath must be absolute. Returns a Program instance."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nproduce XML output for the profile", "response": "def xml(self, indent = \"\"):\n \"\"\"Produce XML output for the profile\"\"\"\n xml = \"\\n\" + indent + \"\\n\"\n xml += indent + \" \\n\"\n for inputtemplate in self.input:\n xml += inputtemplate.xml(indent +\" \") + \"\\n\"\n xml += indent + \" \\n\"\n xml += indent + \" \\n\"\n for outputtemplate in self.output:\n xml += outputtemplate.xml(indent +\" \") + \"\\n\" #works for ParameterCondition as well!\n xml += indent + \" \\n\"\n xml += indent + \"\\n\"\n return xml"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a profile instance from the given XML description. Node can be a string or an etree. _Element.", "response": "def fromxml(node):\n \"\"\"Return a profile instance from the given XML description. Node can be a string or an etree._Element.\"\"\"\n if not isinstance(node,ElementTree._Element): #pylint: disable=protected-access\n node = parsexmlstring(node)\n\n args = []\n\n if node.tag == 'profile':\n for node in node:\n if node.tag == 'input':\n for subnode in node:\n if subnode.tag.lower() == 'inputtemplate':\n args.append(InputTemplate.fromxml(subnode))\n elif node.tag == 'output':\n for subnode in node:\n if subnode.tag.lower() == 'outputtemplate':\n args.append(OutputTemplate.fromxml(subnode))\n elif subnode.tag.lower() == 'parametercondition':\n args.append(ParameterCondition.fromxml(subnode))\n return Profile(*args)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd a new path to the program", "response": "def add(self, outputfilename, outputtemplate, inputfilename=None, inputtemplate=None):\n \"\"\"Add a new path to the program\"\"\"\n if isinstance(outputtemplate,OutputTemplate):\n outputtemplate = outputtemplate.id\n if isinstance(inputtemplate,InputTemplate):\n inputtemplate = inputtemplate.id\n if outputfilename in self:\n outputtemplate, inputfiles = self[outputfilename]\n if inputfilename and inputtemplate:\n inputfiles[inputfilename] = inputtemplate\n else:\n if inputfilename and inputtemplate:\n self[outputfilename] = (outputtemplate, {inputfilename: inputtemplate})\n else:\n self[outputfilename] = (outputtemplate, {})"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef getoutputfiles(self, loadmetadata=True, client=None,requiremetadata=False):\n for outputfilename, outputtemplate in self.outputpairs():\n yield CLAMOutputFile(self.projectpath, outputfilename, loadmetadata,client,requiremetadata), outputtemplate", "response": "Iterates over all output files and their output templates."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\niterate over all input files for the specified outputfile.", "response": "def getinputfiles(self, outputfile, loadmetadata=True, client=None,requiremetadata=False):\n \"\"\"Iterates over all input files for the specified outputfile (you may pass a CLAMOutputFile instance or a filename string). Yields (CLAMInputFile,str:inputtemplate_id) tuples. The last three arguments are passed to its constructor.\"\"\"\n if isinstance(outputfile, CLAMOutputFile):\n outputfilename = str(outputfile).replace(os.path.join(self.projectpath,'output/'),'')\n else:\n outputfilename = outputfile\n outputtemplate, inputfiles = self[outputfilename]\n for inputfilename, inputtemplate in inputfiles.items():\n yield CLAMInputFile(self.projectpath, inputfilename, loadmetadata,client,requiremetadata), inputtemplate"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets one input file for the specified output filename.", "response": "def getinputfile(self, outputfile, loadmetadata=True, client=None,requiremetadata=False):\n \"\"\"Grabs one input file for the specified output filename (raises a KeyError exception if there is no such output, StopIteration if there are no input files for it). Shortcut for getinputfiles()\"\"\"\n if isinstance(outputfile, CLAMOutputFile):\n outputfilename = str(outputfile).replace(os.path.join(self.projectpath,'output/'),'')\n else:\n outputfilename = outputfile\n if outputfilename not in self:\n raise KeyError(\"No such outputfile \" + outputfilename)\n try:\n return next(self.getinputfiles(outputfile,loadmetadata,client,requiremetadata))\n except StopIteration:\n raise StopIteration(\"No input files for outputfile \" + outputfilename)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef getoutputfile(self, loadmetadata=True, client=None,requiremetadata=False):\n return next(self.getoutputfiles(loadmetadata,client,requiremetadata))", "response": "Grabs one output file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nserialise provenance data to XML. This is included in CLAM Metadata files.", "response": "def xml(self, indent = \"\"):\n \"\"\"Serialise provenance data to XML. This is included in CLAM Metadata files\"\"\"\n xml = indent + \"\"\n for filename, metadata in self.inputfiles:\n xml += indent + \" \"\n xml += metadata.xml(indent + \" \") + \"\\n\"\n xml += indent + \" \\n\"\n if self.parameters:\n xml += indent + \" \\n\"\n if isinstance(self.parameters, dict):\n parameters = self.parameters.values()\n elif isinstance(self.parameters, list):\n parameters = self.parameters\n for parameter in parameters:\n xml += parameter.xml(indent +\" \") + \"\\n\"\n xml += indent + \" \\n\"\n xml += indent + \"\"\n return xml"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef fromxml(node):\n if not isinstance(node,ElementTree._Element): #pylint: disable=protected-access\n node = parsexmlstring(node)\n if node.tag == 'provenance': #pylint: disable=too-many-nested-blocks\n if node.attrib['type'] == 'clam':\n serviceid = node.attrib['id']\n servicename = node.attrib['name']\n serviceurl = node.attrib['url']\n timestamp = node.attrib['timestamp']\n outputtemplate = node.attrib['outputtemplate']\n outputtemplatelabel = node.attrib['outputtemplatelabel']\n inputfiles = []\n parameters = []\n for subnode in node:\n if subnode.tag == 'inputfile':\n filename = node.attrib['name']\n metadata = None\n for subsubnode in subnode:\n if subsubnode.tag == 'CLAMMetaData':\n metadata = CLAMMetaData.fromxml(subsubnode)\n break\n inputfiles.append( (filename, metadata) )\n elif subnode.tag == 'parameters':\n for subsubnode in subnode:\n if subsubnode.tag in vars(clam.common.parameters):\n parameters.append(vars(clam.common.parameters)[subsubnode.tag].fromxml(subsubnode))\n else:\n raise Exception(\"Expected parameter class '\" + subsubnode.tag + \"', but not defined!\")\n return CLAMProvenanceData(serviceid,servicename,serviceurl,outputtemplate, outputtemplatelabel, inputfiles, parameters, timestamp)\n else:\n raise NotImplementedError", "response": "Return a CLAMProvenanceData instance from the given XML description. Node can be a string or an lxml. etree. _Element."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nrenders an XML representation of the metadata", "response": "def xml(self, indent = \"\"):\n \"\"\"Render an XML representation of the metadata\"\"\" #(independent of web.py for support in CLAM API)\n if not indent:\n xml = '\\n'\n else:\n xml = \"\"\n xml += indent + \"\\n\"\n\n for key, value in self.data.items():\n xml += indent + \" \"+clam.common.util.xmlescape(str(value))+\"\\n\"\n\n if self.provenance:\n xml += self.provenance.xml(indent + \" \")\n\n xml += indent + \"\"\n return xml"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef save(self, filename):\n with io.open(filename,'w',encoding='utf-8') as f:\n f.write(self.xml())", "response": "Save the current metadata to a file"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreading metadata from XML. Static method returning an CLAMMetaData instance (or rather; the appropriate subclass of CLAMMetaData) from the given XML description. Node can be a string or an etree._Element.", "response": "def fromxml(node, file=None):\n \"\"\"Read metadata from XML. Static method returning an CLAMMetaData instance (or rather; the appropriate subclass of CLAMMetaData) from the given XML description. Node can be a string or an etree._Element.\"\"\"\n if not isinstance(node,ElementTree._Element): #pylint: disable=protected-access\n node = parsexmlstring(node)\n if node.tag == 'CLAMMetaData':\n dataformat = node.attrib['format']\n\n formatclass = None\n for C in CUSTOM_FORMATS: #CUSTOM_FORMATS will be injected by clamservice.py\n if C.__name__ == dataformat:\n formatclass = C\n break\n if formatclass is None and dataformat in vars(clam.common.formats) and issubclass(vars(clam.common.formats)[dataformat], CLAMMetaData):\n formatclass = vars(clam.common.formats)[dataformat]\n if formatclass is None:\n raise Exception(\"Format class \" + dataformat + \" not found!\")\n\n data = {}\n if 'inputtemplate' in node.attrib:\n data['inputtemplate'] = node.attrib['inputtemplate']\n if 'inputtemplatelabel' in node.attrib:\n data['inputtemplatelabel'] = node.attrib['inputtemplatelabel']\n\n\n\n for subnode in node:\n if subnode.tag == 'meta':\n key = subnode.attrib['id']\n value = subnode.text\n data[key] = value\n elif subnode.tag == 'provenance':\n data['provenance'] = CLAMProvenanceData.fromxml(subnode)\n return formatclass(file, **data)\n else:\n raise Exception(\"Invalid CLAM Metadata!\")"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nproduces XML representation of the object.", "response": "def xml(self, indent = \"\"):\n \"\"\"Produce Template XML\"\"\"\n xml = indent + \"\\n\"\n for parameter in self.parameters:\n xml += parameter.xml(indent+\" \") + \"\\n\"\n if self.converters:\n for converter in self.converters:\n xml += indent + \" \"+clam.common.util.xmlescape(converter.label)+\"\\n\"\n if self.inputsources:\n for inputsource in self.inputsources:\n xml += inputsource.xml(indent+\" \")\n xml += indent + \"\"\n return xml"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef json(self):\n d = { 'id': self.id, 'format': self.formatclass.__name__,'label': self.label, 'mimetype': self.formatclass.mimetype, 'schema': self.formatclass.schema }\n if self.unique:\n d['unique'] = True\n if self.filename:\n d['filename'] = self.filename\n if self.extension:\n d['extension'] = self.extension\n if self.acceptarchive:\n d['acceptarchive'] = self.acceptarchive\n #d['parameters'] = {}\n\n #The actual parameters are included as XML, and transformed by clam.js using XSLT (parameter.xsl) to generate the forms\n parametersxml = ''\n for parameter in self.parameters:\n parametersxml += parameter.xml()\n d['parametersxml'] = '' + parametersxml + ''\n d['converters'] = [ {'id':x.id, 'label':x.label} for x in self.converters ]\n d['inputsources'] = [ {'id':x.id, 'label':x.label} for x in self.inputsources ]\n\n return json.dumps(d)", "response": "Produce a JSON representation of the web interface"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef match(self, metadata, user = None):\n assert isinstance(metadata, self.formatclass)\n return self.generate(metadata,user)", "response": "Does the specified metadata match this template? returns success metadata and parameters"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef matchingfiles(self, projectpath):\n results = []\n\n if projectpath[-1] == '/':\n inputpath = projectpath + 'input/'\n else:\n inputpath = projectpath + '/input/'\n\n for linkf,realf in clam.common.util.globsymlinks(inputpath + '/.*.INPUTTEMPLATE.' + self.id + '.*'):\n seqnr = int(linkf.split('.')[-1])\n results.append( (seqnr, realf[len(inputpath):], self) )\n results = sorted(results)\n if self.unique and len(results) != 1:\n return []\n else:\n return results", "response": "Checks if the input conditions are satisfied. Returns a list of matching results."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nvalidate the posted data against the inputtemplate", "response": "def validate(self, postdata, user = None):\n \"\"\"Validate posted data against the inputtemplate\"\"\"\n clam.common.util.printdebug(\"Validating inputtemplate \" + self.id + \"...\")\n errors, parameters, _ = processparameters(postdata, self.parameters, user)\n return errors, parameters"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert the template into instantiated metadata validating the data in the process and returning errors otherwise.", "response": "def generate(self, file, validatedata = None, inputdata=None, user = None):\n \"\"\"Convert the template into instantiated metadata, validating the data in the process and returning errors otherwise. inputdata is a dictionary-compatible structure, such as the relevant postdata. Return (success, metadata, parameters), error messages can be extracted from parameters[].error. Validatedata is a (errors,parameters) tuple that can be passed if you did validation in a prior stage, if not specified, it will be done automatically.\"\"\"\n\n metadata = {}\n\n if not validatedata:\n assert inputdata\n errors, parameters = self.validate(inputdata,user) #pylint: disable=unused-variable\n else:\n errors, parameters = validatedata #pylint: disable=unused-variable\n\n #scan errors and set metadata\n success = True\n for parameter in parameters:\n assert isinstance(parameter, clam.common.parameters.AbstractParameter)\n if parameter.error:\n success = False\n else:\n metadata[parameter.id] = parameter.value\n\n if not success:\n metadata = None\n else:\n try:\n metadata = self.formatclass(file, **metadata)\n except:\n raise\n\n return success, metadata, parameters"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef xml(self, operator='set', indent = \"\"):\n xml = indent + \"\"\n else:\n xml += \">\" + self.value + \"\"\n return xml", "response": "Serialize the metadata field to XML"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fromxml(node):\n if not isinstance(node,ElementTree._Element): #pylint: disable=protected-access\n node = parsexmlstring(node)\n if node.tag.lower() != 'meta':\n raise Exception(\"Expected meta tag but got '\" + node.tag + \"' instead\")\n\n key = node.attrib['id']\n if node.text:\n value = node.text\n else:\n value = None\n operator = 'set'\n if 'operator' in node.attrib:\n operator= node.attrib['operator']\n if operator == 'set':\n cls = SetMetaField\n elif operator == 'unset':\n cls = UnsetMetaField\n elif operator == 'copy':\n cls = CopyMetaField\n elif operator == 'parameter':\n cls = ParameterMetaField\n return cls(key, value)", "response": "Static method returning an instance of MetaField from the given XML description. Node can be a string or an etree. _Element object."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfinding the most suitable parent for this inputtemplate", "response": "def findparent(self, inputtemplates):\n \"\"\"Find the most suitable parent, that is: the first matching unique/multi inputtemplate\"\"\"\n for inputtemplate in inputtemplates:\n if self.unique == inputtemplate.unique:\n return inputtemplate.id\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nresolves a parent ID", "response": "def getparent(self, profile):\n \"\"\"Resolve a parent ID\"\"\"\n assert self.parent\n for inputtemplate in profile.input:\n if inputtemplate == self.parent:\n return inputtemplate\n raise Exception(\"Parent InputTemplate '\"+self.parent+\"' not found!\")"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef generate(self, profile, parameters, projectpath, inputfiles, provenancedata=None):\n\n project = os.path.basename(projectpath)\n\n if self.parent: #pylint: disable=too-many-nested-blocks\n #We have a parent, infer the correct filename\n\n #copy filename from parent\n parent = self.getparent(profile)\n\n #get input files for the parent InputTemplate\n parentinputfiles = parent.matchingfiles(projectpath)\n if not parentinputfiles:\n raise Exception(\"OutputTemplate '\"+self.id + \"' has parent '\" + self.parent + \"', but no matching input files were found!\")\n\n #Do we specify a full filename?\n for seqnr, inputfilename, inputtemplate in parentinputfiles: #pylint: disable=unused-variable\n\n if self.filename:\n filename = self.filename\n parentfile = CLAMInputFile(projectpath, inputfilename)\n elif parent:\n filename = inputfilename\n parentfile = CLAMInputFile(projectpath, inputfilename)\n else:\n raise Exception(\"OutputTemplate '\"+self.id + \"' has no parent nor filename defined!\")\n\n #Make actual CLAMInputFile objects of ALL relevant input files, that is: all unique=True files and all unique=False files with the same sequence number\n relevantinputfiles = []\n for seqnr2, inputfilename2, inputtemplate2 in inputfiles:\n if seqnr2 == 0 or seqnr2 == seqnr:\n relevantinputfiles.append( (inputtemplate2, CLAMInputFile(projectpath, inputfilename2)) )\n\n #resolve # in filename (done later)\n #if not self.unique:\n # filename.replace('#',str(seqnr))\n\n if not self.filename and self.removeextensions:\n #Remove unwanted extensions\n if self.removeextensions is True:\n #Remove any and all extensions\n filename = filename.split('.')[0]\n elif isinstance(self.removeextensions, list):\n #Remove specified extension\n for ext in self.removeextensions:\n if ext:\n if ext[0] != '.' and filename[-len(ext) - 1:] == '.' + ext:\n filename = filename[:-len(ext) - 1]\n elif ext[0] == '.' and filename[-len(ext):] == ext:\n filename = filename[:-len(ext)]\n\n\n if self.extension and not self.filename and filename[-len(self.extension) - 1:] != '.' + self.extension: #(also prevents duplicate extensions)\n filename += '.' + self.extension\n\n\n #Now we create the actual metadata\n metadata = self.generatemetadata(parameters, parentfile, relevantinputfiles, provenancedata)\n\n #Resolve filename\n filename = resolveoutputfilename(filename, parameters, metadata, self, seqnr, project, inputfilename)\n\n yield inputtemplate, inputfilename, filename, metadata\n\n elif self.unique and self.filename:\n #outputtemplate has no parent, but specified a filename and is unique, this implies it is not dependent on input files:\n\n metadata = self.generatemetadata(parameters, None, [], provenancedata)\n\n filename = resolveoutputfilename(self.filename, parameters, metadata, self, 0, project, None)\n\n yield None, None, filename, metadata\n else:\n raise Exception(\"Unable to generate from OutputTemplate, no parent or filename specified\")", "response": "Generate all relevant input files for this output template."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef generatemetadata(self, parameters, parentfile, relevantinputfiles, provenancedata = None):\n assert isinstance(provenancedata,CLAMProvenanceData) or provenancedata is None\n\n data = {}\n\n if self.copymetadata:\n #Copy parent metadata\n for key, value in parentfile.metadata.items():\n data[key] = value\n\n for metafield in self.metafields:\n if isinstance(metafield, ParameterCondition):\n metafield = metafield.evaluate(parameters)\n if not metafield:\n continue\n assert isinstance(metafield, AbstractMetaField)\n metafield.resolve(data, parameters, parentfile, relevantinputfiles)\n\n if provenancedata:\n data['provenance'] = provenancedata\n\n return self.formatclass(None, **data)", "response": "Generate metadata given a filename parameters and a dictionary of inputdata"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn all possible outputtemplates that may occur ( recusrively applied", "response": "def allpossibilities(self):\n \"\"\"Returns all possible outputtemplates that may occur (recusrively applied)\"\"\"\n l = []\n if isinstance(self.then, ParameterCondition):\n #recursive parametercondition\n l += self.then.allpossibilities()\n elif self.then:\n l.append(self.then)\n if self.otherwise:\n if isinstance(self.otherwise, ParameterCondition):\n l += self.otherwise.allpossibilities()\n else:\n l.append(self.otherwise)\n return l"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef evaluate(self, parameters):\n if self.match(parameters):\n if isinstance(self.then, ParameterCondition):\n #recursive parametercondition\n return self.then.evaluate(parameters)\n else:\n return self.then\n elif self.otherwise:\n if isinstance(self.otherwise, ParameterCondition):\n #recursive else\n return self.otherwise.evaluate(parameters)\n else:\n return self.otherwise\n return False", "response": "Evaluates the condition and returns True if there s no match or whatever the ParameterCondition evaluates to ( recursively applied!"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fromxml(node):\n if not isinstance(node,ElementTree._Element): #pylint: disable=protected-access\n node = parsexmlstring(node)\n assert node.tag.lower() == 'action'\n\n kwargs = {}\n args = []\n if 'id' in node.attrib:\n kwargs['id'] = node.attrib['id']\n elif 'name' in node.attrib:\n kwargs['name'] = node.attrib['name']\n elif 'description' in node.attrib:\n kwargs['description'] = node.attrib['description']\n elif 'method' in node.attrib:\n kwargs['method'] = node.attrib['method']\n elif 'mimetype' in node.attrib:\n kwargs['mimetype'] = node.attrib['mimetype']\n elif 'allowanonymous' in node.attrib:\n if node.attrib['allowanonymous'] == \"yes\":\n kwargs['allowanonymous'] = True\n\n found = False\n for subnode in node:\n if subnode.tag.lower() == 'parametercondition':\n kwargs[node.tag] = ParameterCondition.fromxml(subnode)\n elif subnode.tag in vars(clam.common.parameters):\n args.append(vars(clam.common.parameters)[subnode.tag].fromxml(subnode))\n if not found:\n raise Exception(\"No condition found in ParameterCondition!\")\n return Action(*args, **kwargs)", "response": "Static method returning an Action instance from the given XML description. Node can be a string or an etree. _Element."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef crude_tokenizer(line):\n tokens = []\n buffer = ''\n for c in line.strip():\n if c == ' ' or c in string.punctuation:\n if buffer:\n tokens.append(buffer)\n buffer = ''\n else:\n buffer += c\n if buffer: tokens.append(buffer)\n return tokens", "response": "This is a very crude tokenizer from pynlpl"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef dicttotext(d, sort=False, max = 0):\n if sort:\n f = lambda x: sorted(x, key=lambda y: -1 * y[1])\n else:\n f = lambda x: x\n\n output = \"\"\n for i, (key, value) in enumerate(f(d.items())):\n output += key + \"\\t\" + str(value) + \"\\n\"\n if max != 0 and i >= max:\n break\n return output", "response": "Function for converting dictionary to plaintext output optionally with some sort"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ninitialising authentication for internal use", "response": "def initauth(self):\n \"\"\"Initialise authentication, for internal use\"\"\"\n headers = {'User-agent': 'CLAMClientAPI-' + clam.common.data.VERSION}\n if self.oauth:\n if not self.oauth_access_token:\n r = requests.get(self.url,headers=headers, verify=self.verify)\n if r.status_code == 404:\n raise clam.common.data.NotFound(\"Authorization provider not found\")\n elif r.status_code == 403:\n raise clam.common.data.PermissionDenied(\"Authorization provider denies access\")\n elif not (r.status_code >= 200 and r.status_code <= 299):\n raise Exception(\"An error occured, return code \" + str(r.status_code))\n\n\n data = self._parse(r.text)\n if data is True: #indicates failure\n raise Exception(\"No access token provided, but Authorization Provider requires manual user input. Unable to authenticate automatically. Obtain an access token from \" + r.geturl())\n else:\n self.oauth_access_token = data.oauth_access_token\n\n headers['Authorization'] = 'Bearer ' + self.oauth_access_token\n return headers"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef request(self, url='', method = 'GET', data = None, parse=True, encoding=None):\n\n requestparams = self.initrequest(data)\n\n\n if method == 'POST':\n request = requests.post\n elif method == 'DELETE':\n request = requests.delete\n elif method == 'PUT':\n request = requests.put\n else:\n request = requests.get\n\n r = request(self.url + url,**requestparams)\n if encoding is not None:\n r.encoding = encoding\n\n if r.status_code == 400:\n raise clam.common.data.BadRequest()\n elif r.status_code == 401:\n raise clam.common.data.AuthRequired()\n elif r.status_code == 403: #pylint: disable=too-many-nested-blocks\n content = r.text\n if parse:\n data = self._parse(content)\n if data:\n if data.errors:\n error = data.parametererror()\n #print(\"DEBUG: parametererror=\" + str(error),file=sys.stderr)\n #for parametergroup, parameters in data.parameters: #pylint: disable=unused-variable\n # for parameter in parameters:\n # print(\"DEBUG: \", parameter.id, parameter.error,file=sys.stderr)\n if error:\n raise clam.common.data.ParameterError(error)\n #print(content,file=sys.stderr)\n raise clam.common.data.PermissionDenied(data)\n else:\n raise clam.common.data.PermissionDenied(content)\n else:\n raise clam.common.data.PermissionDenied(content)\n elif r.status_code == 404 and data:\n raise clam.common.data.NotFound(r.text)\n elif r.status_code == 500:\n raise clam.common.data.ServerError(r.text)\n elif r.status_code == 405:\n raise clam.common.data.ServerError(\"Server returned 405: Method not allowed for \" + method + \" on \" + self.url + url)\n elif r.status_code == 408:\n raise clam.common.data.TimeOut()\n elif not (r.status_code >= 200 and r.status_code <= 299):\n raise Exception(\"An error occured, return code \" + str(r.status_code))\n\n if parse:\n return self._parse(r.text)\n else:\n return r.text", "response": "Issue a HTTP request and parse the response."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _parse(self, content):\n if content.find('= 200 and r.status_code <= 299):\n raise Exception(\"An error occured, return code \" + str(r.status_code))\n\n return self._parseupload(r.text)", "response": "Adds an input file to the CLAM service."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nadds an input file to the CLAM service.", "response": "def addinput(self, project, inputtemplate, contents, **kwargs):\n \"\"\"Add an input file to the CLAM service. Explictly providing the contents as a string. This is not suitable for large files as the contents are kept in memory! Use ``addinputfile()`` instead for large files.\n\n project - the ID of the project you want to add the file to.\n inputtemplate - The input template you want to use to add this file (InputTemplate instance)\n contents - The contents for the file to add (string)\n\n Keyword arguments:\n * filename - the filename on the server (mandatory!)\n * metadata - A metadata object.\n * metafile - A metadata file (filename)\n\n Any other keyword arguments will be passed as metadata and matched with the input template's parameters.\n\n Example::\n\n client.addinput(\"myproject\", \"someinputtemplate\", \"This is a test.\", filename=\"test.txt\")\n\n With metadata, assuming such metadata parameters are defined::\n\n client.addinput(\"myproject\", \"someinputtemplate\", \"This is a test.\", filename=\"test.txt\", parameter1=\"blah\", parameterX=3.5))\n\n \"\"\"\n if isinstance( inputtemplate, str) or (sys.version < '3' and isinstance( inputtemplate, unicode)): #pylint: disable=undefined-variable\n data = self.get(project) #causes an extra query to server\n inputtemplate = data.inputtemplate(inputtemplate)\n elif not isinstance(inputtemplate, clam.common.data.InputTemplate):\n raise Exception(\"inputtemplate must be instance of InputTemplate. Get from CLAMData.inputtemplate(id)\")\n\n\n if 'filename' in kwargs:\n filename = self.getinputfilename(inputtemplate, kwargs['filename'])\n else:\n raise Exception(\"No filename provided!\")\n\n data = {\"contents\": contents, 'inputtemplate': inputtemplate.id}\n for key, value in kwargs.items():\n if key == 'filename':\n pass #nothing to do\n elif key == 'metadata':\n assert isinstance(value, clam.common.data.CLAMMetaData)\n data['metadata'] = value.xml()\n elif key == 'metafile':\n data['metafile'] = open(value,'r')\n else:\n data[key] = value\n\n\n requestparams = self.initrequest(data)\n r = requests.post(self.url + project + '/input/' + filename,**requestparams)\n\n if r.status_code == 400:\n raise clam.common.data.BadRequest()\n elif r.status_code == 401:\n raise clam.common.data.AuthRequired()\n elif r.status_code == 403:\n if r.text[0] == '<':\n #XML response\n return self._parseupload(r.text)\n else:\n raise clam.common.data.PermissionDenied(r.text)\n elif r.status_code == 404:\n raise clam.common.data.NotFound(r.text)\n elif r.status_code == 500:\n raise clam.common.data.ServerError(r.text)\n elif r.status_code == 405:\n raise clam.common.data.ServerError(\"Server returned 405: Method not allowed for POST on \" + self.url + project + '/input/' + filename)\n elif r.status_code == 408:\n raise clam.common.data.TimeOut()\n elif not (r.status_code >= 200 and r.status_code <= 299):\n raise Exception(\"An error occured, return code \" + str(r.status_code))\n\n return self._parseupload(r.text)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndownload an output file from the specified project to the specified target filename.", "response": "def download(self, project, filename, targetfilename, loadmetadata=None):\n \"\"\"Download an output file\"\"\"\n if loadmetadata is None: loadmetadata = self.loadmetadata\n f = clam.common.data.CLAMOutputFile(self.url + project, filename, loadmetadata, self)\n f.copy(targetfilename)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef set(self, value):\n if self.validate(value):\n #print \"Parameter \" + self.id + \" successfully set to \" + repr(value)\n self.hasvalue = True\n self.value = value\n return True\n else:\n #print \"Parameter \" + self.id + \" COULD NOT BE set to \" + repr(value)\n return False", "response": "This method attempts to set the value for this parameter."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef fromxml(node):\n if not isinstance(node,ElementTree._Element): #pylint: disable=protected-access\n node = ElementTree.parse(StringIO(node)).getroot()\n if node.tag in globals():\n id = ''\n paramflag = ''\n name = ''\n description = ''\n kwargs = {}\n error = None\n for attrib, value in node.attrib.items():\n if attrib == 'id':\n id = value\n elif attrib == 'paramflag':\n paramflag = value\n elif attrib == 'name':\n name = value\n elif attrib == 'description':\n description = value\n elif attrib == 'error':\n error = value\n else:\n kwargs[attrib] = value\n\n #extra parsing for choice parameter (TODO: put in a better spot)\n if 'multi' in kwargs and (kwargs['multi'] == 'yes' or kwargs['multi'] == '1' or kwargs['multi'] == 'true'):\n kwargs['value'] = []\n\n for subtag in node: #parse possible subtags\n if subtag.tag == 'choice': #extra parsing for choice parameter (TODO: put in a better spot)\n if 'choices' not in kwargs: kwargs['choices'] = {}\n kwargs['choices'][subtag.attrib['id']] = subtag.text\n if 'selected' in subtag.attrib and (subtag.attrib['selected'] == '1' or subtag.attrib['selected'] == 'yes'):\n if 'multi' in kwargs and (kwargs['multi'] == 'yes' or kwargs['multi'] == '1' or kwargs['multi'] == 'true'):\n kwargs['value'].append(subtag.attrib['id'])\n else:\n kwargs['value'] = subtag.attrib['id']\n\n parameter = globals()[node.tag](id, name, description, **kwargs) #return parameter instance\n if error:\n parameter.error = error #prevent error from getting reset\n return parameter\n else:\n raise Exception(\"No such parameter exists: \" + node.tag)", "response": "Create a Parameter instance from a XML description. Node can be a string containing XML or an lxml _Element object."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef set(self, value = True):\n value = value in (True,1) or ( (isinstance(value, str) or (sys.version < '3' and isinstance(value, unicode))) and (value.lower() in (\"1\",\"yes\",\"true\",\"enabled\"))) #pylint: disable=undefined-variable\n return super(BooleanParameter,self).set(value)", "response": "Set the boolean parameter value"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef compilearg(self):\n if isinstance(self.value,list):\n value = self.delimiter.join(self.value)\n else:\n value = self.value\n if value.find(\" \") >= 0:\n value = '\"' + value + '\"' #wrap all in quotes\n\n #for some odd, reason this produced an error, as if we're not an instance of choiceparameter\n #return super(ChoiceParameter,self).compilearg(value)\n\n #workaround:\n if self.paramflag and self.paramflag[-1] == '=' or self.nospace:\n sep = ''\n elif self.paramflag:\n sep = ' '\n else:\n return str(value)\n return self.paramflag + sep + str(value)", "response": "This method compiles the parameter into a syntax that can be used on the shell such as - paramflag = value"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef xml(self, indent = \"\"):\n xml = indent + \"<\" + self.__class__.__name__\n xml += ' id=\"'+self.id + '\"'\n xml += ' name=\"'+xmlescape(self.name) + '\"'\n xml += ' description=\"'+xmlescape(self.description) + '\"'\n if self.paramflag:\n xml += ' flag=\"'+self.paramflag + '\"'\n if self.multi:\n xml += ' multi=\"true\"'\n for key, value in self.kwargs.items():\n if key != 'choices' and key != 'default' and key != 'flag' and key != 'paramflag':\n if isinstance(value, bool):\n xml += ' ' + key + '=\"' + str(int(value))+ '\"'\n elif isinstance(value, list):\n xml += ' ' + key + '=\"'+\",\".join(value)+ '\"'\n else:\n xml += ' ' + key + '=\"'+xmlescape(value)+ '\"'\n if self.error:\n xml += ' error=\"'+self.error + '\"'\n xml += \">\"\n for key, value in self.choices:\n if self.value == key or (isinstance(self.value ,list) and key in self.value):\n xml += \" \" + xmlescape(value) + \"\"\n else:\n xml += \" \" + xmlescape(value) + \"\"\n xml += \"\"\n return xml", "response": "This methods renders an XML representation of this parameter along with\n its selected value and feedback on validation errors."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef valuefrompostdata(self, postdata):\n if self.multi: #multi parameters can be passed as parameterid=choiceid1,choiceid2 or by setting parameterid[choiceid]=1 (or whatever other non-zero value)\n found = False\n if self.id in postdata:\n found = True\n passedvalues = postdata[self.id].split(',')\n values = []\n for choicekey in [x[0] for x in self.choices]:\n if choicekey in passedvalues:\n found = True\n values.append(choicekey)\n else:\n values = []\n for choicekey in [x[0] for x in self.choices]:\n if self.id+'['+choicekey+']' in postdata:\n found = True\n if postdata[self.id+'['+choicekey+']']:\n values.append(choicekey)\n if not found:\n return None\n else:\n return values\n else:\n if self.id in postdata:\n return postdata[self.id]\n else:\n return None", "response": "This method searches the POST data and retrieves the values it needs."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef valuefrompostdata(self, postdata):\n if self.id in postdata and postdata[self.id] != '':\n return int(postdata[self.id])\n else:\n return None", "response": "This method searches the POST data and returns the value of the attribute."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef set(self, value):\n if self.validate(value):\n #print \"Parameter \" + self.id + \" successfully set to \" + repr(value)\n self.hasvalue = True\n if isinstance(value, float):\n self.value = round(value)\n else:\n self.value = int(value)\n return True\n else:\n #print \"Parameter \" + self.id + \" COULD NOT BE set to \" + repr(value)\n return False", "response": "This method attempts to set the value of the parameter. If validation fails the parameter will be set to int."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget list of projects or shortcut to other functionality", "response": "def index(credentials = None):\n \"\"\"Get list of projects or shortcut to other functionality\"\"\"\n\n #handle shortcut\n shortcutresponse = entryshortcut(credentials)\n if shortcutresponse is not None:\n return shortcutresponse\n\n projects = []\n user, oauth_access_token = parsecredentials(credentials)\n totalsize = 0.0\n if settings.LISTPROJECTS:\n projects, totalsize = getprojects(user)\n\n errors = \"no\"\n errormsg = \"\"\n\n corpora = CLAMService.corpusindex()\n\n #pylint: disable=bad-continuation\n return withheaders(flask.make_response(flask.render_template('response.xml',\n version=VERSION,\n system_id=settings.SYSTEM_ID,\n system_name=settings.SYSTEM_NAME,\n system_description=settings.SYSTEM_DESCRIPTION,\n system_author=settings.SYSTEM_AUTHOR,\n system_version=settings.SYSTEM_VERSION,\n system_email=settings.SYSTEM_EMAIL,\n user=user,\n project=None,\n url=getrooturl(),\n statuscode=-1,\n statusmessage=\"\",\n statuslog=[],\n completion=0,\n errors=errors,\n errormsg=errormsg,\n parameterdata=settings.PARAMETERS,\n inputsources=corpora,\n outputpaths=None,\n inputpaths=None,\n profiles=settings.PROFILES,\n datafile=None,\n projects=projects,\n totalsize=totalsize,\n actions=settings.ACTIONS,\n disableinterface=not settings.ENABLEWEBAPP,\n info=False,\n accesstoken=None,\n interfaceoptions=settings.INTERFACEOPTIONS,\n customhtml=settings.CUSTOMHTML_INDEX,\n allow_origin=settings.ALLOW_ORIGIN,\n oauth_access_token=oauth_encrypt(oauth_access_token),\n auth_type=auth_type()\n )), headers={'allow_origin':settings.ALLOW_ORIGIN})"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef addfile(project, filename, user, postdata, inputsource=None,returntype='xml'): #pylint: disable=too-many-return-statements\n\n\n def errorresponse(msg, code=403):\n if returntype == 'json':\n return withheaders(flask.make_response(\"{success: false, error: '\" + msg + \"'}\"),'application/json',{'allow_origin': settings.ALLOW_ORIGIN})\n else:\n return withheaders(flask.make_response(msg,403),headers={'allow_origin': settings.ALLOW_ORIGIN}) #(it will have to be explicitly deleted by the client first)\n\n inputtemplate_id = flask.request.headers.get('Inputtemplate','')\n inputtemplate = None\n metadata = None\n\n\n printdebug('Handling addfile, postdata contains fields ' + \",\".join(postdata.keys()) )\n\n if 'inputtemplate' in postdata:\n inputtemplate_id = postdata['inputtemplate']\n\n if inputtemplate_id:\n #An input template must always be provided\n for profile in settings.PROFILES:\n for t in profile.input:\n if t.id == inputtemplate_id:\n inputtemplate = t\n if not inputtemplate:\n #Inputtemplate not found, send 404\n printlog(\"Specified inputtemplate (\" + inputtemplate_id + \") not found!\")\n return withheaders(flask.make_response(\"Specified inputtemplate (\" + inputtemplate_id + \") not found!\",404),headers={'allow_origin': settings.ALLOW_ORIGIN})\n printdebug('Inputtemplate explicitly provided: ' + inputtemplate.id )\n if not inputtemplate:\n #See if an inputtemplate is explicitly specified in the filename\n printdebug('Attempting to determine input template from filename ' + filename )\n if '/' in filename.strip('/'):\n raw = filename.split('/')\n inputtemplate = None\n for profile in settings.PROFILES:\n for it in profile.input:\n if it.id == raw[0]:\n inputtemplate = it\n break\n if inputtemplate:\n filename = raw[1]\n if not inputtemplate:\n #Check if the specified filename can be uniquely associated with an inputtemplate\n for profile in settings.PROFILES:\n for t in profile.input:\n if t.filename == filename:\n if inputtemplate:\n #we found another one, not unique!! reset and break\n inputtemplate = None\n break\n else:\n #good, we found one, don't break cause we want to make sure there is only one\n inputtemplate = t\n if not inputtemplate:\n printlog(\"No inputtemplate specified and filename \" + filename + \" does not uniquely match with any inputtemplate!\")\n return errorresponse(\"No inputtemplate specified nor auto-detected for filename \" + filename + \"!\",404)\n\n\n\n #See if other previously uploaded input files use this inputtemplate\n if inputtemplate.unique:\n nextseq = 0 #unique\n else:\n nextseq = 1 #will hold the next sequence number for this inputtemplate (in multi-mode only)\n\n for seq, inputfile in Project.inputindexbytemplate(project, user, inputtemplate): #pylint: disable=unused-variable\n if inputtemplate.unique:\n return errorresponse(\"You have already submitted a file of this type, you can only submit one. Delete it first. (Inputtemplate=\" + inputtemplate.id + \", unique=True)\")\n else:\n if seq >= nextseq:\n nextseq = seq + 1 #next available sequence number\n\n\n if not filename: #Actually, I don't think this can occur at this stage, but we'll leave it in to be sure (yes it can, when the entry shortcut is used!)\n if inputtemplate.filename:\n filename = inputtemplate.filename\n elif inputtemplate.extension:\n filename = str(nextseq) +'-' + str(\"%034x\" % random.getrandbits(128)) + '.' + inputtemplate.extension\n else:\n filename = str(nextseq) +'-' + str(\"%034x\" % random.getrandbits(128))\n\n #Make sure filename matches (only if not an archive)\n if inputtemplate.acceptarchive and (filename[-7:].lower() == '.tar.gz' or filename[-8:].lower() == '.tar.bz2' or filename[-4:].lower() == '.zip'):\n pass\n else:\n if inputtemplate.filename:\n if filename != inputtemplate.filename:\n filename = inputtemplate.filename\n #return flask.make_response(\"Specified filename must the filename dictated by the inputtemplate, which is \" + inputtemplate.filename)\n #TODO LATER: add support for calling this with an actual number instead of #\n if inputtemplate.extension:\n if filename[-len(inputtemplate.extension) - 1:].lower() == '.' + inputtemplate.extension.lower():\n #good, extension matches (case independent). Let's just make sure the case is as defined exactly by the inputtemplate\n filename = filename[:-len(inputtemplate.extension) - 1] + '.' + inputtemplate.extension\n else:\n filename = filename + '.' + inputtemplate.extension\n #return flask.make_response(\"Specified filename does not have the extension dictated by the inputtemplate (\"+inputtemplate.extension+\")\") #403\n\n if inputtemplate.onlyinputsource and (not 'inputsource' in postdata or not postdata['inputsource']):\n return errorresponse(\"Adding files for this inputtemplate must proceed through inputsource\")\n\n if 'converter' in postdata and postdata['converter'] and not postdata['converter'] in [ x.id for x in inputtemplate.converters]:\n return errorresponse(\"Invalid converter specified: \" + postdata['converter'])\n\n #Make sure the filename is secure\n validfilename = True\n DISALLOWED = ('/','&','|','<','>',';','\"',\"'\",\"`\",\"{\",\"}\",\"\\n\",\"\\r\",\"\\b\",\"\\t\")\n for c in filename:\n if c in DISALLOWED:\n validfilename = False\n break\n\n if not validfilename:\n return errorresponse(\"Filename contains invalid symbols! Do not use /,&,|,<,>,',`,\\\",{,} or ;\")\n\n\n #Create the project (no effect if already exists)\n response = Project.create(project, user)\n if response is not None:\n return response\n\n\n printdebug(\"(Obtaining filename for uploaded file)\")\n head = \"\\n\"\n head += \"\\n\"\n if 'file' in flask.request.files:\n printlog(\"Adding client-side file \" + flask.request.files['file'].filename + \" to input files\")\n sourcefile = flask.request.files['file'].filename\n elif 'url' in postdata and postdata['url']:\n #Download from URL\n printlog(\"Adding web-based URL \" + postdata['url'] + \" to input files\")\n sourcefile = os.path.basename(postdata['url'])\n elif 'contents' in postdata and postdata['contents']:\n #In message\n printlog(\"Adding file \" + filename + \" with explicitly provided contents to input files\")\n sourcefile = \"editor\"\n elif 'inputsource' in postdata and postdata['inputsource']:\n printlog(\"Adding file \" + filename + \" from preinstalled data to input files\")\n if not inputsource:\n inputsource = None\n for s in inputtemplate.inputsources:\n if s.id.lower() == postdata['inputsource'].lower():\n inputsource = s\n if not inputsource:\n return errorresponse(\"Specified inputsource '\" + postdata['inputsource'] + \"' does not exist for inputtemplate '\"+inputtemplate.id+\"'\")\n sourcefile = os.path.basename(inputsource.path)\n elif 'accesstoken' in postdata and 'filename' in postdata:\n #XHR POST, data in body\n printlog(\"Adding client-side file \" + filename + \" to input files. Uploaded using XHR POST\")\n sourcefile = postdata['filename']\n else:\n return errorresponse(\"No file, url or contents specified!\")\n\n\n #============================ Generate metadata ========================================\n printdebug('(Generating and validating metadata)')\n if 'metafile' in flask.request.files: #and (not isinstance(postdata['metafile'], dict) or len(postdata['metafile']) > 0)):\n #an explicit metadata file was provided, upload it:\n printlog(\"Metadata explicitly provided in file, uploading...\")\n #Upload file from client to server\n metafile = Project.path(project, user) + 'input/.' + filename + '.METADATA'\n flask.request.files['metafile'].save(metafile)\n try:\n with io.open(metafile,'r',encoding='utf-8') as f:\n metadata = clam.common.data.CLAMMetaData.fromxml(f.read())\n errors, parameters = inputtemplate.validate(metadata, user)\n validmeta = True\n except Exception as e: #pylint: disable=broad-except\n printlog(\"Uploaded metadata is invalid! \" + str(e))\n metadata = None\n errors = True\n parameters = []\n validmeta = False\n elif 'metadata' in postdata and postdata['metadata']:\n printlog(\"Metadata explicitly provided in message, uploading...\")\n try:\n metadata = clam.common.data.CLAMMetaData.fromxml(postdata['metadata'])\n errors, parameters = inputtemplate.validate(metadata, user)\n validmeta = True\n except: #pylint: disable=bare-except\n printlog(\"Uploaded metadata is invalid!\")\n metadata = None\n errors = True\n parameters = []\n validmeta = False\n elif 'inputsource' in postdata and postdata['inputsource']:\n printlog(\"Getting metadata from inputsource, uploading...\")\n if inputsource.metadata:\n printlog(\"DEBUG: Validating metadata from inputsource\")\n metadata = inputsource.metadata\n errors, parameters = inputtemplate.validate(metadata, user)\n validmeta = True\n else:\n printlog(\"DEBUG: No metadata provided with inputsource, looking for metadata files..\")\n metafilename = os.path.dirname(inputsource.path)\n if metafilename: metafilename += '/'\n metafilename += '.' + os.path.basename(inputsource.path) + '.METADATA'\n if os.path.exists(metafilename):\n try:\n metadata = clam.common.data.CLAMMetaData.fromxml(open(metafilename,'r').readlines())\n errors, parameters = inputtemplate.validate(metadata, user)\n validmeta = True\n except: #pylint: disable=bare-except\n printlog(\"Uploaded metadata is invalid!\")\n metadata = None\n errors = True\n parameters = []\n validmeta = False\n else:\n return withheaders(flask.make_response(\"No metadata found nor specified for inputsource \" + inputsource.id ,500),headers={'allow_origin': settings.ALLOW_ORIGIN})\n else:\n errors, parameters = inputtemplate.validate(postdata, user)\n validmeta = True #will be checked later\n\n\n # ----------- Check if archive are allowed -------------\n archive = False\n addedfiles = []\n if not errors and inputtemplate.acceptarchive: #pylint: disable=too-many-nested-blocks\n printdebug('(Archive test)')\n # -------- Are we an archive? If so, determine what kind\n archivetype = None\n if 'file' in flask.request.files:\n uploadname = sourcefile.lower()\n archivetype = None\n if uploadname[-4:] == '.zip':\n archivetype = 'zip'\n elif uploadname[-7:] == '.tar.gz':\n archivetype = 'tar.gz'\n elif uploadname[-4:] == '.tar':\n archivetype = 'tar'\n elif uploadname[-8:] == '.tar.bz2':\n archivetype = 'tar.bz2'\n xhrpost = False\n elif 'accesstoken' in postdata and 'filename' in postdata:\n xhrpost = True\n if postdata['filename'][-7:].lower() == '.tar.gz':\n uploadname = sourcefile.lower()\n archivetype = 'tar.gz'\n elif postdata['filename'][-8:].lower() == '.tar.bz2':\n uploadname = sourcefile.lower()\n archivetype = 'tar.bz2'\n elif postdata['filename'][-4:].lower() == '.tar':\n uploadname = sourcefile.lower()\n archivetype = 'tar'\n elif postdata['filename'][-4:].lower() == '.zip':\n uploadname = sourcefile.lower()\n archivetype = 'zip'\n\n if archivetype:\n # =============== upload archive ======================\n #random name\n archive = \"%032x\" % random.getrandbits(128) + '.' + archivetype\n\n #Upload file from client to server\n printdebug('(Archive transfer starting)')\n if not xhrpost:\n flask.request.files['file'].save(Project.path(project,user) + archive)\n elif xhrpost:\n with open(Project.path(project,user) + archive,'wb') as f:\n while True:\n chunk = flask.request.stream.read(16384)\n if chunk:\n f.write(chunk)\n else:\n break\n printdebug('(Archive transfer completed)')\n # =============== Extract archive ======================\n\n #Determine extraction command\n if archivetype == 'zip':\n cmd = 'unzip -u'\n elif archivetype == 'tar':\n cmd = 'tar -xvf'\n elif archivetype == 'tar.gz':\n cmd = 'tar -xvzf'\n elif archivetype == 'tar.bz2':\n cmd = 'tar -xvjf'\n else:\n raise Exception(\"Invalid archive format: \" + archivetype) #invalid archive, shouldn't happen\n\n #invoke extractor\n printlog(\"Extracting '\" + archive + \"'\" )\n try:\n process = subprocess.Popen(cmd + \" \" + archive, cwd=Project.path(project,user), stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)\n except: #pylint: disable=bare-except\n return withheaders(flask.make_response(\"Unable to extract archive\",500),headers={'allow_origin': settings.ALLOW_ORIGIN})\n out, _ = process.communicate() #waits for process to end\n\n if sys.version < '3':\n if isinstance(out, str):\n out = unicode(out,'utf-8') #pylint: disable=undefined-variable\n else:\n if isinstance(out, bytes):\n out = str(out,'utf-8')\n\n #Read filename results\n\n firstline = True\n for line in out.split(\"\\n\"):\n line = line.strip()\n if line:\n printdebug('(Extraction output: ' + line+')')\n subfile = None\n if archivetype[0:3] == 'tar':\n subfile = line\n elif archivetype == 'zip' and not firstline: #firstline contains archive name itself, skip it\n colon = line.find(\":\")\n if colon:\n subfile = line[colon + 1:].strip()\n if subfile and os.path.isfile(Project.path(project, user) + subfile):\n subfile_newname = clam.common.data.resolveinputfilename(os.path.basename(subfile), parameters, inputtemplate, nextseq+len(addedfiles), project)\n printdebug('(Extracted file ' + subfile + ', moving to input/' + subfile_newname+')')\n os.rename(Project.path(project, user) + subfile, Project.path(project, user) + 'input/' + subfile_newname)\n addedfiles.append(subfile_newname)\n firstline = False\n\n #all done, remove archive\n os.unlink(Project.path(project, user) + archive)\n\n if not archive:\n addedfiles = [clam.common.data.resolveinputfilename(filename, parameters, inputtemplate, nextseq, project)]\n\n fatalerror = None\n\n jsonoutput = {'success': False if errors else True, 'isarchive': archive}\n\n\n output = head\n for filename in addedfiles: #pylint: disable=too-many-nested-blocks\n output += \"\\n\"\n if not errors:\n output += \"\"\n else:\n output += \"\"\n jsonoutput['error'] = 'There were parameter errors, file not uploaded: '\n for parameter in parameters:\n output += parameter.xml()\n if parameter.error:\n jsonoutput['error'] += parameter.error + \". \"\n output += \"\"\n\n\n\n if not errors:\n if not archive:\n #============================ Transfer file ========================================\n printdebug('(Start file transfer: ' + Project.path(project, user) + 'input/' + filename+' )')\n if 'file' in flask.request.files:\n printdebug('(Receiving data by uploading file)')\n #Upload file from client to server\n flask.request.files['file'].save(Project.path(project, user) + 'input/' + filename)\n elif 'url' in postdata and postdata['url']:\n printdebug('(Receiving data via url)')\n #Download file from 3rd party server to CLAM server\n try:\n r = requests.get(postdata['url'])\n except:\n raise flask.abort(404)\n if not (r.status_code >= 200 and r.status_code < 300):\n raise flask.abort(404)\n\n CHUNK = 16 * 1024\n f = open(Project.path(project, user) + 'input/' + filename,'wb')\n for chunk in r.iter_content(chunk_size=CHUNK):\n if chunk: # filter out keep-alive new chunks\n f.write(chunk)\n f.flush()\n f.close()\n elif 'inputsource' in postdata and postdata['inputsource']:\n #Copy (symlink!) from preinstalled data\n printdebug('(Creating symlink to file ' + inputsource.path + ' <- ' + Project.path(project,user) + '/input/ ' + filename + ')')\n os.symlink(inputsource.path, Project.path(project, user) + 'input/' + filename)\n elif 'contents' in postdata and postdata['contents']:\n printdebug('(Receiving data via from contents variable)')\n #grab encoding\n encoding = 'utf-8'\n for p in parameters:\n if p.id == 'encoding':\n encoding = p.value\n #Contents passed in POST message itself\n try:\n f = io.open(Project.path(project, user) + 'input/' + filename,'w',encoding=encoding)\n f.write(postdata['contents'])\n f.close()\n except UnicodeError:\n return errorresponse(\"Input file \" + str(filename) + \" is not in the expected encoding!\")\n elif 'accesstoken' in postdata and 'filename' in postdata:\n printdebug('(Receiving data directly from post body)')\n with open(Project.path(project,user) + 'input/' + filename,'wb') as f:\n while True:\n chunk = flask.request.stream.read(16384)\n if chunk:\n f.write(chunk)\n else:\n break\n\n printdebug('(File transfer completed)')\n\n\n\n #Create a file object\n file = clam.common.data.CLAMInputFile(Project.path(project, user), filename, False) #get CLAMInputFile without metadata (chicken-egg problem, this does not read the actual file contents!\n\n\n\n #============== Generate metadata ==============\n\n metadataerror = None\n if not metadata and not errors: #check if it has not already been set in another stage\n printdebug('(Generating metadata)')\n #for newly generated metadata\n try:\n #Now we generate the actual metadata object (unsaved yet though). We pass our earlier validation results to prevent computing it again\n validmeta, metadata, parameters = inputtemplate.generate(file, (errors, parameters ))\n if validmeta:\n #And we tie it to the CLAMFile object\n file.metadata = metadata\n #Add inputtemplate ID to metadata\n metadata.inputtemplate = inputtemplate.id\n else:\n metadataerror = \"Undefined error\"\n except ValueError as msg:\n validmeta = False\n metadataerror = msg\n except KeyError as msg:\n validmeta = False\n metadataerror = msg\n elif validmeta:\n #for explicitly uploaded metadata\n metadata.file = file\n file.metadata = metadata\n metadata.inputtemplate = inputtemplate.id\n\n if metadataerror:\n printdebug('(Metadata could not be generated, ' + str(metadataerror) + ', this usually indicated an error in service configuration)')\n #output += \"\" #This usually indicates an error in service configuration!\n fatalerror = \"Metadata could not be generated for \" + filename + \": \" + str(metadataerror) + \" (this usually indicates an error in service configuration!)\"\n jsonoutput['error'] = \"Metadata could not be generated! \" + str(metadataerror) + \" (this usually indicates an error in service configuration!)\"\n elif validmeta:\n #=========== Convert the uploaded file (if requested) ==============\n\n conversionerror = False\n if 'converter' in postdata and postdata['converter']:\n for c in inputtemplate.converters:\n if c.id == postdata['converter']:\n converter = c\n break\n if converter: #(should always be found, error already provided earlier if not)\n printdebug('(Invoking converter)')\n try:\n success = converter.convertforinput(Project.path(project, user) + 'input/' + filename, metadata)\n except: #pylint: disable=bare-except\n success = False\n if not success:\n conversionerror = True\n fatalerror = \"The file \" + xmlescape(filename) + \" could not be converted\"\n jsonoutput['error'] = \"The file could not be converted\"\n jsonoutput['success'] = False\n\n #====================== Validate the file itself ====================\n if not conversionerror:\n valid = file.validate()\n\n if valid:\n printdebug('(Validation ok)')\n output += \"yes\"\n\n #Great! Everything ok, save metadata\n metadata.save(Project.path(project, user) + 'input/' + file.metafilename())\n\n #And create symbolic link for inputtemplates\n linkfilename = os.path.dirname(filename)\n if linkfilename: linkfilename += '/'\n linkfilename += '.' + os.path.basename(filename) + '.INPUTTEMPLATE' + '.' + inputtemplate.id + '.' + str(nextseq)\n os.symlink(Project.path(project, user) + 'input/' + filename, Project.path(project, user) + 'input/' + linkfilename)\n else:\n printdebug('(Validation error)')\n #Too bad, everything worked out but the file itself doesn't validate.\n #output += \"no\"\n fatalerror = \"The file \" + xmlescape(filename) + \" did not validate, it is not in the proper expected format.\"\n jsonoutput['error'] = \"The file \" + filename.replace(\"'\",\"\") + \" did not validate, it is not in the proper expected format.\"\n jsonoutput['success'] = False\n #remove upload\n os.unlink(Project.path(project, user) + 'input/' + filename)\n\n\n output += \"\\n\"\n\n output += \"\"\n\n\n\n if returntype == 'boolean':\n return jsonoutput['success']\n elif fatalerror:\n #fatal error return error message with 403 code\n printlog('Fatal Error during upload: ' + fatalerror)\n return errorresponse(head + fatalerror,403)\n elif errors:\n #parameter errors, return XML output with 403 code\n printdebug('There were parameter errors during upload!')\n if returntype == 'json':\n jsonoutput['xml'] = output #embed XML in JSON for complete client-side processing\n return withheaders(flask.make_response(json.dumps(jsonoutput)), 'application/json', {'allow_origin': settings.ALLOW_ORIGIN})\n else:\n return withheaders(flask.make_response(output,403),headers={'allow_origin': settings.ALLOW_ORIGIN})\n elif returntype == 'xml': #success\n printdebug('Returning xml')\n return withheaders(flask.make_response(output), 'text/xml', {'allow_origin': settings.ALLOW_ORIGIN})\n elif returntype == 'json': #success\n printdebug('Returning json')\n #everything ok, return JSON output (caller decides)\n jsonoutput['xml'] = output #embed XML in JSON for complete client-side processing\n return withheaders(flask.make_response(json.dumps(jsonoutput)), 'application/json', {'allow_origin': settings.ALLOW_ORIGIN})\n elif returntype == 'true_on_success':\n return True\n else:\n printdebug('Invalid return type')\n raise Exception(\"invalid return type\")", "response": "Add a new file to the input file list"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nruns CLAM in WSGI mode", "response": "def run_wsgi(settings_module):\n \"\"\"Run CLAM in WSGI mode\"\"\"\n global settingsmodule, DEBUG #pylint: disable=global-statement\n printdebug(\"Initialising WSGI service\")\n\n globals()['settings'] = settings_module\n settingsmodule = settings_module.__name__\n\n try:\n if settings.DEBUG:\n DEBUG = True\n setdebug(True)\n except:\n pass\n\n test_version()\n if DEBUG:\n setlog(sys.stderr)\n else:\n setlog(None)\n try:\n if settings.LOGFILE:\n setlogfile(settings.LOGFILE)\n except:\n pass\n set_defaults() #host, port\n test_dirs()\n\n if DEBUG:\n from werkzeug.debug import DebuggedApplication\n return DebuggedApplication(CLAMService('wsgi').service.wsgi_app, True)\n else:\n return CLAMService('wsgi').service.wsgi_app"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets list of projects", "response": "def index(credentials=None):\n \"\"\"Get list of projects\"\"\"\n user, oauth_access_token = parsecredentials(credentials)\n if not settings.ADMINS or user not in settings.ADMINS:\n return flask.make_response('You shall not pass!!! You are not an administrator!',403)\n\n usersprojects = {}\n totalsize = {}\n for f in glob.glob(settings.ROOT + \"projects/*\"):\n if os.path.isdir(f):\n u = os.path.basename(f)\n usersprojects[u], totalsize[u] = getprojects(u)\n usersprojects[u].sort()\n\n return withheaders(flask.make_response(flask.render_template('admin.html',\n version=VERSION,\n system_id=settings.SYSTEM_ID,\n system_name=settings.SYSTEM_NAME,\n system_description=settings.SYSTEM_DESCRIPTION,\n system_author=settings.SYSTEM_AUTHOR,\n system_version=settings.SYSTEM_VERSION,\n system_email=settings.SYSTEM_EMAIL,\n user=user,\n url=getrooturl(),\n usersprojects = sorted(usersprojects.items()),\n totalsize=totalsize,\n allow_origin=settings.ALLOW_ORIGIN,\n oauth_access_token=oauth_encrypt(oauth_access_token)\n )), \"text/html; charset=UTF-8\", {'allow_origin':settings.ALLOW_ORIGIN})"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef path(project, credentials):\n user, oauth_access_token = parsecredentials(credentials) #pylint: disable=unused-variable\n return settings.ROOT + \"projects/\" + user + '/' + project + \"/\"", "response": "Get the path to the project."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create(project, credentials): #pylint: disable=too-many-return-statements\n\n if not settings.COMMAND:\n return flask.make_response(\"Projects disabled, no command configured\",404)\n\n user, oauth_access_token = parsecredentials(credentials) #pylint: disable=unused-variable\n if not Project.validate(project):\n return withheaders(flask.make_response('Invalid project ID. Note that only alphanumerical characters are allowed.',403),headers={'allow_origin': settings.ALLOW_ORIGIN})\n printdebug(\"Checking if \" + settings.ROOT + \"projects/\" + user + '/' + project + \" exists\")\n if not project:\n return withheaders(flask.make_response('No project name',403),headers={'allow_origin': settings.ALLOW_ORIGIN})\n if not os.path.isdir(settings.ROOT + \"projects/\" + user):\n printlog(\"Creating user directory '\" + user + \"'\")\n os.makedirs(settings.ROOT + \"projects/\" + user)\n if not os.path.isdir(settings.ROOT + \"projects/\" + user): #verify:\n return withheaders(flask.make_response(\"Directory \" + settings.ROOT + \"projects/\" + user + \" could not be created succesfully\",403),headers={'allow_origin': settings.ALLOW_ORIGIN})\n\n\n #checking user quota\n if settings.USERQUOTA > 0:\n _, totalsize = getprojects(user)\n if totalsize > settings.USERQUOTA:\n printlog(\"User \" + user + \" exceeded quota, refusing to create new project...\")\n return withheaders(flask.make_response(\"Unable to create new project because you are exceeding your disk quota (max \" + str(settings.USERQUOTA) + \" MB, you now use \" + str(totalsize) + \" MB). Please delete some projects and try again.\",403),headers={'allow_origin': settings.ALLOW_ORIGIN})\n\n if not os.path.isdir(settings.ROOT + \"projects/\" + user + '/' + project):\n printlog(\"Creating project '\" + project + \"'\")\n os.makedirs(settings.ROOT + \"projects/\" + user + '/' + project)\n\n #project index will need to be regenerated, remove cache\n if os.path.exists(os.path.join(settings.ROOT + \"projects/\" + user,'.index')):\n os.unlink(os.path.join(settings.ROOT + \"projects/\" + user,'.index'))\n\n if not os.path.isdir(settings.ROOT + \"projects/\" + user + '/' + project + '/input/'):\n os.makedirs(settings.ROOT + \"projects/\" + user + '/' + project + \"/input\")\n if not os.path.isdir(settings.ROOT + \"projects/\" + user + '/' + project + '/input'):\n return withheaders(flask.make_response(\"Input directory \" + settings.ROOT + \"projects/\" + user + '/' + project + \"/input/ could not be created succesfully\",403),headers={'allow_origin': settings.ALLOW_ORIGIN})\n if not os.path.isdir(settings.ROOT + \"projects/\" + user + '/' + project + '/output/'):\n os.makedirs(settings.ROOT + \"projects/\" + user + '/' + project + \"/output\")\n if not os.path.isdir(settings.ROOT + \"projects/\" + user + '/' + project + '/output'):\n return withheaders(flask.make_response(\"Output directory \" + settings.ROOT + \"projects/\" + user + '/' + project + \"/output/ could not be created succesfully\",403),headers={'allow_origin': settings.ALLOW_ORIGIN})\n if not os.path.isdir(settings.ROOT + \"projects/\" + user + '/' + project + '/tmp/'):\n os.makedirs(settings.ROOT + \"projects/\" + user + '/' + project + \"/tmp\")\n if not os.path.isdir(settings.ROOT + \"projects/\" + user + '/' + project + '/tmp'):\n return withheaders(flask.make_response(\"tmp directory \" + settings.ROOT + \"projects/\" + user + '/' + project + \"/tmp/ could not be created succesfully\",403),headers={'allow_origin': settings.ALLOW_ORIGIN})\n\n return None", "response": "Create a new project if it does not already exist."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nchecking if the project exists", "response": "def exists(project, credentials):\n \"\"\"Check if the project exists\"\"\"\n user, oauth_access_token = parsecredentials(credentials) #pylint: disable=unused-variable\n printdebug(\"Checking if project \" + project + \" exists for \" + user)\n return os.path.isdir(Project.path(project, user))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef inputindexbytemplate(project, user, inputtemplate):\n index = [] #pylint: disable=redefined-outer-name\n prefix = Project.path(project, user) + 'input/'\n for linkf, f in globsymlinks(prefix + '.*.INPUTTEMPLATE.' + inputtemplate.id + '.*'):\n seq = int(linkf.split('.')[-1])\n index.append( (seq,f) )\n\n #yield CLAMFile objects in proper sequence\n for seq, f in sorted(index):\n yield seq, clam.common.data.CLAMInputFile(Project.path(project, user), f[len(prefix):])", "response": "Retrieve sorted index for the specified input template"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nretrieving sorted index for the specified output template", "response": "def outputindexbytemplate(project, user, outputtemplate):\n \"\"\"Retrieve sorted index for the specified input template\"\"\"\n index = [] #pylint: disable=redefined-outer-name\n prefix = Project.path(project, user) + 'output/'\n for linkf, f in globsymlinks(prefix + '.*.OUTPUTTEMPLATE.' + outputtemplate.id + '.*'):\n seq = int(linkf.split('.')[-1])\n index.append( (seq,f) )\n\n #yield CLAMFile objects in proper sequence\n for seq, f in sorted(index):\n yield seq, clam.common.data.CLAMOutputFile(Project.path(project, user), f[len(prefix):])"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates an empty project", "response": "def new(project, credentials=None):\n \"\"\"Create an empty project\"\"\"\n user, oauth_access_token = parsecredentials(credentials)\n response = Project.create(project, user)\n if response is not None:\n return response\n msg = \"Project \" + project + \" has been created for user \" + user\n if oauth_access_token:\n extraloc = '?oauth_access_token=' + oauth_access_token\n else:\n extraloc = ''\n return flask.make_response(msg, 201, {'Location': getrooturl() + '/' + project + '/' + extraloc, 'Content-Type':'text/plain','Content-Length': len(msg),'Access-Control-Allow-Origin': settings.ALLOW_ORIGIN, 'Access-Control-Allow-Methods': 'GET, POST, PUT, DELETE', 'Access-Control-Allow-Headers': 'Authorization'})"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef start(project, credentials=None): #pylint: disable=too-many-return-statements\n\n #handle shortcut\n shortcutresponse = entryshortcut(credentials, True) #protected against endless recursion, will return None when no shortcut is found, True when one is found and starting should continue\n if shortcutresponse is not None and shortcutresponse is not True:\n return shortcutresponse\n\n user, oauth_access_token = parsecredentials(credentials)\n response = Project.create(project, user)\n if response is not None:\n return response\n #if user and not Project.access(project, user):\n # return flask.make_response(\"Access denied to project \" + project + \" for user \" + user,401) #401\n\n statuscode, _, _, _ = Project.status(project, user)\n if statuscode != clam.common.status.READY:\n if oauth_access_token:\n return withheaders(flask.redirect(getrooturl() + '/' + project + '/?oauth_access_token=' + oauth_access_token),headers={'allow_origin': settings.ALLOW_ORIGIN})\n else:\n return withheaders(flask.redirect(getrooturl() + '/' + project),headers={'allow_origin': settings.ALLOW_ORIGIN})\n\n #Generate arguments based on POSTed parameters\n commandlineparams = []\n postdata = flask.request.values\n\n errors, parameters, commandlineparams = clam.common.data.processparameters(postdata, settings.PARAMETERS)\n\n sufresources, resmsg = sufficientresources()\n if not sufresources:\n printlog(\"*** NOT ENOUGH SYSTEM RESOURCES AVAILABLE: \" + resmsg + \" ***\")\n return withheaders(flask.make_response(\"There are not enough system resources available to accommodate your request. \" + resmsg + \" .Please try again later.\",503),headers={'allow_origin': settings.ALLOW_ORIGIN})\n if not errors: #We don't even bother running the profiler if there are errors\n matchedprofiles, program = clam.common.data.profiler(settings.PROFILES, Project.path(project, user), parameters, settings.SYSTEM_ID, settings.SYSTEM_NAME, getrooturl(), printdebug)\n #converted matched profiles to a list of indices\n matchedprofiles_byindex = []\n for i, profile in enumerate(settings.PROFILES):\n if profile in matchedprofiles:\n matchedprofiles_byindex.append(i)\n\n if errors:\n #There are parameter errors, return 403 response with errors marked\n printlog(\"There are parameter errors, not starting.\")\n return Project.response(user, project, parameters,\"\",False,oauth_access_token, http_code=403)\n elif not matchedprofiles:\n printlog(\"No profiles matching, not starting.\")\n return Project.response(user, project, parameters, \"No profiles matching input and parameters, unable to start. Are you sure you added all necessary input files and set all necessary parameters?\", False, oauth_access_token,http_code=403)\n else:\n #everything good, write clam.xml output file and start\n with io.open(Project.path(project, user) + \"clam.xml\",'wb') as f:\n f.write(Project.response(user, project, parameters, \"\",True, oauth_access_token, \",\".join([str(x) for x in matchedprofiles_byindex]), program).data)\n\n\n\n #Start project with specified parameters\n cmd = settings.COMMAND\n cmd = cmd.replace('$PARAMETERS', \" \".join(commandlineparams)) #commandlineparams is shell-safe\n #if 'usecorpus' in postdata and postdata['usecorpus']:\n # corpus = postdata['usecorpus'].replace('..','') #security\n # #use a preinstalled corpus:\n # if os.path.exists(settings.ROOT + \"corpora/\" + corpus):\n # cmd = cmd.replace('$INPUTDIRECTORY', settings.ROOT + \"corpora/\" + corpus + \"/\")\n # else:\n # raise web.webapi.NotFound(\"Corpus \" + corpus + \" not found\")\n #else:\n cmd = cmd.replace('$INPUTDIRECTORY', Project.path(project, user) + 'input/')\n cmd = cmd.replace('$OUTPUTDIRECTORY',Project.path(project, user) + 'output/')\n cmd = cmd.replace('$TMPDIRECTORY',Project.path(project, user) + 'tmp/')\n cmd = cmd.replace('$STATUSFILE',Project.path(project, user) + '.status')\n cmd = cmd.replace('$DATAFILE',Project.path(project, user) + 'clam.xml')\n cmd = cmd.replace('$USERNAME',user if user else \"anonymous\")\n cmd = cmd.replace('$PROJECT',project) #alphanumberic only, shell-safe\n cmd = cmd.replace('$OAUTH_ACCESS_TOKEN',oauth_access_token)\n cmd = clam.common.data.escapeshelloperators(cmd)\n #everything should be shell-safe now\n cmd += \" 2> \" + Project.path(project, user) + \"output/error.log\" #add error output\n\n pythonpath = ''\n try:\n pythonpath = ':'.join(settings.DISPATCHER_PYTHONPATH)\n except AttributeError:\n pass\n if pythonpath:\n pythonpath = os.path.dirname(settings.__file__) + ':' + pythonpath\n else:\n pythonpath = os.path.dirname(settings.__file__)\n\n #if settings.DISPATCHER == 'clamdispatcher' and os.path.exists(settings.CLAMDIR + '/' + settings.DISPATCHER + '.py') and stat.S_IXUSR & os.stat(settings.CLAMDIR + '/' + settings.DISPATCHER+'.py')[stat.ST_MODE]:\n # #backward compatibility for old configurations without setuptools\n # cmd = settings.CLAMDIR + '/' + settings.DISPATCHER + '.py'\n #else:\n cmd = settings.DISPATCHER + ' ' + pythonpath + ' ' + settingsmodule + ' ' + Project.path(project, user) + ' ' + cmd\n if settings.REMOTEHOST:\n if settings.REMOTEUSER:\n cmd = \"ssh -o NumberOfPasswordPrompts=0 \" + settings.REMOTEUSER + \"@\" + settings.REMOTEHOST + \" \" + cmd\n else:\n cmd = \"ssh -o NumberOfPasswordPrompts=0 \" + settings.REMOTEHOST + \" \" + cmd\n printlog(\"Starting dispatcher \" + settings.DISPATCHER + \" with \" + settings.COMMAND + \": \" + repr(cmd) + \" ...\" )\n #process = subprocess.Popen(cmd,cwd=Project.path(project), shell=True)\n process = subprocess.Popen(cmd,cwd=settings.CLAMDIR, shell=True)\n if process:\n pid = process.pid\n printlog(\"Started dispatcher with pid \" + str(pid) )\n with open(Project.path(project, user) + '.pid','w') as f: #will be handled by dispatcher!\n f.write(str(pid))\n if shortcutresponse is True:\n #redirect to project page to lose parameters in URL\n if oauth_access_token:\n return withheaders(flask.redirect(getrooturl() + '/' + project + '/?oauth_access_token=' + oauth_access_token),headers={'allow_origin': settings.ALLOW_ORIGIN})\n else:\n return withheaders(flask.redirect(getrooturl() + '/' + project),headers={'allow_origin': settings.ALLOW_ORIGIN})\n else:\n #normal response (202)\n return Project.response(user, project, parameters,\"\",False,oauth_access_token,\",\".join([str(x) for x in matchedprofiles_byindex]), program,http_code=202) #returns 202 - Accepted\n else:\n return withheaders(flask.make_response(\"Unable to launch process\",500),headers={'allow_origin': settings.ALLOW_ORIGIN})", "response": "Start execution of a project."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef deleteoutputfile(project, filename, credentials=None):\n\n user, oauth_access_token = parsecredentials(credentials) #pylint: disable=unused-variable\n if filename: filename = filename.replace(\"..\",\"\") #Simple security\n\n if not filename or len(filename) == 0:\n #Deleting all output files and resetting\n Project.reset(project, user)\n msg = \"Deleted\"\n return withheaders(flask.make_response(msg), 'text/plain',{'Content-Length':len(msg), 'allow_origin': settings.ALLOW_ORIGIN}) #200\n elif os.path.isdir(Project.path(project, user) + filename):\n #Deleting specified directory\n shutil.rmtree(Project.path(project, user) + filename)\n msg = \"Deleted\"\n return withheaders(flask.make_response(msg), 'text/plain',{'Content-Length':len(msg), 'allow_origin': settings.ALLOW_ORIGIN}) #200\n else:\n try:\n file = clam.common.data.CLAMOutputFile(Project.path(project, user), filename)\n except:\n raise flask.abort(404)\n\n success = file.delete()\n if not success:\n raise flask.abort(404)\n else:\n msg = \"Deleted\"\n return withheaders(flask.make_response(msg), 'text/plain',{'Content-Length':len(msg), 'allow_origin': settings.ALLOW_ORIGIN})", "response": "Delete an output file"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef reset(project, user):\n d = Project.path(project, user) + \"output\"\n if os.path.isdir(d):\n shutil.rmtree(d)\n os.makedirs(d)\n else:\n raise flask.abort(404)\n if os.path.exists(Project.path(project, user) + \".done\"):\n os.unlink(Project.path(project, user) + \".done\")\n if os.path.exists(Project.path(project, user) + \".status\"):\n os.unlink(Project.path(project, user) + \".status\")", "response": "Reset system delete all output files and prepare for a new run"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef getarchive(project, user, format=None):\n if os.path.isfile(Project.path(project, user) + '.download'):\n #make sure we don't start two compression processes at the same time\n return withheaders(flask.make_response('Another compression is already running',403),headers={'allow_origin': settings.ALLOW_ORIGIN})\n else:\n if not format:\n data = flask.request.values\n if 'format' in data:\n format = data['format']\n else:\n format = 'zip' #default\n\n #validation, security\n contentencoding = None\n if format == 'zip':\n contenttype = 'application/zip'\n command = \"/usr/bin/zip -r\" #TODO: do not hard-code path!\n if os.path.isfile(Project.path(project, user) + \"output/\" + project + \".tar.gz\"):\n os.unlink(Project.path(project, user) + \"output/\" + project + \".tar.gz\")\n if os.path.isfile(Project.path(project, user) + \"output/\" + project + \".tar.bz2\"):\n os.unlink(Project.path(project, user) + \"output/\" + project + \".tar.bz2\")\n elif format == 'tar.gz':\n contenttype = 'application/x-tar'\n contentencoding = 'gzip'\n command = \"/bin/tar -czf\"\n if os.path.isfile(Project.path(project, user) + \"output/\" + project + \".zip\"):\n os.unlink(Project.path(project, user) + \"output/\" + project + \".zip\")\n if os.path.isfile(Project.path(project, user) + \"output/\" + project + \".tar.bz2\"):\n os.unlink(Project.path(project, user) + \"output/\" + project + \".tar.bz2\")\n elif format == 'tar.bz2':\n contenttype = 'application/x-bzip2'\n command = \"/bin/tar -cjf\"\n if os.path.isfile(Project.path(project, user) + \"output/\" + project + \".tar.gz\"):\n os.unlink(Project.path(project, user) + \"output/\" + project + \".tar.gz\")\n if os.path.isfile(Project.path(project, user) + \"output/\" + project + \".zip\"):\n os.unlink(Project.path(project, user) + \"output/\" + project + \".zip\")\n else:\n return withheaders(flask.make_response('Invalid archive format',403) ,headers={'allow_origin': settings.ALLOW_ORIGIN})#TODO: message won't show\n\n path = Project.path(project, user) + \"output/\" + project + \".\" + format\n\n if not os.path.isfile(path):\n printlog(\"Building download archive in \" + format + \" format\")\n cmd = command + ' ' + project + '.' + format + ' *'\n printdebug(cmd)\n printdebug(Project.path(project, user)+'output/')\n process = subprocess.Popen(cmd, cwd=Project.path(project, user)+'output/', shell=True)\n if not process:\n return withheaders(flask.make_response(\"Unable to make download package\",500),headers={'allow_origin': settings.ALLOW_ORIGIN})\n else:\n pid = process.pid\n f = open(Project.path(project, user) + '.download','w')\n f.write(str(pid))\n f.close()\n os.waitpid(pid, 0) #wait for process to finish\n os.unlink(Project.path(project, user) + '.download')\n\n extraheaders = {'allow_origin': settings.ALLOW_ORIGIN }\n if contentencoding:\n extraheaders['Content-Encoding'] = contentencoding\n return withheaders(flask.Response( getbinarydata(path) ), contenttype, extraheaders )", "response": "Generates and returns a download package"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef deleteinputfile(project, filename, credentials=None):\n\n user, oauth_access_token = parsecredentials(credentials) #pylint: disable=unused-variable\n filename = filename.replace(\"..\",\"\") #Simple security\n\n if len(filename) == 0:\n #Deleting all input files\n shutil.rmtree(Project.path(project, user) + 'input')\n os.makedirs(Project.path(project, user) + 'input') #re-add new input directory\n return \"Deleted\" #200\n elif os.path.isdir(Project.path(project, user) + filename):\n #Deleting specified directory\n shutil.rmtree(Project.path(project, user) + filename)\n return \"Deleted\" #200\n else:\n try:\n file = clam.common.data.CLAMInputFile(Project.path(project, user), filename)\n except:\n raise flask.abort(404)\n\n success = file.delete()\n if not success:\n raise flask.abort(404)\n else:\n msg = \"Deleted\"\n return withheaders(flask.make_response(msg),'text/plain', {'Content-Length': len(msg), 'allow_origin': settings.ALLOW_ORIGIN})", "response": "Delete an input file"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef addinputfile(project, filename, credentials=None): #pylint: disable=too-many-return-statements\n\n printdebug('Addinputfile: Initialising' )\n\n user, oauth_access_token = parsecredentials(credentials) #pylint: disable=unused-variable\n response = Project.create(project, user)\n if response is not None:\n return response\n postdata = flask.request.values\n\n if filename == '': #pylint: disable=too-many-nested-blocks\n #Handle inputsource\n printdebug('Addinputfile: checking for input source' )\n if 'inputsource' in postdata and postdata['inputsource']:\n inputsource = None\n inputtemplate = None\n for profile in settings.PROFILES:\n for t in profile.input:\n for s in t.inputsources:\n if s.id == postdata['inputsource']:\n inputsource = s\n inputsource.inputtemplate = t.id\n inputtemplate = t\n break\n if not inputsource:\n for s in settings.INPUTSOURCES:\n if s.id == postdata['inputsource']:\n inputsource = s\n if not inputsource:\n return withheaders(flask.make_response(\"No such inputsource exists\",403),headers={'allow_origin': settings.ALLOW_ORIGIN})\n if not inputtemplate:\n for profile in settings.PROFILES:\n for t in profile.input:\n if inputsource.inputtemplate == t.id:\n inputtemplate = t\n if inputtemplate is None:\n return withheaders(flask.make_response(\"No input template found for inputsource\",500),headers={'allow_origin': settings.ALLOW_ORIGIN})\n if inputsource.isfile():\n if inputtemplate.filename:\n filename = inputtemplate.filename\n else:\n filename = os.path.basename(inputsource.path)\n xml = addfile(project, filename, user, {'inputsource': postdata['inputsource'], 'inputtemplate': inputtemplate.id}, inputsource)\n return xml\n elif inputsource.isdir():\n if inputtemplate.filename:\n filename = inputtemplate.filename\n for f in glob.glob(inputsource.path + \"/*\"):\n if not inputtemplate.filename:\n filename = os.path.basename(f)\n if f[0] != '.':\n tmpinputsource = clam.common.data.InputSource(id='tmp',label='tmp',path=f, metadata=inputsource.metadata)\n addfile(project, filename, user, {'inputsource':'tmp', 'inputtemplate': inputtemplate.id}, tmpinputsource)\n #WARNING: Output is dropped silently here!\n return \"\" #200\n else:\n assert False\n else:\n return withheaders(flask.make_response(\"No filename or inputsource specified\",403),headers={'allow_origin': settings.ALLOW_ORIGIN})\n else:\n #Simply forward to addfile\n return addfile(project,filename,user, postdata)", "response": "Add a new input file to the database"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting list of pre - installed corpora", "response": "def corpusindex():\n \"\"\"Get list of pre-installed corpora\"\"\"\n corpora = []\n for f in glob.glob(settings.ROOT + \"corpora/*\"):\n if os.path.isdir(f):\n corpora.append(os.path.basename(f))\n return corpora"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef validate(self, nonce):\n if self.debug: print(\"Checking nonce \" + str(nonce),file=sys.stderr)\n try:\n opaque, ip, expiretime = self.get(nonce) #pylint: disable=unused-variable\n if expiretime < time.time():\n if self.debug: print(\"Nonce expired\",file=sys.stderr)\n self.remove(nonce)\n return False\n elif ip != flask.request.remote_addr:\n if self.debug: print(\"Nonce IP mismatch\",file=sys.stderr)\n self.remove(nonce)\n return False\n else:\n return True\n except KeyError:\n if self.debug: print(\"Nonce \" + nonce + \" does not exist\",file=sys.stderr)\n return False", "response": "Does the nonce exist and is it valid for the request?"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef json(self):\n return {\n \"id\": self.ID,\n \"steps\": self.steps,\n \"graph_source\": self.source,\n \"errored\": self.errored\n }", "response": "Retrun JSON representation of this run"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a new RunData object and increments pointers", "response": "def new_run(self):\n \"\"\"Creates a new RunData object and increments pointers\"\"\"\n self.current_run += 1\n self.runs.append(RunData(self.current_run + 1))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef follow_log(self):\n # Server needs to be up and running before starting sending POST requests\n time.sleep(5)\n try:\n if self.remote:\n logger.debug('Logfile in remote host!')\n cl = client.SSHClient()\n # Try to load system keys\n cl.load_system_host_keys()\n cl.connect(self.remote['host'], port=self.remote.get('port', 22), username=self.remote.get('username', None), \\\n password=self.remote.get('password', None))\n sftp = cl.open_sftp()\n f = sftp.open(self.logfile, 'r')\n f.settimeout(5) # Set 5 seconds timeout for read operations\n else:\n f = open(self.logfile, 'r')\n except IOError:\n raise RuntimeError(\"Provided logfile does not exist or its not readable\")\n self.analysis_finished = False\n last_line_read = False\n while not self.analysis_finished:\n try:\n line = f.readline()\n except timeout:\n logger.error(\"Connection with the server lost, trying to reconnect and continue reading\")\n current_pos = f.tell()\n try:\n cl.connect(self.remote['host'], port=self.remote.get('port', 22), username=self.remote.get('username', None), \\\n password=self.remote.get('password', None), timeout=300, banner_timeout=300)\n except error:\n logger.error(\"Couldn't connect to the server after 5 minutes, aborting.\")\n os._exit(0)\n else:\n logger.info(\"Connection restablished!! Will continue reading the logfile\")\n sftp = cl.open_sftp()\n f = sftp.open(self.logfile, 'r')\n f.seek(current_pos)\n else:\n if not line:\n self.finished_reading = True\n if not last_line_read:\n self.update_frontend({'finished_reading': True})\n if self.update:\n self.update_frontend(self._last_message)\n last_line_read = True\n time.sleep(1)\n continue\n parsed_line = ps.parse_log_line(line)\n self._last_message = parsed_line\n self.analysis_finished = parsed_line['step'] == 'finished'\n\n # If this is a new step, update internal data\n parsed_line['new_run'] = False\n if parsed_line['step'] and not parsed_line['step'] == 'error':\n if self.FIRST_STEP is None:\n self.FIRST_STEP = parsed_line['step']\n elif parsed_line['step'] == self.FIRST_STEP:\n parsed_line['new_run'] = True\n self.new_run()\n node_id = 'run-{}_'.format(self.current_run + 1) + '_'.join(parsed_line['step'].lower().split())\n parsed_line['step_id'] = node_id\n self.runs[self.current_run].steps.append(parsed_line)\n self.runs[self.current_run].node(node_id, parsed_line['step'])\n self.runs[self.current_run]._nodes.append(node_id)\n n_nodes = len(self.runs[self.current_run]._nodes)\n if n_nodes > 1:\n self.runs[self.current_run].edge(self.runs[self.current_run]._nodes[n_nodes - 2], self.runs[self.current_run]._nodes[n_nodes -1])\n parsed_line['graph_source'] = self.runs[self.current_run].source\n\n elif parsed_line['step'] == 'error':\n self.runs[self.current_run].errored = True\n\n # Update frontend only if its a new step _or_ the update flag is set to true and we are\n # not loading the log for the first time\n if (last_line_read and self.update) or parsed_line['step']:\n self.update_frontend(parsed_line)\n f.close()", "response": "Reads a logfile continuously and updates the internal graph if new step is found."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef update_frontend(self, info):\n headers = {'Content-Type': 'text/event-stream'}\n if info.get('when'):\n info['when'] = info['when'].isoformat()\n requests.post(self.base_url + '/publish', data=json.dumps(info), headers=headers)", "response": "Updates the frontend with info from the log. i. e regular line new step."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns some summary data for a finished analysis", "response": "def get_summary(self):\n \"\"\"Returns some summary data for a finished analysis\"\"\"\n if not self.analysis_finished:\n return []\n summary = {'times_summary': []}\n for i in range(len(self.runs[self.current_run].steps) - 1):\n step = self.runs[self.current_run].steps[i]\n begin = parse(step['when'])\n end = parse(self.runs[self.current_run].steps[i + 1]['when'])\n duration = end - begin\n summary['times_summary'].append((step['step'], duration.seconds))\n return summary"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _grabContentFromUrl(self, url):\n # Defining an empty object for the response\n info = {}\n \n # This part has to be modified... \n try: \n # Configuring the socket\n queryURL = \"http://\" + self.info[\"host\"] + \":\" + self.info[\"port\"] + \"/\" + url\n response = urllib2.urlopen(queryURL)\n\n # Rebuilding data to be processed\n data = str(response.headers) + \"\\n\"\n\n data += response.read()\n\n # Processing data as expected\n info = self._createDataStructure(data) \n \n # Try to make the errors clear for other users\n except Exception, e:\n errMsg = \"ERROR Exception. Something seems to be wrong with the Zeronet Bundler.\" \n raise Exception( errMsg + \" \" + str(e)) \n \n return info", "response": "Function that abstracts capturing a URL. This method rewrites the one from Wrapper."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nhandling all commands that take a filename and 0 or more extra arguments.", "response": "def _handle_rundebug_from_shell(cmd_line):\n \"\"\"\n Handles all commands that take a filename and 0 or more extra arguments.\n Passes the command to backend.\n\n (Debugger plugin may also use this method)\n \"\"\"\n command, args = parse_shell_command(cmd_line)\n\n if len(args) >= 1:\n get_workbench().get_editor_notebook().save_all_named_editors()\n\n origcommand=command\n if command == \"Ev3RemoteRun\":\n command=\"Run\"\n if command == \"Ev3RemoteDebug\":\n command=\"Debug\"\n\n cmd = ToplevelCommand(command=command,\n filename=args[0],\n args=args[1:])\n\n if origcommand == \"Ev3RemoteRun\" or origcommand == \"Ev3RemoteDebug\":\n cmd.environment={ \"EV3MODE\" : \"remote\", \"EV3IP\": get_workbench().get_option(\"ev3.ip\") }\n\n if os.path.isabs(cmd.filename):\n cmd.full_filename = cmd.filename\n else:\n runner=get_runner()\n cmd.full_filename = os.path.join(runner.get_cwd(), cmd.filename)\n\n if command in [\"Run\", \"run\", \"Debug\", \"debug\"]:\n with tokenize.open(cmd.full_filename) as fp:\n cmd.source = fp.read()\n\n get_runner().send_command(cmd)\n else:\n\n print_error_in_backend(\"Command '{}' takes at least one argument\".format(command))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef upload_current_script():\n current_editor = get_workbench().get_editor_notebook().get_current_editor()\n code = current_editor.get_text_widget().get(\"1.0\", \"end\")\n try:\n filename= current_editor.get_filename()\n if (not filename) or (not filename.endswith(\".py\")):\n return\n\n ast.parse(code)\n\n if not code.startswith('#!'):\n if tkMessageBox.askokcancel(\"Shebang\", \"To make a python file on the EV3 executable Thonny prepends the shebang line:\\n\\n !#/usr/bin/env python3\\n\"):\n current_editor.get_text_widget().direct_insert(\"1.0\", \"#!/usr/bin/env python3\\n\")\n else:\n return\n\n # automatically save file (without confirm dialog, nor dialog for asking filename )\n # save_file does return None, if script is not saved and user closed file saving window, otherwise return file name.\n py_file = current_editor.save_file(False)\n if py_file is None:\n return\n\n upload(py_file)\n except Exception:\n error_msg = traceback.format_exc(0)+'\\n'\n showerror(\"Error\", error_msg)", "response": "Upload current python script to EV3"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nuploads current python script to EV3", "response": "def start_current_script():\n \"\"\"upload current python script to EV3\"\"\"\n current_editor = get_workbench().get_editor_notebook().get_current_editor()\n code = current_editor.get_text_widget().get(\"1.0\", \"end\")\n try:\n ast.parse(code)\n #Return None, if script is not saved and user closed file saving window, otherwise return file name.\n py_file = get_workbench().get_current_editor().save_file(False)\n if py_file is None:\n return\n start(py_file)\n except Exception:\n error_msg = traceback.format_exc(0)+'\\n'\n showerror(\"Error\", error_msg)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef download_log(currentfile=None):\n\n if currentfile == None:\n return\n \n # add \".err.log\" if file doesn't end with it!\n if not currentfile.endswith(\".err.log\"):\n currentfile=currentfile + \".err.log\"\n \n list = get_base_ev3dev_cmd() + ['download','--force']\n list.append(currentfile)\n\n env = os.environ.copy()\n env[\"PYTHONUSERBASE\"] = THONNY_USER_BASE\n proc = subprocess.Popen(list, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,\n universal_newlines=True, env=env)\n dlg = MySubprocessDialog(get_workbench(), proc, \"Downloading log of program from EV3\", autoclose=True)\n dlg.wait_window()\n if dlg.returncode == 0:\n from pathlib import Path\n home = str(Path.home())\n open_file(currentfile,home,True)", "response": "downloads log of given. py file from EV3"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef download_log_of_current_script():\n\n try:\n #Return None, if script is not saved and user closed file saving window, otherwise return file name.\n src_file = get_workbench().get_current_editor().get_filename(False)\n if src_file is None:\n return\n download_log(src_file)\n except Exception:\n error_msg = traceback.format_exc(0)+'\\n'\n showerror(\"Error\", error_msg)", "response": "download log of current python script from EV3"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nloading the plugin and adds buttons on toolbar and commands under Run and Tools menu.", "response": "def load_plugin():\n \"\"\"Adds EV3 buttons on toolbar and commands under Run and Tools menu. Add EV3 configuration window.\"\"\"\n\n # Add EV3 configuration window\n workbench=get_workbench()\n workbench.set_default(\"ev3.ip\", \"192.168.0.1\")\n workbench.set_default(\"ev3.username\", \"robot\")\n workbench.set_default(\"ev3.password\", \"maker\")\n workbench.add_configuration_page(\"EV3\", Ev3ConfigurationPage)\n\n\n # icons\n\n image_path_remoterun = os.path.join(os.path.dirname(__file__), \"res\", \"remoterun.gif\")\n image_path_remotedebug = os.path.join(os.path.dirname(__file__), \"res\", \"remotedebug.gif\")\n\n\n image_path_upload = os.path.join(os.path.dirname(__file__), \"res\", \"up.gif\")\n image_path_run = os.path.join(os.path.dirname(__file__), \"res\", \"flash.gif\")\n image_path_log = os.path.join(os.path.dirname(__file__), \"res\", \"log.gif\")\n\n image_path_clean = os.path.join(os.path.dirname(__file__), \"res\", \"clean.gif\")\n\n\n\n\n\n # menu buttons\n\n get_workbench().add_command(\"ev3remoterun\", \"run\", \"Run current script using the EV3 API in remote control mode\" ,\n get_button_handler_for_magiccmd_on_current_file(\"Ev3RemoteRun\"),\n currentscript_and_command_enabled,\n default_sequence=\"\",\n group=20,\n image_filename=image_path_remoterun,\n include_in_toolbar=True)\n get_workbench().add_command(\"ev3remotedebug\", \"run\", \"Debug current script using the EV3 API in remote control mode\" ,\n get_button_handler_for_magiccmd_on_current_file(\"Ev3RemoteDebug\"),\n currentscript_and_command_enabled,\n default_sequence=\"\",\n group=20,\n image_filename=image_path_remotedebug,\n include_in_toolbar=True)\n\n get_workbench().add_command(\"ev3patch\", \"tools\", \"Install ev3dev additions to the ev3dev sdcard on the EV3\",\n patch_ev3,\n command_enabled,\n default_sequence=None,\n group=270,\n #image_filename=image_path_upload,\n include_in_toolbar=False)\n\n get_workbench().add_command(\"ev3softreset\", \"tools\", \"Soft reset the EV3 (stop programs,rpyc started sound/motors,restart brickman and rpycd service)\",\n soft_reset_ev3,\n command_enabled,\n default_sequence=None,\n group=275,\n #image_filename=image_path_clean,\n include_in_toolbar=False)\n\n\n\n get_workbench().add_command(\"ev3upload\", \"tools\", \"Upload current script to EV3\",\n upload_current_script,\n currentscript_and_command_enabled,\n default_sequence=\"\",\n group=280,\n image_filename=image_path_upload,\n include_in_toolbar=True)\n get_workbench().add_command(\"ev3run\", \"tools\", \"Start current script on the EV3\",\n start_current_script,\n currentscript_and_command_enabled,\n default_sequence=\"\",\n group=280,\n image_filename=image_path_run,\n include_in_toolbar=True)\n get_workbench().add_command(\"ev3log\", \"tools\", \"Download log of current script from EV3\",\n download_log_of_current_script,\n currentscript_and_command_enabled,\n default_sequence=None,\n group=280,\n image_filename=image_path_log,\n include_in_toolbar=True)\n\n get_workbench().add_command(\"ev3clean\", \"tools\", \"Cleanup EV3 by deleting all files stored in homedir on EV3\",\n cleanup_files_on_ev3,\n command_enabled,\n default_sequence=None,\n group=290,\n image_filename=image_path_clean,\n include_in_toolbar=True)\n\n\n\n orig_interrupt_backend=get_runner().interrupt_backend\n def wrapped_interrupt_backend():\n\n # kill program on pc\n orig_interrupt_backend()\n\n # stop programmings running on ev3 and stop sound/motors via rpyc\n stop_ev3_programs__and__rpyc_motors_sound()\n\n\n get_runner().interrupt_backend = wrapped_interrupt_backend\n\n\n\n # magic commands\n shell = get_workbench().get_view(\"ShellView\")\n\n shell.add_command(\"Ev3RemoteRun\", _handle_rundebug_from_shell)\n shell.add_command(\"Ev3RemoteDebug\", _handle_rundebug_from_shell)\n\n\n shell.add_command(\"Reset\", _handle_reset_from_shell)\n\n shell.add_command(\"pwd\", _handle_pwd_from_shell)\n shell.add_command(\"cd\", _handle_cd_from_shell)\n shell.add_command(\"ls\", _handle_ls_from_shell)\n shell.add_command(\"help\", _handle_help_from_shell)\n shell.add_command(\"open\", _handle_open_from_shell)\n shell.add_command(\"reload\", _handle_reload_from_shell)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ninitialize ubersmith API module with HTTP request handler.", "response": "def init(base_url, username=None, password=None, verify=True):\n \"\"\"Initialize ubersmith API module with HTTP request handler.\"\"\"\n handler = RequestHandler(base_url, username, password, verify)\n set_default_request_handler(handler)\n return handler"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef getConfigPath(configFileName = None):\n if configFileName != None:\n # Returning the path of the configuration file\n if sys.platform == 'win32':\n return os.path.expanduser(os.path.join('~\\\\', 'Deepify', configFileName))\n else:\n return os.path.expanduser(os.path.join('~/', '.config', 'Deepify', configFileName))\n else:\n # Returning the path of the configuration folder\n if sys.platform == 'win32':\n return os.path.expanduser(os.path.join('~\\\\', 'Deepify'))\n else:\n return os.path.expanduser(os.path.join('~/', '.config', 'Deepify'))", "response": "Returns the path to the configuration file or folder depending on the system."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreads the configuration file and returns the dictionary containing the information stored in the. cfg file.", "response": "def getConfiguration(configPath = None):\n \"\"\"\n Reading the configuration file to look for where the different gates are running.\n \n :return: A json containing the information stored in the .cfg file.\n \"\"\"\n if configPath == None:\n # If a current.cfg has not been found, creating it by copying from default\n configPath = getConfigPath(\"browser.cfg\")\n\n # Checking if the configuration file exists\n if not os.path.exists(configPath):\n try:\n # Copy the data from the default folder\n defaultConfigPath = getConfigPath(os.path.join(\"default\", \"browser.cfg\"))\n \n with open(configPath, \"w\") as oF:\n with open(defaultConfigPath) as iF:\n cont = iF.read()\n oF.write(cont) \n \n except Exception, e:\n errMsg = \"ERROR. No configuration file could be found and the default file was not found either. You might need to reset it manually.\"\n raise Exception( errMsg + \" \" + str(e)) \n\n try:\n # Reading the configuration file\n config = ConfigParser.ConfigParser()\n config.read(configPath)\n\n info = {}\n\n # Iterating through all the sections, which contain the platforms\n for section in config.sections():\n current = {}\n # Iterating through parametgers\n for (param, value) in config.items(section):\n current[param] = value\n \n # Loading the configuration in the info dictionary\n info[section] = current\n \n except Exception, e:\n errMsg = \"ERROR. Something happened when processing the Configuration file (some kind of malform?). Check it before running it again.\"\n raise Exception( errMsg + \" \" + str(e)) \n\n return info"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating a new logger and log file from an existing logger.", "response": "def new_log_file(logger, suffix, file_type='tcl'):\n \"\"\" Create new logger and log file from existing logger.\n\n The new logger will be create in the same directory as the existing logger file and will be named\n as the existing log file with the requested suffix.\n\n :param logger: existing logger\n :param suffix: string to add to the existing log file name to create the new log file name.\n :param file_type: logger file type (tcl. txt. etc.)\n :return: the newly created logger\n \"\"\"\n\n file_handler = None\n for handler in logger.handlers:\n if isinstance(handler, logging.FileHandler):\n file_handler = handler\n new_logger = logging.getLogger(file_type + suffix)\n if file_handler:\n logger_file_name = path.splitext(file_handler.baseFilename)[0]\n tcl_logger_file_name = logger_file_name + '-' + suffix + '.' + file_type\n new_logger.addHandler(logging.FileHandler(tcl_logger_file_name, 'w'))\n new_logger.setLevel(logger.getEffectiveLevel())\n\n return new_logger"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nprocess a request over HTTP to the Ubersmith instance.", "response": "def process_request(self, method, data=None):\n \"\"\"Process request over HTTP to ubersmith instance.\n\n method: Ubersmith API method string\n data: dict of method arguments\n\n \"\"\"\n # make sure requested method is valid\n self._validate_request_method(method)\n\n # attempt the request multiple times\n attempts = 3\n for i in range(attempts):\n response = self._send_request(method, data)\n\n # handle case where ubersmith is 'updating token'\n # see: https://github.com/jasonkeene/python-ubersmith/issues/1\n if self._is_token_response(response):\n if i < attempts - 1:\n # wait 2 secs before retrying request\n time.sleep(2)\n continue\n else:\n raise UpdatingTokenResponse\n break\n\n resp = BaseResponse(response)\n\n # test for error in json response\n if response.headers.get('content-type') == 'application/json':\n if not resp.json.get('status'):\n if all([\n resp.json.get('error_code') == 1,\n resp.json.get('error_message') == u\"We are currently \"\n \"undergoing maintenance, please check back shortly.\",\n ]):\n raise MaintenanceResponse(response=resp.json)\n else:\n raise ResponseError(response=resp.json)\n return resp"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a nested string representation of the dictionary.", "response": "def dumps(self, indent=1):\n \"\"\" Returns nested string representation of the dictionary (like json.dumps).\n\n :param indent: indentation level.\n \"\"\"\n\n str_keys_dict = OrderedDict({str(k): v for k, v in self.items()})\n for k, v in str_keys_dict.items():\n if isinstance(v, dict):\n str_keys_dict[k] = OrderedDict({str(k1): v1 for k1, v1 in v.items()})\n for k1, v1 in str_keys_dict[k].items():\n if isinstance(v1, dict):\n str_keys_dict[k][k1] = OrderedDict({str(k2): v2 for k2, v2 in v1.items()})\n return json.dumps(str_keys_dict, indent=indent)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the objects of the specified types in the TGN object store.", "response": "def get_objects_by_type(self, *types):\n \"\"\" Returned objects stored in memory (without re-reading them from the TGN).\n\n Use this method for fast access to objects in case of static configurations.\n\n :param types: requested object types.\n :return: all children of the specified types.\n \"\"\"\n\n if not types:\n return self.objects.values()\n types_l = [o.lower() for o in types]\n return [o for o in self.objects.values() if o.obj_type().lower() in types_l]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_object_by_type(self, *types):\n children = self.get_objects_by_type(*types)\n return children[0] if any(children) else None", "response": "get_object_by_type - Gets the first child of the requested object types."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets all objects of the specified types in the tree.", "response": "def get_objects_by_type_in_subtree(self, *types):\n \"\"\"\n :param types: requested object types.\n :return: all children of the specified types.\n \"\"\"\n\n typed_objects = self.get_objects_by_type(*types)\n for child in self.objects.values():\n typed_objects += child.get_objects_by_type_in_subtree(*types)\n return typed_objects"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets objects if children already been read or get children of the requested types.", "response": "def get_objects_or_children_by_type(self, *types):\n \"\"\" Get objects if children already been read or get children.\n\n Use this method for fast access to objects in case of static configurations.\n\n :param types: requested object types.\n :return: all children of the specified types.\n \"\"\"\n\n objects = self.get_objects_by_type(*types)\n return objects if objects else self.get_children(*types)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_object_or_child_by_type(self, *types):\n\n objects = self.get_objects_or_children_by_type(*types)\n return objects[0] if any(objects) else None", "response": "Get object if child already been read or get child."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_objects_with_object(self, obj_type, *child_types):\n\n return [o for o in self.get_objects_by_type(obj_type) if\n o.get_objects_by_type(*child_types)]", "response": "Get all objects of the requested type and all child types."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_objects_without_object(self, obj_type, *child_types):\n return [o for o in self.get_objects_by_type(obj_type) if\n not o.get_objects_by_type(*child_types)]", "response": "Returns a list of objects of the requested type but not including the requested child types."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a list of objects of the requested type with the requested attribute and value.", "response": "def get_objects_with_attribute(self, obj_type, attribute, value):\n \"\"\"\n :param obj_type: requested object type.\n :param attribute: requested attribute.\n :param value: requested attribute value.\n :return: all children of the requested type that have the requested attribute == requested value.\n \"\"\"\n return [o for o in self.get_objects_by_type(obj_type) if o.get_attribute(attribute) == value]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_ancestor_object_by_type(self, obj_type):\n\n if self.type.lower() == obj_type.lower():\n return self\n else:\n if not self.parent:\n return None\n return self.parent.get_ancestor_object_by_type(obj_type)", "response": "returns the ancestor of the object of the requested type."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef del_object_from_parent(self):\n if self.parent:\n self.parent.objects.pop(self.ref)", "response": "Delete object from parent object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_objects_of_class(cls):\n return list(o for o in gc.get_objects() if isinstance(o, cls))", "response": "Returns a list of all objects of the requested class."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncloses (terminates) all of its clients. Though keeps server running.", "response": "def _handle_sighup(myrpcserver, signum, unused):\n \"\"\"Closes (terminates) all of its clients. Though keeps server running.\"\"\"\n print(\"SIGHUP: stopping all clients\",sys.stderr)\n if myrpcserver._closed:\n return\n for c in set(myrpcserver.clients):\n try:\n c.shutdown(socket.SHUT_RDWR)\n except Exception:\n pass\n c.close()\n myrpcserver.clients.clear()"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsplitting a string into two lists of strings.", "response": "def split_strings(string, separators):\n \"\"\"\n Split a string with arbitrary number of separators.\n Return a list with the splitted values\n \n Arguments:\n string (str): ex. \"a:1|2,b:2\"\n separators (list): ex. [',',':','|']\n \n Returns:\n results (list) : ex. ['a','1','2','b','2']\n \"\"\"\n logger = logging.getLogger('extract_vcf.split_strings')\n logger.debug(\"splitting string '{0}' with separators {1}\".format(\n string, separators\n ))\n results = []\n \n def recursion(recursive_string, separators, i=1):\n \"\"\"\n Split a string with arbitrary number of separators.\n Add the elements of the string to global list result.\n \n Arguments:\n string : ex. \"a:1|2,b:2\"\n separators (list): ex. [',',':','|']\n \n Returns:\n Adds splitted string to results. ex. ['a','1','2','b','2']\n \"\"\"\n if i == len(separators):\n for value in recursive_string.split(separators[i-1]):\n logger.debug(\"Adding {0} to results\".format(value))\n results.append(value)\n else:\n for value in recursive_string.split(separators[i-1]):\n recursion(value, separators, i+1)\n if len(separators) > 0:\n recursion(string, separators)\n else:\n results = [string]\n \n return results"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting VCD id in int to string", "response": "def _idToStr(self, x):\n \"\"\"\n Convert VCD id in int to string\n \"\"\"\n if x < 0:\n sign = -1\n elif x == 0:\n return self._idChars[0]\n else:\n sign = 1\n x *= sign\n digits = []\n while x:\n digits.append(self._idChars[x % self._idCharsCnt])\n x //= self._idCharsCnt\n if sign < 0:\n digits.append('-')\n digits.reverse()\n\n return ''.join(digits)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef addVar(self, sig: object, name: str, sigType: VCD_SIG_TYPE, width: int,\n valueFormatter: Callable[[\"Value\"], str]):\n \"\"\"\n Add variable to scope\n\n :ivar sig: user specified object to keep track of VcdVarInfo in change() \n :ivar sigType: vcd type name\n :ivar valueFormatter: value which converts new value in change() to vcd string\n \"\"\"\n vInf = self._writer._idScope.registerVariable(sig, name, self, width,\n sigType, valueFormatter)\n self.children[vInf.name] = vInf\n self._writer._oFile.write(\"$var %s %d %s %s $end\\n\" % (\n sigType, vInf.width, vInf.vcdId, vInf.name))", "response": "Add a variable to the current scope"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a sub variable scope with defined name.", "response": "def varScope(self, name):\n \"\"\"\n Create sub variable scope with defined name\n \"\"\"\n ch = VcdVarWritingScope(name, self._writer, parent=self)\n assert name not in self.children, name\n self.children[name] = ch\n return ch"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef multi_run_wrapper(params):\n '''\n Wrapper for being able to launch all the threads.\n :param params: We receive the parameters as a tuple.\n '''\n zeronet, index, total, output_folder, overwrite = params\n print \"[\" + str(index) + \"/\" + str(total) + \"] \", dt.datetime.now(), \":\\tRecovering information from \", onion\n try: \n # Reading content\n zeronetWrapper = Zeronet()\n \n domain = zeronetWrapper.getDomainFromUrl(zeronet)\n \n # Only doing something if the Json files does NOT exist \n if overwrite or not os.path.exists (os.path.join( output_folder, domain +\".json\")): \n response = zeronetWrapper.getResponse(zeronet)\n\n if response[\"status\"][\"code\"] != 200:\n print dt.datetime.now(), \":\\tSomething happened when launching the query for <\" + zeronet +\">.\\nError message: \" + response[\"status\"][\"desc\"]\n\n print \"[\" + str(index) + \"/\" + str(total) + \"] \", dt.datetime.now(), \":\\tStoring information from\", zeronet\n # Storing the full processed response\n with open(os.path.join( output_folder, response[\"domain\"] +\".json\"), \"w\") as oF:\n try:\n oF.write(json.dumps(response, indent = 2))\n except Exception, e:\n # Grabbing possible exceptions that may occur when using json library...\n oF.write(response) \n\n except:\n print \"ERROR. SOMETHING HAPPENED WITH THE FOLLOWING DOMAIN: \" + zeronet\n with open(\"./errors.log\", \"a\") as errorFile:\n errorFile.write(zeronet+\"\\n\")", "response": "Wrapper for being able to launch all the threads."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef getDomainFromUrl(self, url):\n try:\n domain = re.findall( self.domainRegexp, url )[0]\n except Exception, e:\n errMsg = \"ERROR. Something happened when trying to find the domain from <\" + url + \">. Are you sure that the following regular expression matches a domain in the url provided?\\n\\t\" + self.domainRegexp\n raise Exception( errMsg + \"\\n\" + str(e) )\n \n return domain", "response": "Extract the domain from the URL."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfunction that abstracts capturing a URL. This method will be rewritten in child classes.", "response": "def _grabContentFromUrl(self, url):\n \"\"\" \n Function that abstracts capturing a URL. This method will be rewritten in child classes.\n \n :param url: The URL to be processed.\n :return: The response in a Json format.\n \"\"\"\n # Defining an empty object for the response\n response = {}\n\n # This part has to be modified... \n try:\n import urllib2\n\n # Grabbing the data \n data = urllib2.urlopen(url).read()\n\n # Processing the data as expectesd\n response = self._createDataStructure(data) \n \n # Try to make the errors clear for other users\n except Exception, e:\n errMsg = \"ERROR. Something happened.\" \n raise Exception( errMsg + \" \" + str(e))\n \n return response"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef getResponse(self, url):\n # Defining an empty object for the response\n response = {}\n try:\n # This receives only the parameters needed by this Tor Wrapper:\n # {\n # \"host\" : \"127.0.0.1\"\n # \"port\" : \"9150\"\n # } \n self.info = self._getConfiguration()\n \n response = self._grabContentFromUrl(url)\n except Exception, e:\n response[\"status\"] = { \"code\" : 400, \"desc\" : str(e) } \n \n # Adding other known data \n response[\"time_processed\"] = str(dt.datetime.now())\n \n response[\"domain\"] = self.getDomainFromUrl(url)\n response[\"url\"] = url \n\n try:\n # We'll check if something happened... \n response[\"status\"]\n except Exception, e:\n # If nothing happened till now, we'll set the code as 200 \n response[\"status\"] = { \"code\" : 200, \"desc\" : \"OK.\" }\n \n return response", "response": "This method wraps the extraction of the content and returns the response in a Json format."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nappending query_string values to an existing URL and return it as a string.", "response": "def append_qs(url, query_string):\n \"\"\"Append query_string values to an existing URL and return it as a string.\n\n query_string can be:\n * an encoded string: 'test3=val1&test3=val2'\n * a dict of strings: {'test3': 'val'}\n * a dict of lists of strings: {'test3': ['val1', 'val2']}\n * a list of tuples: [('test3', 'val1'), ('test3', 'val2')]\n\n \"\"\"\n parsed_url = urlsplit(url)\n parsed_qs = parse_qsl(parsed_url.query, True)\n\n if isstr(query_string):\n parsed_qs += parse_qsl(query_string)\n elif isdict(query_string):\n for item in list(query_string.items()):\n if islist(item[1]):\n for val in item[1]:\n parsed_qs.append((item[0], val))\n else:\n parsed_qs.append(item)\n elif islist(query_string):\n parsed_qs += query_string\n else:\n raise TypeError('Unexpected query_string type')\n\n return urlunsplit((\n parsed_url.scheme,\n parsed_url.netloc,\n parsed_url.path,\n urlencode_unicode(parsed_qs),\n parsed_url.fragment,\n ))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef urlencode_unicode(data, doseq=0):\n data_iter = None\n if isdict(data):\n data_iter = list(data.items())\n elif islist(data):\n data_iter = data\n\n if data_iter:\n for i, (key, value) in enumerate(data_iter):\n if isinstance(value, text_type):\n # try to convert to str\n try:\n safe_val = str(value)\n except UnicodeEncodeError:\n # try to encode as utf-8\n # if an exception is raised here then idk what to do\n safe_val = value.encode('utf-8')\n finally:\n if isdict(data):\n data[key] = safe_val\n else:\n data[i] = (key, safe_val)\n\n return urlencode(data, doseq=doseq)", "response": "URL encode unicode data."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef to_nested_php_args(data, prefix_key=None):\n is_root = prefix_key is None\n prefix_key = prefix_key if prefix_key else ''\n\n if islist(data):\n data_iter = data if is_root else enumerate(data)\n new_data = [] if is_root else {}\n elif isdict(data):\n data_iter = list(data.items())\n new_data = {}\n else:\n raise TypeError('expected dict or list, got {0}'.format(type(data)))\n\n if islist(new_data):\n def data_set(k, v):\n new_data.append((k, v))\n def data_update(d):\n for k, v in list(d.items()):\n new_data.append((k, v))\n else:\n def data_set(k, v):\n new_data[k] = v\n data_update = new_data.update\n\n for key, value in data_iter:\n end_key = prefix_key + (str(key) if is_root else '[{0}]'.format(key))\n if _is_leaf(value):\n data_set(end_key, value)\n else:\n nested_args = to_nested_php_args(value, end_key)\n data_update(nested_args)\n\n return new_data", "response": "This function will recursively loop through the values of a dict or list and return a list of all the keys and values in the order. create method."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_filename(disposition):\n if disposition:\n params = [param.strip() for param in disposition.split(';')[1:]]\n for param in params:\n if '=' in param:\n name, value = param.split('=', 1)\n if name == 'filename':\n return value.strip('\"')", "response": "Parse Content - Disposition header to pull out the filename bit."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_args_pairs(arguments):\n\n return ' '.join(' '.join(['-' + k, tcl_str(str(v))]) for k, v in arguments.items())", "response": "Returns a Tcl list of argument pairs."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef tcl_list_2_py_list(tcl_list, within_tcl_str=False):\n\n if not within_tcl_str:\n tcl_list = tcl_str(tcl_list)\n return tcl_interp_g.eval('join ' + tcl_list + ' LiStSeP').split('LiStSeP') if tcl_list else []", "response": "Convert Tcl list to Python list using Tcl interpreter."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert a Python list to Tcl list using Tcl interpreter.", "response": "def py_list_to_tcl_list(py_list):\n \"\"\" Convert Python list to Tcl list using Tcl interpreter.\n\n :param py_list: Python list.\n :type py_list: list\n :return: string representing the Tcl string equivalent to the Python list.\n \"\"\"\n\n py_list_str = [str(s) for s in py_list]\n return tcl_str(tcl_interp_g.eval('split' + tcl_str('\\t'.join(py_list_str)) + '\\\\t'))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef eval(self, command):\n # Some operations (like take ownership) may take long time.\n con_command_out = self._con.send_cmd(command, timeout=256)\n if 'ERROR_SEND_CMD_EXIT_DUE_TO_TIMEOUT' in con_command_out:\n raise Exception('{} - command timeout'.format(command))\n command = command.replace('\\\\', '/')\n con_command_out = con_command_out.replace('\\\\', '/')\n command = command.replace('(', '\\(').replace(')', '\\)')\n command = command.replace('{', '\\{').replace('}', '\\}')\n m = re.search(command + '(.*)' + '%', con_command_out, re.DOTALL)\n command_out = m.group(1).strip()\n if 'couldn\\'t read file' in command_out or 'RuntimeError' in command_out:\n raise Exception(command_out)\n return command_out", "response": "@summary: Evaluate Tcl command.\n\n @param command: command to evaluate.\n @return: command output."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef eval(self, command):\n\n if self.logger.handlers:\n self.logger.debug(command.decode('utf-8'))\n if self.tcl_script:\n self.tcl_script.info(command)\n self.rc = self.tcl_interp.eval(command)\n if self.logger.handlers:\n self.logger.debug('\\t' + self.rc.decode('utf-8'))\n return self.rc", "response": "Execute Tcl command and return the command raw output."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfunctioning that abstracts capturing a URL. This method rewrites the one from Wrapper. :param url: The URL to be processed. :return: The response in a Json format.", "response": "def _grabContentFromUrl(self, url):\n \"\"\" \n Function that abstracts capturing a URL. This method rewrites the one from Wrapper.\n \n :param url: The URL to be processed.\n :return: The response in a Json format.\n \"\"\"\n # Defining an empty object for the response\n response = {}\n \n # This part has to be modified... \n try: \n # Configuring the socket\n socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, self.info[\"host\"], int(self.info[\"port\"]), True)\n s = socks.socksocket()\n \n # Extracting the domain from the URL\n domain = self.getDomainFromUrl(url) \n s.connect((domain, 80))\n\n message = 'GET ' + url + ' HTTP/1.0\\r\\n\\r\\n'\n s.sendall(message)\n \n data = \"\"\n while True:\n reply = s.recv(4096)\n\n if not reply:\n break \n else:\n data += reply \n \n # Processing data as expected\n response = self._createDataStructure(data) \n \n # Try to make the errors clear for other users\n except socks.ProxyConnectionError, sPCE:\n errMsg = \"ERROR socks.ProxyConnectionError. Something seems to be wrong with the Tor Bundler.\" \n raise Exception( errMsg + \" \" + str(sPCE)) \n \n return response"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef value_change(self, vcdId, value):\n '''append change from VCD file signal data series'''\n self.idcode2series[vcdId].append((self.now, value))", "response": "append change from VCD file signal data series"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef edit_record(self, new_record):\n try:\n record_id = new_record[\"id\"]\n except KeyError:\n raise ValueError(\"No record ID provided!\")\n\n record_type = self.resource_type(record_id)\n if record_type is None:\n raise ArchivistsToolkitError(\n \"Could not determine type for record with ID {}; not in database?\".format(\n record_id\n )\n )\n\n clause = []\n values = []\n if \"title\" in new_record:\n clause.append(\"title=%s\")\n values.append(new_record[\"title\"])\n if \"levelOfDescription\" in new_record:\n clause.append(\"resourceLevel=%s\")\n values.append(new_record[\"levelOfDescription\"])\n\n # nothing to update\n if not clause:\n raise ValueError(\"No fields to update specified!\")\n\n clause = \", \".join(clause)\n if record_type == ArchivistsToolkitClient.RESOURCE:\n db_type = \"Resources\"\n db_id_field = \"resourceId\"\n else:\n db_type = \"ResourcesComponents\"\n db_id_field = \"resourceComponentId\"\n sql = \"UPDATE {} SET {} WHERE {}=%s\".format(db_type, clause, db_id_field)\n cursor = self.db.cursor()\n cursor.execute(sql, tuple(values))", "response": "Edit a record in Archivist s Toolkit using the provided new_record."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_levels_of_description(self):\n if not hasattr(self, \"levels_of_description\"):\n cursor = self.db.cursor()\n levels = set()\n cursor.execute(\"SELECT distinct(resourceLevel) FROM Resources\")\n for row in cursor:\n levels.add(row)\n cursor.execute(\"SELECT distinct(resourceLevel) FROM ResourcesComponents\")\n for row in cursor:\n levels.add(row)\n self.levels_of_description = list(levels)\n\n return self.levels_of_description", "response": "Returns an array of all levels of description defined in this Archivist s Toolkit instance."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfetching a list of all resource and component IDs within the specified resource.", "response": "def collection_list(self, resource_id, resource_type=\"collection\"):\n \"\"\"\n Fetches a list of all resource and component IDs within the specified resource.\n\n :param long resource_id: The ID of the resource to fetch children from.\n :param string resource_type: Specifies whether the resource to fetch is a collection or a child element.\n Defaults to 'collection'.\n\n :return: A list of longs representing the database resource IDs for all children of the requested record.\n :rtype list:\n \"\"\"\n ret = []\n\n cursor = self.db.cursor()\n if resource_type == \"collection\":\n cursor.execute(\n \"SELECT resourceComponentId FROM ResourcesComponents WHERE parentResourceComponentId IS NULL AND resourceId=%s\",\n (resource_id),\n )\n else:\n ret.append(resource_id)\n cursor.execute(\n \"SELECT resourceComponentId FROM ResourcesComponents WHERE parentResourceComponentId=%s\",\n (resource_id),\n )\n\n rows = cursor.fetchall()\n if len(rows):\n for row in rows:\n ret.extend(self.collection_list(row[0], \"description\"))\n\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_resource_component_and_children(\n self, resource_id, resource_type=\"collection\", level=1, sort_data={}, **kwargs\n ):\n \"\"\"\n Fetch detailed metadata for the specified resource_id and all of its children.\n\n :param long resource_id: The resource for which to fetch metadata.\n :param string resource_type: The level of description of the record.\n :param int recurse_max_level: The maximum depth level to fetch when fetching children.\n Default is to fetch all of the resource's children, descending as deeply as necessary.\n Pass 1 to fetch no children.\n :param string search_pattern: If specified, limits fetched children to those whose titles or IDs match the provided query.\n See ArchivistsToolkitClient.find_collection_ids for documentation of the query format.\n\n :return: A dict containing detailed metadata about both the requested resource and its children.\n The dict follows this format:\n {\n 'id': '31',\n 'type': 'resource',\n 'sortPosition': '1',\n 'identifier': 'PR01',\n 'title': 'Parent',\n 'levelOfDescription': 'collection',\n 'dates': '1880-1889',\n 'date_expression': '1880 to 1889',\n 'notes': [\n 'type': 'odd',\n 'content': 'This is a note',\n ],\n 'children': [{\n 'id': '23',\n 'type': 'resource_component',\n 'sortPosition': '2',\n 'identifier': 'CH01',\n 'title': 'Child A',\n 'levelOfDescription': 'Sousfonds',\n 'dates': '1880-1888',\n 'date_expression': '1880 to 1888',\n 'notes': [],\n 'children': [{\n 'id': '24',\n 'type': 'resource_component',\n 'sortPosition': '3',\n 'identifier': 'GR01',\n 'title': 'Grandchild A',\n 'levelOfDescription': 'Item',\n 'dates': '1880-1888',\n 'date_expression': '1880 to 1888',\n 'notes': [],\n 'children': False\n },\n {\n 'id': '25',\n 'type': 'resource_component',\n 'sortPosition': '4',\n 'identifier': 'GR02',\n 'title': 'Grandchild B',\n 'levelOfDescription': 'Item',\n 'notes': [],\n 'children': False\n }]\n },\n {\n 'id': '26',\n 'type': 'resource_component',\n 'sortPosition': '5',\n 'identifier': 'CH02',\n 'title': 'Child B',\n 'levelOfDescription': 'Sousfonds',\n 'dates': '1889',\n 'date_expression': '1889',\n 'notes': [],\n 'children': False\n }]\n }\n :rtype list:\n \"\"\"\n # we pass the sort position as a dict so it passes by reference and we\n # can use it to share state during recursion\n\n recurse_max_level = kwargs.get(\"recurse_max_level\", False)\n query = kwargs.get(\"search_pattern\", \"\")\n\n # intialize sort position if this is the beginning of recursion\n if level == 1:\n sort_data[\"position\"] = 0\n\n sort_data[\"position\"] = sort_data[\"position\"] + 1\n\n resource_data = {}\n\n cursor = self.db.cursor()\n\n if resource_type == \"collection\":\n cursor.execute(\n \"SELECT title, dateExpression, resourceIdentifier1, resourceLevel FROM Resources WHERE resourceid=%s\",\n (resource_id),\n )\n\n for row in cursor.fetchall():\n resource_data[\"id\"] = resource_id\n resource_data[\"type\"] = \"resource\"\n resource_data[\"sortPosition\"] = sort_data[\"position\"]\n resource_data[\"title\"] = row[0]\n # TODO reformat dates from the separate date fields, like ArchivesSpaceClient?\n resource_data[\"dates\"] = row[1]\n resource_data[\"date_expression\"] = row[1]\n resource_data[\"identifier\"] = row[2]\n resource_data[\"levelOfDescription\"] = row[3]\n else:\n cursor.execute(\n \"SELECT title, dateExpression, persistentID, resourceLevel FROM ResourcesComponents WHERE resourceComponentId=%s\",\n (resource_id),\n )\n\n for row in cursor.fetchall():\n resource_data[\"id\"] = resource_id\n resource_data[\"type\"] = \"resource_component\"\n resource_data[\"sortPosition\"] = sort_data[\"position\"]\n resource_data[\"title\"] = row[0]\n resource_data[\"dates\"] = row[1]\n resource_data[\"date_expression\"] = row[1]\n resource_data[\"identifier\"] = row[2]\n resource_data[\"levelOfDescription\"] = row[3]\n\n # fetch children if we haven't reached the maximum recursion level\n if resource_type == \"collection\":\n if query == \"\":\n cursor.execute(\n \"SELECT resourceComponentId FROM ResourcesComponents WHERE parentResourceComponentId IS NULL AND resourceId=%s ORDER BY FIND_IN_SET(resourceLevel, 'subseries,file'), title ASC\",\n (resource_id),\n )\n else:\n cursor.execute(\n \"SELECT resourceComponentId FROM ResourcesComponents WHERE parentResourceComponentId IS NULL AND resourceId=%s AND (title LIKE %s OR persistentID LIKE %s) ORDER BY FIND_IN_SET(resourceLevel, 'subseries,file'), title ASC\",\n (resource_id, \"%\" + query + \"%\", \"%\" + query + \"%\"),\n )\n else:\n if query == \"\":\n cursor.execute(\n \"SELECT resourceComponentId FROM ResourcesComponents WHERE parentResourceComponentId=%s ORDER BY FIND_IN_SET(resourceLevel, 'subseries,file'), title ASC\",\n (resource_id),\n )\n else:\n cursor.execute(\n \"SELECT resourceComponentId FROM ResourcesComponents WHERE parentResourceComponentId=%s AND (title LIKE %s OR persistentID LIKE %s) ORDER BY FIND_IN_SET(resourceLevel, 'subseries,file'), title ASC\",\n (resource_id, \"%\" + query + \"%\", \"%\" + query + \"%\"),\n )\n\n rows = cursor.fetchall()\n\n if (not recurse_max_level) or level < recurse_max_level:\n if len(rows):\n resource_data[\"children\"] = []\n resource_data[\"has_children\"] = True\n\n for row in rows:\n resource_data[\"children\"].append(\n self.get_resource_component_and_children(\n row[0], \"description\", level + 1, sort_data\n )\n )\n else:\n if len(rows):\n resource_data[\"children\"] = []\n resource_data[\"has_children\"] = True\n else:\n resource_data[\"children\"] = False\n resource_data[\"has_children\"] = False\n\n # TODO: implement fetching notes\n resource_data[\"notes\"] = []\n\n return resource_data", "response": "This function returns detailed metadata about the specified resource and its children."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfind the parent resource ID for a given component.", "response": "def find_resource_id_for_component(self, component_id):\n \"\"\"\n Given the ID of a component, returns the parent resource ID.\n\n If the immediate parent of the component is itself a component, this method will progress up the tree until a resource is found.\n\n :param long component_id: The ID of the ResourceComponent.\n :return: The ID of the component's parent resource.\n :rtype: long\n \"\"\"\n cursor = self.db.cursor()\n\n sql = \"SELECT resourceId, parentResourceComponentId FROM ResourcesComponents WHERE resourceComponentId=%s\"\n cursor.execute(sql, (component_id,))\n resource_id, parent_id = cursor.fetchone()\n\n if resource_id is None:\n return self.find_resource_id_for_component(parent_id)\n else:\n return resource_id"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngive the ID of a component returns the parent component s ID.", "response": "def find_parent_id_for_component(self, component_id):\n \"\"\"\n Given the ID of a component, returns the parent component's ID.\n\n :param string component_id: The ID of the component.\n :return: A tuple containing:\n * The type of the parent record; valid values are ArchivesSpaceClient.RESOURCE and ArchivesSpaceClient.RESOURCE_COMPONENT.\n * The ID of the parent record.\n :rtype tuple:\n \"\"\"\n cursor = self.db.cursor()\n\n sql = \"SELECT parentResourceComponentId FROM ResourcesComponents WHERE resourceComponentId=%s\"\n count = cursor.execute(sql, (component_id,))\n if count > 0:\n return (ArchivistsToolkitClient.RESOURCE_COMPONENT, cursor.fetchone())\n\n return (\n ArchivistsToolkitClient.RESOURCE,\n self.find_resource_id_for_component(component_id),\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef find_collection_ids(\n self, search_pattern=\"\", identifier=\"\", page=None, page_size=30\n ):\n \"\"\"\n Fetches a list of all resource IDs for every resource in the database.\n\n :param string search_pattern: A search pattern to use in looking up resources by title or resourceid.\n The search will match any title or resourceid containing this string;\n for example, \"text\" will match \"this title has this text in it\".\n If omitted, then all resources will be fetched.\n :param string identifier: Restrict records to only those with this identifier.\n This refers to the human-assigned record identifier, not the automatically generated internal ID.\n Unlike the ArchivesSpaceClient version of this method, wildcards are not supported; however, identifiers which begin or end with this string will be returned.\n For example, if the passed identifier is \"A1\", then records with an identifier of \"A1\", \"SAA1\", \"A10\", and \"SAA10\" will all be returned.\n\n :return: A list containing every matched resource's ID.\n :rtype: list\n \"\"\"\n cursor = self.db.cursor()\n\n if search_pattern == \"\" and identifier == \"\":\n sql = \"SELECT resourceId FROM Resources ORDER BY title\"\n params = ()\n else:\n clause = \"resourceid LIKE %s\"\n params = [\"%\" + search_pattern + \"%\"]\n\n if search_pattern != \"\":\n clause = \"title LIKE %s OR\" + clause\n params.insert(0, \"%\" + search_pattern + \"%\")\n\n if identifier != \"\":\n clause = \"resourceIdentifier1 LIKE %s OR \" + clause\n params.insert(0, \"%\" + identifier + \"%\")\n\n params = tuple(params)\n\n sql = \"SELECT resourceId FROM Resources WHERE ({}) AND resourceLevel in ('recordgrp', 'collection') ORDER BY title\".format(\n clause\n )\n\n if page is not None:\n start = (page - 1) * page_size\n sql = sql + \" LIMIT {},{}\".format(start, page_size)\n\n cursor.execute(sql, params)\n\n return [r[0] for r in cursor]", "response": "This method returns a list of all resource IDs for every resource in the database."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nevaluates the Gaussian model for a specific entry.", "response": "def evaluate(self, flux, xo, yo, a, b, c):\n \"\"\"\n Evaluate the Gaussian model\n\n Parameters\n ----------\n flux : tf.Variable\n xo, yo : tf.Variable, tf.Variable\n Center coordiantes of the Gaussian.\n a, b, c : tf.Variable, tf.Variable\n Parameters that control the rotation angle\n and the stretch along the major axis of the Gaussian,\n such that the matrix M = [a b ; b c] is positive-definite.\n\n References\n ----------\n https://en.wikipedia.org/wiki/Gaussian_function#Two-dimensional_Gaussian_function\n \"\"\"\n dx = self.x - xo\n dy = self.y - yo\n psf = tf.exp(-(a * dx ** 2 + 2 * b * dx * dy + c * dy ** 2))\n psf_sum = tf.reduce_sum(psf)\n return flux * psf / psf_sum"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nparse a line from a bcbio - nextgen log file and returns it with more information", "response": "def parse_log_line(line):\n \"\"\"Parses a log line and returns it with more information\n\n :param line: str - A line from a bcbio-nextgen log\n :returns dict: A dictionary containing the line, if its a new step if its a Traceback or if the\n analysis is finished\n \"\"\"\n matches = re.search(r'^\\[([^\\]]+)\\] ([^:]+: .*)', line)\n error = re.search(r'Traceback', line)\n if error:\n return {'line': line, 'step': 'error'}\n if not matches:\n return {'line': line, 'step': None}\n\n tstamp = matches.group(1)\n msg = matches.group(2)\n\n if not msg.find('Timing: ') >= 0:\n return {'line': line, 'step': None}\n\n when = datetime.strptime(tstamp, '%Y-%m-%dT%H:%MZ').replace(\n tzinfo=pytz.timezone('UTC'))\n step = msg.split(\":\")[-1].strip()\n return {'line': line, 'step': step, 'when': when}"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a dictionary representation of the Event appropriate for JSON - encoding.", "response": "def message(self):\n \"\"\"Dictionary representation of the Event appropriate for JSON-encoding.\"\"\"\n return {\n 'timestamp': time.mktime(self.timestamp.timetuple())*1e3 + self.timestamp.microsecond/1e3,\n 'group': self.group_pk,\n 'participant': None if not self.participant else self.participant.code,\n 'channel': self.channel,\n 'value': self.value\n }"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef save(self, *args, **kwargs):\n if self.timestamp is None:\n self.timestamp = timezone.now()\n\n super().save(*args, **kwargs)", "response": "Save an Event automatically sets the timestamp if not already set."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _on_connect(self, participant):\n lock = get_redis_lock()\n if not lock:\n lock = fake_lock()\n\n with lock:\n self.refresh_from_db()\n if self.ran_ready_function:\n return\n for player in self.get_players():\n if Connection.objects.filter(participant__code=player.participant.code).count() == 0:\n return\n \n self.when_all_players_ready()\n self.ran_ready_function = timezone.now()\n self.save()\n\n self.send('state', 'period_start')\n\n if self.period_length():\n # TODO: Should replace this with something like Huey/Celery so it'll survive a server restart.\n self._timer = threading.Timer(\n self.period_length(),\n lambda: self.send('state', 'period_end'))\n self._timer.start()", "response": "Called by the WebSocket consumer when all players have connected."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _on_disconnect(self, participant):\n player = None\n for p in self.get_players():\n if p.participant == participant:\n player = p\n break\n self.when_player_disconnects(player)", "response": "Trigger the : meth : when_player_disconnects callback."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsend a message with the given payload on the given channel.", "response": "def send(self, channel, payload):\n \"\"\"Send a message with the given payload on the given channel.\n Messages are broadcast to all players in the group.\n \"\"\"\n with track('send_channel=' + channel):\n with track('create event'):\n Event.objects.create(\n group=self,\n channel=channel,\n value=payload)\n ChannelGroup(str(self.pk)).send(\n {'text': json.dumps({\n 'channel': channel,\n 'payload': payload\n })})"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef save(self, *args, **kwargs):\n super().save(*args, **kwargs)\n if self.pk is not None:\n update_fields = kwargs.get('update_fields')\n json_fields = {}\n for field in self._meta.get_fields():\n if isinstance(field, JSONField) and (update_fields is None or field.attname in update_fields):\n json_fields[field.attname] = getattr(self, field.attname)\n self.__class__._default_manager.filter(pk=self.pk).update(**json_fields)", "response": "Save the object to the database."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncalls by the AAF manager when all players are ready to run the main task.", "response": "def when_all_players_ready(self):\n \"\"\"Initializes decisions based on ``player.initial_decision()``.\n If :attr:`num_subperiods` is set, starts a timed task to run the\n sub-periods.\n \"\"\"\n self.group_decisions = {}\n self.subperiod_group_decisions = {}\n for player in self.get_players():\n self.group_decisions[player.participant.code] = player.initial_decision()\n self.subperiod_group_decisions[player.participant.code] = player.initial_decision()\n if self.num_subperiods():\n emitter = DiscreteEventEmitter(\n self.period_length() / self.num_subperiods(), \n self.period_length(),\n self,\n self._subperiod_tick)\n emitter.start()\n elif self.rate_limit():\n def _tick(current_interval, intervals):\n self.refresh_from_db()\n if self._group_decisions_updated:\n self.send('group_decisions', self.group_decisions)\n self._group_decisions_updated = False\n self.save(update_fields=['_group_decisions_updated'])\n\n update_period = self.rate_limit()\n emitter = DiscreteEventEmitter(\n update_period, \n self.period_length(),\n self,\n _tick)\n emitter.start()\n self.save()"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nticks each sub - period copying group_decisions to subperiod_group_decisions.", "response": "def _subperiod_tick(self, current_interval, intervals):\n \"\"\"Tick each sub-period, copying group_decisions to subperiod_group_decisions.\"\"\" \n self.refresh_from_db()\n for key, value in self.group_decisions.items():\n self.subperiod_group_decisions[key] = value\n self.send('group_decisions', self.subperiod_group_decisions)\n self.save(update_fields=['subperiod_group_decisions'])"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncalls when an event is received on the decisions channel. Saves the value in group_decisions and sends the event back out on the group_decisions channel.", "response": "def _on_decisions_event(self, event=None, **kwargs):\n \"\"\"Called when an Event is received on the decisions channel. Saves\n the value in group_decisions. If num_subperiods is None, immediately\n broadcasts the event back out on the group_decisions channel.\n \"\"\"\n if not self.ran_ready_function:\n logger.warning('ignoring decision from {} before when_all_players_ready: {}'.format(event.participant.code, event.value))\n return\n with track('_on_decisions_event'):\n self.group_decisions[event.participant.code] = event.value\n self._group_decisions_updated = True\n self.save(update_fields=['group_decisions', '_group_decisions_updated'])\n if not self.num_subperiods() and not self.rate_limit():\n self.send('group_decisions', self.group_decisions)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a copy of given matrix where keys associated to space values are discarded", "response": "def clean(matrix):\n \"\"\"Return a copy of given matrix where keys associated\n to space values are discarded\"\"\"\n return defaultdict(lambda: ' ', {\n k: v for k, v in matrix.items() if v != ' '\n })"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef build(matrix):\n max_x = max(matrix, key=lambda t: t[0])[0]\n min_x = min(matrix, key=lambda t: t[0])[0]\n max_y = max(matrix, key=lambda t: t[1])[1]\n min_y = min(matrix, key=lambda t: t[1])[1]\n yield from (\n # '{}:'.format(j).ljust(4) + ''.join(matrix[i, j] for i in range(min_x, max_x+1))\n ''.join(matrix[i, j] for i in range(min_x, max_x+1))\n for j in range(min_y, max_y+1)\n )", "response": "Yield lines generated from given matrix"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the position of the next unwrittable letter on a row.", "response": "def next_unwrittable_on_row(view, coords):\n \"\"\"Return position of the next (in row) letter that is unwrittable\"\"\"\n x, y = coords\n maxx = max(view.keys(), key=itemgetter(0))[0]\n for offset in range(x + 1, maxx):\n letter = view[offset, y]\n if letter not in REWRITABLE_LETTERS:\n return offset\n return None"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef next_unwrittable_on_col(view, coords):\n x, y = coords\n maxy = max(view.keys(), key=itemgetter(1))[1]\n for offset in range(y + 1, maxy):\n letter = view[x, offset]\n if letter not in REWRITABLE_LETTERS:\n return offset\n return None", "response": "Return the position of the next unwrittable letter in the column coords."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef previous_unwrittable_on_row(view, coords):\n x, y = coords\n minx = -1\n for offset in range(x - 1, minx, -1):\n letter = view[offset, y]\n if letter not in REWRITABLE_LETTERS:\n return offset\n return None", "response": "Return the position of the previous unwrittable letter on a given row."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the position of the previous unwrittable letter that is unwrittable on the column coords.", "response": "def previous_unwrittable_on_col(view, coords):\n \"\"\"Return position of the previous (in column) letter that is unwrittable\"\"\"\n x, y = coords\n miny = -1\n for offset in range(y - 1, miny, -1):\n letter = view[x, offset]\n if letter not in REWRITABLE_LETTERS:\n return offset\n return None"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _build_base_url(self, host, port):\n parsed = urlparse(host)\n if not parsed.scheme:\n parsed = parsed._replace(scheme=\"http\")\n parsed = parsed._replace(path=\"\")\n netloc, parts = host, host.partition(\":\")\n if parts[1] == \"\" and port is not None:\n netloc = \"{}:{}\".format(parts[0], port)\n parsed = parsed._replace(netloc=netloc)\n parsed = parsed._replace(path=parsed.path.rstrip(\"/\"))\n return parsed.geturl()", "response": "Build the base URL string based on host and port."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nextract notes from a record and reformats them in a simplified format.", "response": "def _format_notes(self, record):\n \"\"\"\n Extracts notes from a record and reformats them in a simplified format.\n \"\"\"\n notes = []\n for note in record[\"notes\"]:\n if note.get(\"type\"):\n n = {}\n n[\"type\"] = note[\"type\"]\n try:\n if note[\"jsonmodel_type\"] == \"note_singlepart\":\n n[\"content\"] = note[\"content\"][0]\n else:\n n[\"content\"] = note[\"subnotes\"][0][\"content\"]\n except (IndexError, KeyError):\n n[\"content\"] = \"\"\n\n notes.append(n)\n\n return notes"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nprocess the notes field in the record and update the internal notes property with the new_record.", "response": "def _process_notes(record, new_record):\n \"\"\"\n Populate the notes property using the provided new_record.\n\n If the new_record field was populated, assume that we want to replace\n the notes. If there are valid changes to be made, they will be added to\n the new_notes list. An empty list is counted as a request to delete all\n notes.\n\n Returns a boolean indicating whether changes were made.\n \"\"\"\n if \"notes\" not in new_record or not new_record[\"notes\"]:\n return False\n\n # This assumes any notes passed into the edit record are intended to\n # replace the existing set.\n new_notes = []\n for note in new_record[\"notes\"]:\n # Whitelist of supported types of notes to edit\n # A note with an empty string as content is counted as a request to\n # delete the note, and will not be added to the list.\n if note[\"type\"] in (\"odd\", \"accessrestrict\") and note.get(\"content\"):\n new_notes.append(\n {\n \"jsonmodel_type\": \"note_multipart\",\n \"publish\": True,\n \"subnotes\": [\n {\n \"content\": note[\"content\"],\n \"jsonmodel_type\": \"note_text\",\n \"publish\": True,\n }\n ],\n \"type\": note[\"type\"],\n }\n )\n\n record[\"notes\"] = new_notes\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nescaping special characters in Solr queries.", "response": "def _escape_solr_query(query, field=\"title\"):\n \"\"\"\n Escapes special characters in Solr queries.\n Note that this omits * - this is intentionally permitted in user queries.\n The list of special characters is located at http://lucene.apache.org/core/4_0_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#Escaping_Special_Characters\n \"\"\"\n # Different rules for \"title\" and \"identifier\" fields :/\n if field == \"title\":\n replacement = r\"\\\\\\\\\\1\"\n else:\n replacement = r\"\\\\\\1\"\n\n return re.sub(r'([\\'\" +\\-!\\(\\)\\{\\}\\[\\]^\"~?:\\\\/]|&&|\\|\\|)', replacement, query)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndetermines whether a given resource is a resource or a resource_component.", "response": "def resource_type(self, resource_id):\n \"\"\"\n Given an ID, determines whether a given resource is a resource or a resource_component.\n\n :param resource_id string: The URI of the resource whose type to determine.\n :raises ArchivesSpaceError: if the resource_id does not appear to be either type.\n \"\"\"\n match = re.search(\n r\"repositories/\\d+/(resources|archival_objects)/\\d+\", resource_id\n )\n if match and match.groups():\n type_ = match.groups()[0]\n return \"resource\" if type_ == \"resources\" else \"resource_component\"\n else:\n raise ArchivesSpaceError(\n \"Unable to determine type of provided ID: {}\".format(resource_id)\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef edit_record(self, new_record):\n try:\n record_id = new_record[\"id\"]\n except KeyError:\n raise ValueError(\"No record ID provided!\")\n\n record = self.get_record(record_id)\n\n # TODO: add more fields?\n field_map = {\"title\": \"title\", \"level\": \"levelOfDescription\"}\n fields_updated = False\n for field, targetfield in field_map.items():\n try:\n record[targetfield] = new_record[field]\n fields_updated = True\n except KeyError:\n continue\n\n if self._process_notes(record, new_record):\n fields_updated = True\n\n # Create dates object if any of the date fields is populated\n if (\n \"start_date\" in new_record\n or \"end_date\" in new_record\n or \"date_expression\" in new_record\n ):\n date = {\n \"jsonmodel_type\": \"date\",\n \"date_type\": \"inclusive\",\n \"label\": \"creation\",\n }\n if \"date_expression\" in new_record:\n date[\"expression\"] = new_record[\"date_expression\"]\n if \"start_date\" in new_record:\n date[\"begin\"] = new_record[\"start_date\"]\n if \"end_date\" in new_record:\n date[\"end\"] = new_record[\"end_date\"]\n\n if len(record[\"dates\"]) == 0:\n record[\"dates\"] = [date]\n else:\n record[\"dates\"][0] = date\n\n fields_updated = True\n\n if not fields_updated:\n raise ValueError(\"No fields to update specified!\")\n\n self._post(record_id, data=json.dumps(record))", "response": "Updates a record in ArchivesSpace using the provided new_record."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_levels_of_description(self):\n if not hasattr(self, \"levels_of_description\"):\n # TODO: * fetch human-formatted strings\n # * is hardcoding this ID okay?\n self.levels_of_description = self._get(\"/config/enumerations/32\").json()[\n \"values\"\n ]\n\n return self.levels_of_description", "response": "Returns an array of all levels of description defined in this ArcivesSpace instance."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget detailed metadata for a resource component and all of its children.", "response": "def get_resource_component_children(self, resource_component_id):\n \"\"\"\n Given a resource component, fetches detailed metadata for it and all of its children.\n\n This is implemented using ArchivesSpaceClient.get_resource_component_children and uses its default options when fetching children.\n\n :param string resource_component_id: The URL of the resource component from which to fetch metadata.\n \"\"\"\n resource_type = self.resource_type(resource_component_id)\n return self.get_resource_component_and_children(\n resource_component_id, resource_type\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_resource_component_and_children(\n self,\n resource_id,\n resource_type=\"collection\",\n level=1,\n sort_data={},\n recurse_max_level=False,\n sort_by=None,\n **kwargs\n ):\n \"\"\"\n Fetch detailed metadata for the specified resource_id and all of its children.\n\n :param long resource_id: The resource for which to fetch metadata.\n :param str resource_type: no-op; not required or used in this implementation.\n :param int recurse_max_level: The maximum depth level to fetch when fetching children.\n Default is to fetch all of the resource's children, descending as deeply as necessary.\n Pass 1 to fetch no children.\n :param string search_pattern: If specified, limits fetched children to those whose titles or IDs match the provided query.\n See ArchivistsToolkitClient.find_collection_ids for documentation of the query format.\n\n :return: A dict containing detailed metadata about both the requested resource and its children.\n Consult ArchivistsToolkitClient.get_resource_component_and_children for the output format.\n :rtype dict:\n \"\"\"\n resource_type = self.resource_type(resource_id)\n if resource_type == \"resource\":\n return self._get_resources(\n resource_id, recurse_max_level=recurse_max_level, sort_by=sort_by\n )\n else:\n return self._get_components(\n resource_id, recurse_max_level=recurse_max_level, sort_by=sort_by\n )", "response": "This function returns detailed metadata about the specified resource and all of its children."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef find_parent_id_for_component(self, component_id):\n response = self.get_record(component_id)\n if \"parent\" in response:\n return (ArchivesSpaceClient.RESOURCE_COMPONENT, response[\"parent\"][\"ref\"])\n # if this is the top archival object, return the resource instead\n elif \"resource\" in response:\n return (ArchivesSpaceClient.RESOURCE, response[\"resource\"][\"ref\"])\n # resource was passed in, which has no higher-up record;\n # return the same ID\n else:\n return (ArchivesSpaceClient.RESOURCE, component_id)", "response": "Given the URL to a component returns the parent component s URL."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef find_collection_ids(self, search_pattern=\"\", identifier=\"\", fetched=0, page=1):\n params = {\"page\": page, \"q\": \"primary_type:resource\"}\n\n if search_pattern != \"\":\n search_pattern = self._escape_solr_query(search_pattern, field=\"title\")\n params[\"q\"] = params[\"q\"] + \" AND title:{}\".format(search_pattern)\n\n if identifier != \"\":\n identifier = self._escape_solr_query(identifier, field=\"identifier\")\n params[\"q\"] = params[\"q\"] + \" AND identifier:{}\".format(identifier)\n\n response = self._get(self.repository + \"/search\", params=params)\n hits = response.json()\n results = [r[\"uri\"] for r in hits[\"results\"]]\n\n results_so_far = fetched + hits[\"this_page\"]\n if hits[\"total_hits\"] > results_so_far:\n results.extend(\n self.find_collection_ids(fetched=results_so_far, page=page + 1)\n )\n\n return results", "response": "Fetches a list of resource URLs for every resource in the database."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef find_collections(\n self,\n search_pattern=\"\",\n identifier=\"\",\n fetched=0,\n page=1,\n page_size=30,\n sort_by=None,\n ):\n \"\"\"\n Fetches a list of all resource IDs for every resource in the database.\n\n :param string search_pattern: A search pattern to use in looking up resources by title or resourceid.\n The search will match any title or resourceid containing this string;\n for example, \"text\" will match \"this title has this text in it\".\n If omitted, then all resources will be fetched.\n :param string identifier: Restrict records to only those with this identifier.\n This refers to the human-assigned record identifier, not the automatically generated internal ID.\n This value can contain wildcards.\n\n :return: A list containing every matched resource's ID.\n :rtype: list\n \"\"\"\n\n def format_record(record):\n dates = self._fetch_dates_from_record(record)\n date_expression = self._fetch_date_expression_from_record(record)\n identifier = (\n record[\"id_0\"] if \"id_0\" in record else record.get(\"component_id\", \"\")\n )\n\n has_children = (\n len(\n self._get(record[\"uri\"] + \"/tree\", params={\"page\": 1}).json()[\n \"children\"\n ]\n )\n > 0\n )\n\n return {\n \"id\": record[\"uri\"],\n \"type\": \"resource\",\n \"sortPosition\": 1,\n \"identifier\": identifier,\n \"title\": record.get(\"title\", \"\"),\n \"dates\": dates,\n \"date_expression\": date_expression,\n \"levelOfDescription\": record[\"level\"],\n \"children\": [] if has_children else False,\n \"has_children\": has_children,\n \"notes\": self._format_notes(record),\n }\n\n params = {\"page\": page, \"page_size\": page_size, \"q\": \"primary_type:resource\"}\n\n if search_pattern != \"\":\n search_pattern = self._escape_solr_query(search_pattern, field=\"title\")\n params[\"q\"] = params[\"q\"] + \" AND title:{}\".format(search_pattern)\n\n if identifier != \"\":\n identifier = self._escape_solr_query(identifier, field=\"identifier\")\n params[\"q\"] = params[\"q\"] + \" AND identifier:{}\".format(identifier)\n\n if sort_by is not None:\n params[\"sort\"] = \"title_sort \" + sort_by\n\n response = self._get(self.repository + \"/search\", params=params)\n hits = response.json()\n return [format_record(json.loads(r[\"json\"])) for r in hits[\"results\"]]", "response": "Fetches a list of all resource IDs for every resource in the database."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfind resource by a specific ID.", "response": "def find_by_id(self, object_type, field, value):\n \"\"\"\n Find resource by a specific ID.\n\n Results are a dict in the format:\n {\n 'id': ,\n 'identifier': ,\n 'title': ,\n 'levelOfDescription': <level of description>,\n }\n\n :param str object_type: One of 'digital_object_components' or 'archival_objects'\n :param str field: Name of the field to search. One of 'component_id' or 'ref_id'.\n :param value: Value of the field to search for\n :return: List of dicts containing results.\n \"\"\"\n\n def format_record(record):\n resolved = record[\"_resolved\"]\n identifier = (\n resolved[\"ref_id\"]\n if \"ref_id\" in resolved\n else resolved.get(\"component_id\", \"\")\n )\n return {\n \"id\": record[\"ref\"],\n \"type\": self.resource_type(record[\"ref\"]),\n \"identifier\": identifier,\n \"title\": resolved.get(\"title\", \"\"),\n \"levelOfDescription\": resolved.get(\"level\", \"\"),\n \"fullrecord\": resolved,\n }\n\n if object_type not in (\"digital_object_components\", \"archival_objects\"):\n raise ValueError(\n \"object_type must be 'digital_object_components' or 'archival_objects'\"\n )\n if field not in (\"ref_id\", \"component_id\"):\n raise ValueError(\"field must be 'component_id' or 'ref_id'\")\n\n params = {field + \"[]\": value, \"resolve[]\": object_type}\n\n url = self.repository + \"/find_by_id/\" + object_type\n response = self._get(url, params=params)\n hits = response.json()\n return [format_record(r) for r in hits[object_type]]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef add_digital_object(\n self,\n parent_archival_object,\n identifier,\n title=None,\n uri=None,\n location_of_originals=None,\n object_type=\"text\",\n xlink_show=\"embed\",\n xlink_actuate=\"onLoad\",\n restricted=False,\n use_statement=\"\",\n use_conditions=None,\n access_conditions=None,\n size=None,\n format_name=None,\n format_version=None,\n inherit_dates=False,\n inherit_notes=False,\n ):\n \"\"\"\n Creates a new digital object.\n\n :param string parent_archival_object: The archival object to which the newly-created digital object will be parented.\n :param string identifier: A unique identifier for the digital object, in any format.\n :param string title: The title of the digital object.\n :param string uri: The URI to an instantiation of the digital object.\n :param string location_of_originals: If provided, will create an `originalsloc` (location of originals) note in the digital object using this text.\n :param string object_type: The type of the digital object.\n Defaults to \"text\".\n :param string xlink_show: Controls how the file will be displayed.\n For supported values, see: http://www.w3.org/TR/xlink/#link-behaviors\n :param string xlink_actuate:\n :param string use_statement:\n :param string use_conditions: A paragraph of human-readable text to specify conditions of use for the digital object.\n If provided, creates a \"conditions governing use\" note in the digital object.\n :param string access_conditions: A paragraph of human-readable text to specify conditions of use for the digital object.\n If provided, creates a \"conditions governing access\" note in the digital object.\n :param int size: Size in bytes of the digital object\n :param str format_name: Name of the digital object's format\n :param str format_version: Name of the digital object's format version\n :param bool inherit_dates: Inherit dates\n :param bool inherit_notes: Inherit parent notes\n \"\"\"\n parent_record = self.get_record(parent_archival_object)\n repository = parent_record[\"repository\"][\"ref\"]\n language = parent_record.get(\"language\", \"\")\n\n if not title:\n filename = os.path.basename(uri) if uri is not None else \"Untitled\"\n title = parent_record.get(\"display_string\", filename)\n\n new_object = {\n \"title\": title,\n \"digital_object_id\": identifier,\n \"digital_object_type\": object_type,\n \"language\": language,\n \"notes\": [],\n \"restrictions\": restricted,\n \"subjects\": parent_record[\"subjects\"],\n \"linked_agents\": parent_record[\"linked_agents\"],\n }\n\n if inherit_dates:\n new_object[\"dates\"] = parent_record[\"dates\"]\n\n if location_of_originals is not None:\n new_object[\"notes\"].append(\n {\n \"jsonmodel_type\": \"note_digital_object\",\n \"type\": \"originalsloc\",\n \"content\": [location_of_originals],\n \"publish\": False,\n }\n )\n\n if uri is not None:\n new_object[\"file_versions\"] = [\n {\n \"file_uri\": uri,\n \"use_statement\": use_statement,\n \"xlink_show_attribute\": xlink_show,\n \"xlink_actuate_attribute\": xlink_actuate,\n }\n ]\n\n note_digital_object_type = [\n \"summary\",\n \"bioghist\",\n \"accessrestrict\",\n \"userestrict\",\n \"custodhist\",\n \"dimensions\",\n \"edition\",\n \"extent\",\n \"altformavail\",\n \"originalsloc\",\n \"note\",\n \"acqinfo\",\n \"inscription\",\n \"langmaterial\",\n \"legalstatus\",\n \"physdesc\",\n \"prefercite\",\n \"processinfo\",\n \"relatedmaterial\",\n ]\n\n if inherit_notes:\n for pnote in parent_record[\"notes\"]:\n if pnote[\"type\"] in note_digital_object_type:\n dnote = pnote[\"type\"]\n else:\n dnote = \"note\"\n if \"subnotes\" in pnote:\n content = []\n for subnote in pnote[\"subnotes\"]:\n if \"content\" in subnote:\n content.append(subnote[\"content\"])\n else:\n LOGGER.info(\n \"No content field in %s, skipping adding to child digital object.\",\n subnote,\n )\n else:\n content = pnote.get(\"content\", \"\")\n\n new_object[\"notes\"].append(\n {\n \"jsonmodel_type\": \"note_digital_object\",\n \"type\": dnote,\n \"label\": pnote.get(\"label\", \"\"),\n \"content\": content,\n \"publish\": pnote[\"publish\"],\n }\n )\n\n if use_conditions:\n new_object[\"notes\"].append(\n {\n \"jsonmodel_type\": \"note_digital_object\",\n \"type\": \"userestrict\",\n \"content\": [use_conditions],\n \"publish\": True,\n }\n )\n if access_conditions:\n new_object[\"notes\"].append(\n {\n \"jsonmodel_type\": \"note_digital_object\",\n \"type\": \"accessrestrict\",\n \"content\": [access_conditions],\n \"publish\": True,\n }\n )\n if restricted:\n new_object[\"file_versions\"][0][\"publish\"] = False\n new_object[\"publish\"] = False\n\n if size:\n new_object[\"file_versions\"][0][\"file_size_bytes\"] = size\n if format_name:\n new_object[\"file_versions\"][0][\"file_format_name\"] = format_name\n if format_version:\n new_object[\"file_versions\"][0][\"file_format_version\"] = format_version\n\n new_object_uri = self._post(\n repository + \"/digital_objects\", data=json.dumps(new_object)\n ).json()[\"uri\"]\n\n # Now we need to update the parent object with a link to this instance\n parent_record[\"instances\"].append(\n {\n \"instance_type\": \"digital_object\",\n \"digital_object\": {\"ref\": new_object_uri},\n }\n )\n self._post(parent_archival_object, data=json.dumps(parent_record))\n\n new_object[\"id\"] = new_object_uri\n return new_object", "response": "Adds a digital object to the archival object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nadding a new child record to the current record.", "response": "def add_child(\n self,\n parent,\n title=\"\",\n level=\"\",\n start_date=\"\",\n end_date=\"\",\n date_expression=\"\",\n notes=[],\n ):\n \"\"\"\n Adds a new resource component parented within `parent`.\n\n :param str parent: The ID to a resource or a resource component.\n :param str title: A title for the record.\n :param str level: The level of description.\n\n :return: The ID of the newly-created record.\n \"\"\"\n parent_record = self.get_record(parent)\n record_type = self.resource_type(parent)\n repository = parent_record[\"repository\"][\"ref\"]\n\n if record_type == \"resource\":\n resource = parent\n else:\n resource = parent_record[\"resource\"][\"ref\"]\n\n new_object = {\n \"title\": title,\n \"level\": level,\n \"jsonmodel_type\": \"archival_object\",\n \"resource\": {\"ref\": resource},\n }\n\n # Create dates object if any of the date fields is populated\n if date_expression or start_date or end_date:\n date = {\n \"jsonmodel_type\": \"date\",\n \"date_type\": \"inclusive\",\n \"label\": \"creation\",\n }\n if date_expression:\n date[\"expression\"] = date_expression\n if start_date:\n date[\"begin\"] = start_date\n if end_date:\n date[\"end\"] = end_date\n\n new_object[\"dates\"] = [date]\n\n new_object[\"notes\"] = []\n for note in notes:\n note_type = note.get(\"type\", \"odd\")\n # If there is a note, but it's an empty string, skip this;\n # ArchivesSpace doesn't allow subnote content to be empty.\n content = note.get(\"content\")\n if not content:\n continue\n new_note = {\n \"jsonmodel_type\": \"note_multipart\",\n \"publish\": True,\n \"subnotes\": [\n {\"content\": content, \"jsonmodel_type\": \"note_text\", \"publish\": True}\n ],\n \"type\": note_type,\n }\n new_object[\"notes\"].append(new_note)\n\n # \"parent\" always refers to an archival_object instance; if this is rooted\n # directly to a resource, leave it out.\n if record_type == \"resource_component\":\n new_object[\"parent\"] = {\"ref\": parent}\n\n return self._post(\n repository + \"/archival_objects\", data=json.dumps(new_object)\n ).json()[\"uri\"]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef attribute(func):\n attr = abc.abstractmethod(func)\n attr.__iattribute__ = True\n attr = _property(attr)\n return attr", "response": "Wrap a function as an attribute."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef property(func):\n attr = abc.abstractmethod(func)\n attr.__iproperty__ = True\n attr = Property(attr)\n return attr", "response": "Wrap a function as a property."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef classattribute(func):\n attr = abc.abstractmethod(func)\n attr.__iclassattribute__ = True\n attr = _property(attr)\n return attr", "response": "Wrap a function as a class attribute."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef method(func):\n attr = abc.abstractmethod(func)\n attr.__imethod__ = True\n return attr", "response": "Wrap a function as a method."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nwraps a function as a classmethod.", "response": "def classmethod(func):\n \"\"\"Wrap a function as a classmethod.\n\n This applies the classmethod decorator.\n \"\"\"\n attr = abc.abstractmethod(func)\n attr.__iclassmethod__ = True\n attr = _classmethod(attr)\n return attr"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse_config(config_file):\n try:\n with open(config_file, 'r') as f:\n return yaml.load(f)\n except IOError:\n print \"Configuration file {} not found or not readable.\".format(config_file)\n raise", "response": "Parse a YAML configuration file"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _ensure_ifaces_tuple(ifaces):\n try:\n\n ifaces = tuple(ifaces)\n\n except TypeError:\n\n ifaces = (ifaces,)\n\n for iface in ifaces:\n\n if not _issubclass(iface, ibc.Iface):\n\n raise TypeError('Can only compare against interfaces.')\n\n return ifaces", "response": "Convert to a tuple of interfaces and raise if not interfaces."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _check_for_definition(iface, cls, tag, defines):\n attributes = (\n attr\n for attr in iface.__abstractmethods__\n if hasattr(getattr(iface, attr), tag)\n )\n for attribute in attributes:\n\n for node in cls.__mro__:\n\n if hasattr(node, attribute) and defines(getattr(node, attribute)):\n\n return True\n\n try:\n\n attribute\n return False\n\n except NameError:\n\n # Pass the test if the loop was never executed. This indicates there\n # were no iface elements defined in the search.\n return True", "response": "Checks if a value is a valid definition of a value."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if the given class is an implementation of the given iface.", "response": "def issubclass(cls, ifaces):\n \"\"\"Check if the given class is an implementation of the given iface.\"\"\"\n ifaces = _ensure_ifaces_tuple(ifaces)\n for iface in ifaces:\n\n return all((\n _check_for_definition(\n iface,\n cls,\n '__iclassattribute__',\n _is_attribute,\n ),\n _check_for_definition(\n iface,\n cls,\n '__iproperty__',\n _is_property,\n ),\n _check_for_definition(\n iface,\n cls,\n '__imethod__',\n _is_method,\n ),\n _check_for_definition(\n iface,\n cls,\n '__iclassmethod__',\n _is_classmethod,\n ),\n ))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef isinstance(instance, ifaces):\n ifaces = _ensure_ifaces_tuple(ifaces)\n for iface in ifaces:\n\n attributes = (\n attr\n for attr in iface.__abstractmethods__\n if hasattr(getattr(iface, attr), '__iattribute__')\n )\n for attribute in attributes:\n\n if not hasattr(instance, attribute):\n\n return False\n\n if not issubclass(type(instance), ifaces):\n\n return False\n\n return True", "response": "Check if a given instance is an implementation of the interface."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfit the optimizer on the TPF - like pixel flux time series.", "response": "def fit(self, pixel_flux, data_placeholder, var_list, session, feed_dict={}):\n \"\"\"\n Parameters\n ----------\n pixel_flux : ndarray\n The TPF-like pixel flux time series. The first dimension\n must represent time, and the remaining two dimensions\n must represent the spatial dimensions.\n data_placeholder : tf.placeholder\n A placeholder which will be used to pass the n-th time stamp\n to `self.optimizer.minimize`.\n var_list : list\n The list of parameters (as tensors) to optimize for.\n session : instance of tf.Session\n feed_dict : dict\n Dictionary of additional arguments used to feed the loss function.\n \"\"\"\n opt_params = []\n cadences = range(pixel_flux.shape[0])\n\n for n in tqdm.tqdm(cadences):\n feed_dict[data_placeholder] = pixel_flux[n]\n self.optimizer.minimize(session=session, feed_dict=feed_dict)\n opt_params.append([session.run(var) for var in var_list])\n\n return opt_params"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_entry(self, variant_line=None, variant_dict=None, raw_entry=None, \n vcf_header=None, csq_format=None, dict_key=None, individual_id=None):\n \"\"\"Return the splitted entry from variant information\n \n Args:\n variant_line (str): A vcf formated variant line\n vcf_header (list): A list with the vcf header line\n csq_format (list): A list with the csq headers\n family_id (str): The family id that should be searched. If no id \n the first family found will be used\n \n Returns:\n entry (list): A list with the splitted entry\n \"\"\"\n if not raw_entry:\n raw_entry = self.get_raw_entry(\n variant_line=variant_line, \n variant_dict=variant_dict, \n vcf_header=vcf_header, \n individual_id=individual_id,\n dict_key=dict_key\n )\n \n entry = []\n if raw_entry:\n \n if self.field in ['CHROM', 'POS', 'REF', 'QUAL']:\n # We know these fields allways has one entry\n entry = [raw_entry]\n \n elif self.field in ['ID', 'FILTER']:\n # We know ID is allways splitted on ';'\n entry = raw_entry.split(';')\n \n elif self.field == 'ALT':\n # We know ALT is allways splitted on ','\n entry = raw_entry.split(',')\n \n elif self.field == 'FORMAT':\n entry = raw_entry.split(':')\n \n elif self.field == 'INFO':\n # We are going to treat csq fields separately\n if self.info_key == 'CSQ':\n if not csq_format:\n raise IOError(\"If CSQ the csq format must be provided\")\n if not self.csq_key:\n raise IOError(\"If CSQ a csq key must be provided\")\n for i, head in enumerate(csq_format):\n if head == self.csq_key:\n # This is the csq entry we are looking for\n csq_column = i\n # CSQ entries are allways splitted on ','\n for csq_entry in raw_entry.split(','):\n entry += split_strings(csq_entry.split('|')[csq_column], self.separators)\n else:\n if self.dict_entry:\n separators = self.separators[2:]\n else:\n separators = self.separators\n \n entry = split_strings(raw_entry, separators)\n \n elif self.field == 'sample_id':\n if not self.separators:\n entry = split_strings(raw_entry, '/')\n #If variant calls are phased we need to split on '|'\n if len(entry) == 1:\n entry = split_strings(raw_entry, '|')\n else:\n entry = split_strings(raw_entry, self.separators)\n \n return entry", "response": "Returns the splitted entry from the variant information"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the raw entry from the vcf field if no entry was found return None", "response": "def get_raw_entry(self, variant_line=None, variant_dict=None, \n vcf_header=None, individual_id=None, dict_key=None):\n \"\"\"Return the raw entry from the vcf field\n \n If no entry was found return None\n \n Args:\n variant_line (str): A vcf formated variant line\n vcf_header (list): A list with the vcf header line\n individual_id (str): The individual id to get gt call\n Returns:\n The raw entry found in variant line\n \"\"\"\n if variant_line:\n variant_line = variant_line.rstrip().split()\n \n entry = None\n \n if self.field == 'CHROM':\n if variant_line:\n entry = variant_line[0]\n elif variant_dict:\n entry = variant_dict['CHROM']\n \n elif self.field == 'POS':\n if variant_line:\n entry = variant_line[1]\n elif variant_dict:\n entry = variant_dict['POS']\n \n elif self.field == 'ID':\n if variant_line:\n entry = variant_line[2]\n elif variant_dict:\n entry = variant_dict['ID']\n \n elif self.field == 'REF':\n if variant_line:\n entry = variant_line[3]\n elif variant_dict:\n entry = variant_dict['REF']\n \n elif self.field == 'ALT':\n if variant_line:\n entry = variant_line[4]\n elif variant_dict:\n entry = variant_dict['ALT']\n \n elif self.field == 'QUAL':\n if variant_line:\n entry = variant_line[5]\n elif variant_dict:\n entry = variant_dict['QUAL']\n \n elif self.field == 'FILTER':\n if variant_line:\n entry = variant_line[6]\n elif variant_dict:\n entry = variant_dict['FILTER']\n \n elif self.field == 'INFO':\n if variant_line:\n for info_annotation in variant_line[7].split(';'):\n splitted_annotation = info_annotation.split('=')\n if self.info_key == splitted_annotation[0]:\n if len(splitted_annotation) == 2:\n entry = splitted_annotation[1]\n \n elif variant_dict:\n entry = variant_dict.get('info_dict',{}).get(self.info_key)\n \n if self.dict_entry and entry:\n #First we split the \"dictionaries\"\n first_split = entry.split(self.separators[0])\n for annotation in first_split:\n # Then we search for the dict key\n splitted_entry = annotation.split(self.separators[1])\n key = splitted_entry[0] \n value = splitted_entry[1]\n if dict_key:\n if key == dict_key:\n entry = value\n #If no key we just return the last entry\n else:\n entry = value\n \n \n elif self.field == 'FORMAT':\n if variant_line:\n entry = variant_line[8]\n elif variant_dict:\n entry = variant_dict['FORMAT']\n \n elif self.field == \"sample_id\":\n \n if not individual_id:\n raise IOError(\"If 'sample_id' a individual id must be provided\")\n if not self.gt_key:\n raise IOError(\"If 'sample_id' a genotype key must be provided\")\n \n if variant_line:\n if not vcf_header:\n raise IOError(\"If 'sample_id' the vcf header must be provided\")\n \n format_info = variant_line[8]\n \n for i, head in enumerate(vcf_header):\n if head == individual_id:\n raw_gt_call = variant_line[i]\n elif variant_dict:\n format_info = variant_dict['FORMAT']\n raw_gt_call = variant_dict[individual_id]\n \n entry_dict = dict(zip(\n format_info.split(':'), raw_gt_call.split(':')\n ))\n entry = entry_dict.get(self.gt_key, '.')\n \n return entry"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_value(self, variant_line=None, variant_dict=None, entry=None, \n raw_entry=None, vcf_header=None, csq_format=None, dict_key=None, \n individual_id=None):\n \"\"\"\n Return the value as specified by plugin\n \n Get value will return one value or None if no correct value is found.\n \n Arguments:\n variant_line (str): A vcf variant line\n variant_dict (dict): A variant dictionary\n entry (list): A splitted entry\n raw_entry (str): The raw entry from the vcf file\n vcf_header (list): The vcf header line with sample ids\n csq_format (list): The CSQ format\n family_id (str): The family id\n individual_id (str): The individual id\n \n Returns:\n value (str): A string that represents the correct value\n \n \"\"\"\n value = None\n \n raw_entry = self.get_raw_entry(\n variant_line = variant_line, \n variant_dict = variant_dict, \n vcf_header=vcf_header, \n individual_id=individual_id,\n dict_key=dict_key\n )\n # If data type is flag we only need to check if any entry exists\n if self.data_type == 'flag':\n if self.field == 'INFO':\n if variant_line:\n for info_entry in variant_line.split()[7].split(';'):\n if self.info_key == info_entry.split('=')[0]:\n value = True\n elif variant_dict:\n if self.info_key in variant_dict.get('info_dict',{}):\n value = True\n else:\n if raw_entry != '.':\n value = True\n \n # If we have a record rule we need to return the correct value\n elif raw_entry:\n # If there was no raw entry we will return None\n if self.record_rule:\n \n if self.data_type == 'string':\n \n if self.record_rule == 'max':\n sorted_strings = sorted(\n self.string_rules.items(), \n key=operator.itemgetter(1), \n reverse=True\n )\n \n if self.record_rule == 'min':\n sorted_strings = sorted(\n self.string_rules.items(), \n key=operator.itemgetter(1)\n )\n \n for string_rule in sorted_strings:\n if string_rule[0].lower() in raw_entry.lower():\n value = string_rule[0]\n break\n else:\n \n typed_annotations = []\n \n for value in self.get_entry(\n raw_entry=raw_entry,\n vcf_header=vcf_header, \n csq_format=csq_format, \n dict_key=dict_key, \n individual_id=individual_id):\n \n if self.data_type == 'float':\n try:\n typed_annotations.append(float(value))\n except ValueError:\n pass\n \n elif self.data_type == 'integer':\n try:\n typed_annotations.append(int(value))\n except ValueError:\n pass\n \n if typed_annotations:\n if self.record_rule == 'max':\n value = max(typed_annotations)\n \n elif self.record_rule == 'min':\n value = min(typed_annotations)\n else:\n value = None\n \n # If no record rule is given we return the raw annotation\n # Here the data_type is not flag, and there is no record rule\n # We know that there exists a raw annotation\n else:\n # We will just return the first annotation found\n value = self.get_entry(\n raw_entry=raw_entry,\n vcf_header=vcf_header, \n csq_format=csq_format, \n dict_key=dict_key, \n individual_id=individual_id)[0]\n \n if self.data_type == 'float':\n try:\n value = float(value)\n except ValueError:\n pass\n \n elif self.data_type == 'integer':\n try:\n value = int(value)\n except ValueError:\n pass\n \n return value", "response": "Get the value of the attribute in the correct format"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a layout for a given graph", "response": "def create_layout(graph, graphviz_prog=DEFAULT_GRAPHVIZ_PROG):\n \"\"\"Return {node: position} for given graph\"\"\"\n graphviz_layout = graphutils.graphviz_layout(graph, prog=graphviz_prog)\n # print('GRAPHIZ LAYOUT:', graphviz_layout)\n layout = {k: (int(x // 10), int(y // 10))\n for k, (x, y) in graphviz_layout.items()}\n # apply an offset for layouts to get all position >= 0\n max_x = max(layout.values(), key=lambda t: t[0])[0]\n min_x = min(layout.values(), key=lambda t: t[0])[0]\n max_y = max(layout.values(), key=lambda t: t[1])[1]\n min_y = min(layout.values(), key=lambda t: t[1])[1]\n offset_x = - min(0, min_x)\n offset_y = - min(0, min_y)\n return {\n node: (offset_x + x, offset_y + y)\n for node, (x, y) in layout.items()\n }"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nyields a string representation of a graph in a pretty way.", "response": "def pretty_view(graph, oriented=False, construction=False,\n graphviz_prog=DEFAULT_GRAPHVIZ_PROG):\n \"\"\"Yield strings, printable view of given graph\"\"\"\n layout = create_layout(graph, graphviz_prog=graphviz_prog)\n matrix_view = defaultdict(lambda: ' ')\n\n # Add the edge to the view\n\n # print('GRAPH EDGES:', tuple(graph.edges()))\n # print('LAYOUT:', layout)\n\n edges = ((layout[source], layout[target])\n for source, target in graph.edges())\n # print('EDGES:', tuple(edges))\n\n for source, target in edges:\n previous_edge_char = None\n previous_position = source\n while source != target:\n for is_dir, transform, edge_char in DIRECTIONS:\n first_loop = previous_edge_char is None\n if is_dir(source, target):\n previous_source = source\n source = transform(source)\n if not first_loop: # first loop: no previous char\n char = CHARACTER[previous_edge_char, edge_char]\n if not isinstance(char, str):\n char = char.value\n matrix_view[previous_source] = char\n if construction:\n old = defaultdict(lambda: ' ', matrix_view)\n yield view.build(matrix_view)\n assert isinstance(matrix_view[previous_source], str)\n if source != target:\n previous_edge_char = edge_char\n previous_position = previous_source\n break # for loop ; don't test the remain directions\n\n if oriented:\n matrix_view[previous_position] = ORIENTATION[previous_edge_char, edge_char]\n if construction:\n yield view.build(matrix_view)\n\n # mark the place where nodes labels will be added\n # for node, coords in layout.items():\n # matrix_view[coords] = node[0]\n # Add the node labels to the view\n # matrix_view = view.clean(matrix_view)\n for node, (x, y) in layout.items():\n if len(node) == 1:\n matrix_view[x, y] = node\n continue\n row_min, row_max = (view.previous_unwrittable_on_row(matrix_view, (x, y)),\n view.next_unwrittable_on_row(matrix_view, (x, y)))\n col_min, col_max = (view.previous_unwrittable_on_col(matrix_view, (x, y)),\n view.next_unwrittable_on_col(matrix_view, (x, y)))\n # print('NODE ' + node + ':',\n # '\u2192row: [{};{}]'.format(row_min, row_max).ljust(20),\n # '\u2193col: [{};{}]'.format(col_min, col_max))\n print_coords = [itertools.count(x), itertools.cycle((y,))]\n if row_min is None: # write left to right, end at (x, y)\n if row_max is None or row_max > (x + len(node) / 2): # enough space at the right\n factor = 2\n else:\n factor = 1\n print_coords[0] = tuple(\n x - (len(node) // factor) + offset + 1\n for offset in range(len(node))\n )\n # print('DEBUG 1:', y, len(node), print_coords[0])\n elif row_max is None: # write left to right, beginning at (x, y)\n if row_min < (x - len(node) / 2): # enough space at the left\n factor = 1\n else:\n factor = 0\n print_coords[0] = tuple(\n x + offset - (len(node) // 2) * factor\n for offset in range(len(node))\n )\n # print('DEBUG 2:', print_coords[0])\n elif (row_max - row_min) > len(node) + 1: # write left to right, if enough place\n print_coords[0] = tuple(\n x + offset\n for offset in range(len(node))\n )\n # print('DEBUG 3:', print_coords[0])\n\n elif col_min is None: # write up to down, end at (x, y)\n if col_max is None or col_max > (x + len(node) / 2): # enough space at the right\n factor = 2\n else:\n factor = 1\n print_coords = (itertools.cycle((x,)), tuple(\n y - (len(node) // factor) + offset + 1\n for offset in range(len(node))\n ))\n # print('DEBUG 4:', y, len(node), print_coords[1])\n elif col_max is None: # write up to down, beginning at (x, y)\n if col_min < (x - len(node) / 2): # enough space at the left\n factor = 1\n else:\n factor = 0\n print_coords = (itertools.cycle((x,)), tuple(\n y + offset - (len(node) // 2) * factor\n for offset in range(len(node))\n ))\n # print('DEBUG 5:', print_coords[1])\n elif (col_max - col_min) > len(node) + 1: # write up to down, if enough place\n print_coords = (itertools.cycle((x,)), tuple(\n y + offset\n for offset in range(len(node))\n ))\n # print('DEBUG 6:', print_coords[1])\n else: # not enough space\n if (row_max - row_min) > (col_max - col_min):\n # more space on Y axis\n node = node[:row_max - row_min] # cut the node\n print_coords = (itertools.cycle((x,)), tuple(\n x + offset\n for offset in range(len(node))\n ))\n # print('DEBUG 7:', print_coords[1])\n else:\n # more space on X axis\n node = node[:col_max - col_min] # cut the node\n print_coords[0] = tuple(\n x + offset\n for offset in range(len(node))\n )\n # print('DEBUG 8:', print_coords[0])\n\n for letter, i, j in zip(node, *print_coords):\n matrix_view[i, j] = letter\n\n\n\n\n if construction:\n yield view.build(matrix_view)\n else:\n yield from view.build(matrix_view)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfinds the call class for a given method if it exists else create one.", "response": "def _get_call_class(method):\n \"\"\"Find the call class for method if it exists else create one.\"\"\"\n call_base, call_name = method.split('.', 1)\n # import the call class's module\n mod = __import__('ubersmith.calls.{0}'.format(call_base), fromlist=[''])\n # grab all the public members of the module\n gen = (getattr(mod, x) for x in dir(mod) if not x.startswith('_'))\n # filter them down to subclasses of BaseCall\n gen = (x for x in gen if type(x) is type and issubclass(x, BaseCall))\n # return first one that matches our method\n for call_class in gen:\n if call_class.method == method:\n return call_class\n else:\n class GenericCall(BaseCall):\n method = '.'.join((call_base, call_name))\n return GenericCall"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef render(self):\n if not self.validate():\n raise ValidationError\n\n self.process_request()\n self.clean()\n\n return self.response", "response": "Validate process clean and return the result of the call."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nvalidating request data before sending it out. Return True or False.", "response": "def validate(self):\n \"\"\"Validate request data before sending it out. Return True/False.\"\"\"\n # check if required_fields aren't present\n for field in set(self.required_fields) - set(self.request_data):\n if not isinstance(field, string_types):\n # field was a collection, iterate over it and check by OR\n return bool(set(field) & set(self.request_data))\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nprocesses the call and set response_data.", "response": "def process_request(self):\n \"\"\"Processing the call and set response_data.\"\"\"\n self.response = self.request_handler.process_request(\n self.method, self.request_data)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nclass decorator that creates a new class that extends the base class with the total ordering methods.", "response": "def total_ordering(cls): # pragma: no cover\n \"\"\"Class decorator that fills in missing ordering methods\"\"\"\n convert = {\n '__lt__': [('__gt__', lambda self, other: not (self < other or self == other)),\n ('__le__', lambda self, other: self < other or self == other),\n ('__ge__', lambda self, other: not self < other)],\n '__le__': [('__ge__', lambda self, other: not self <= other or self == other),\n ('__lt__', lambda self, other: self <= other and not self == other),\n ('__gt__', lambda self, other: not self <= other)],\n '__gt__': [('__lt__', lambda self, other: not (self > other or self == other)),\n ('__ge__', lambda self, other: self > other or self == other),\n ('__le__', lambda self, other: not self > other)],\n '__ge__': [('__le__', lambda self, other: (not self >= other) or self == other),\n ('__gt__', lambda self, other: self >= other and not self == other),\n ('__lt__', lambda self, other: not self >= other)]\n }\n roots = set(dir(cls)) & set(convert)\n if not roots:\n raise ValueError('must define at least one ordering operation: < > <= >=')\n root = max(roots) # prefer __lt__ to __le__ to __gt__ to __ge__\n for opname, opfunc in convert[root]:\n if opname not in roots:\n opfunc.__name__ = opname\n opfunc.__doc__ = getattr(int, opname).__doc__\n setattr(cls, opname, opfunc)\n return cls"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ninitializes the logger with a file.", "response": "def init_logger_file(log_file, log_level='INFO'):\n \"\"\" Append a FileHandler to the root logger.\n :param str log_file: Path to the log file\n :param str log_level: Logging level\n \"\"\"\n log_level = LOG_LEVELS[log_level] if log_level in LOG_LEVELS.keys() else logging.INFO\n\n ROOT_LOG.setLevel(log_level)\n\n file_handle = logging.FileHandler(log_file)\n file_handle.setLevel(log_level)\n file_handle.setFormatter(formatter)\n ROOT_LOG.addHandler(file_handle)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nload a private key from the cryptography library.", "response": "def load_signing_key(signing_key, crypto_backend=default_backend()):\n \"\"\" Optional: crypto backend object from the \"cryptography\" python library\n \"\"\"\n if not isinstance(crypto_backend, (Backend, MultiBackend)):\n raise ValueError('backend must be a valid Backend object')\n\n if isinstance(signing_key, EllipticCurvePrivateKey):\n return signing_key\n elif isinstance(signing_key, (str, unicode)):\n invalid_strings = [b'-----BEGIN PUBLIC KEY-----']\n invalid_string_matches = [\n string_value in signing_key\n for string_value in invalid_strings\n ]\n if any(invalid_string_matches):\n raise ValueError(\n 'Signing key must be a private key, not a public key.')\n\n try:\n return load_der_private_key(\n signing_key, password=None, backend=crypto_backend)\n except:\n try:\n return load_pem_private_key(\n signing_key, password=None, backend=crypto_backend)\n except Exception:\n raise ValueError(\n 'Signing key must be a valid private key PEM or DER.')\n else:\n raise ValueError('Signing key must be in string or unicode format.')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load_verifying_key(verifying_key, crypto_backend=default_backend()):\n if not isinstance(crypto_backend, (Backend, MultiBackend)):\n raise ValueError('backend must be a valid Backend object')\n\n if isinstance(verifying_key, EllipticCurvePublicKey):\n return verifying_key\n elif isinstance(verifying_key, (str, unicode)):\n try:\n return load_der_public_key(\n verifying_key, backend=crypto_backend)\n except:\n try:\n return load_pem_public_key(\n verifying_key, backend=crypto_backend)\n except Exception:\n raise ValueError('Invalid verifying key format')\n else:\n raise ValueError('Invalid verification key type')", "response": "Load a verification key from the cryptography library."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _format_notes(self, record):\n notes = []\n\n if \"notes\" in record:\n for note in record[\"notes\"]:\n self._append_note_dict_to_list(notes, \"general\", note)\n\n if \"language_and_script_notes\" in record:\n self._append_note_dict_to_list(\n notes, \"language_and_script\", record[\"language_and_script_notes\"]\n )\n\n if \"publication_notes\" in record:\n self._append_note_dict_to_list(\n notes, \"publication_notes\", record[\"publication_notes\"]\n )\n\n if \"physical_characteristics_and_technical_requirements\" in record:\n self._append_note_dict_to_list(\n notes,\n \"physical_condition\",\n record[\"physical_characteristics_and_technical_requirements\"],\n )\n\n return notes", "response": "Extracts notes from a record and reformats them in a simplified format."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _escape_lucene_query(query, field=None):\n replacement = r\"\\\\\\1\"\n return re.sub(r'([\\'\" +\\-!\\(\\)\\{\\}\\[\\]^\"~?:\\\\/]|&&|\\|\\|)', replacement, query)", "response": "Escapes special characters in Solr queries."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nupdates a record in AtoM using the provided new_record.", "response": "def edit_record(self, new_record):\n \"\"\"\n Update a record in AtoM using the provided new_record.\n\n The format of new_record is identical to the format returned by get_resource_component_and_children and related methods; consult the documentation for that method in ArchivistsToolkitClient to see the format.\n This means it's possible, for example, to request a record, modify the returned dict, and pass that dict to this method to update the server.\n\n Currently supported fields are:\n * title\n * notes\n * start_date\n * end_date\n * date_expression\n\n :raises ValueError: if the 'slug' field isn't specified, or no fields to edit were specified.\n \"\"\"\n try:\n record_id = new_record[\"slug\"]\n except KeyError:\n raise ValueError(\"No slug provided!\")\n\n record = self.get_record(record_id)\n\n field_map = {\"title\": \"title\", \"level\": \"levelOfDescription\"}\n fields_updated = False\n for field, targetfield in field_map.items():\n try:\n record[targetfield] = new_record[field]\n fields_updated = True\n except KeyError:\n continue\n\n # Optionally add notes\n if \"notes\" in new_record and new_record[\"notes\"]:\n note = new_record[\"notes\"][0]\n new_note = {\"content\": note[\"content\"], \"type\": note[\"type\"]}\n # This only supports editing a single note, and a single piece of content\n # within that note.\n # If the record already has at least one note, then replace the first note\n # within that record with this one.\n if \"notes\" not in record or record[\"notes\"] == []:\n record[\"notes\"] = [new_note]\n else:\n record[\"notes\"][0] = new_note\n\n fields_updated = True\n else:\n # Remove existing notes if the record didn't have a valid note;\n # a note with an empty string as content should be counted as\n # a request to delete the note.\n record[\"notes\"] = []\n\n # Update date\n updated_date = {}\n\n # Only single dates are currently supported\n if \"dates\" in new_record and type(new_record[\"dates\"]) is list:\n new_record[\"dates\"] = new_record[\"dates\"][0]\n\n # Map agentarchives date specification to AtoM specification\n date_mapping = {\n \"start_date\": \"start_date\",\n # 'begin': 'start_date',\n \"end_date\": \"end_date\",\n # 'end': 'end_date',\n \"date_expression\": \"date\",\n }\n\n for date_field in date_mapping:\n if date_field in new_record:\n updated_date[date_mapping[date_field]] = new_record[date_field]\n\n # Add updated date specification to record update\n if updated_date != {}:\n record[\"dates\"] = [updated_date]\n fields_updated = True\n\n if not fields_updated:\n raise ValueError(\"No fields to update specified!\")\n\n self._put(\n urljoin(self.base_url, \"informationobjects/{}\".format(record_id)),\n data=json.dumps(record),\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_levels_of_description(self):\n if not hasattr(self, \"levels_of_description\"):\n self.levels_of_description = [\n item[\"name\"]\n for item in self._get(urljoin(self.base_url, \"taxonomies/34\")).json()\n ]\n\n return self.levels_of_description", "response": "Returns an array of all levels of description defined in this AtoM instance."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef collection_list(self, resource_id, resource_type=\"collection\"):\n\n def fetch_children(children):\n results = []\n\n for child in children:\n results.append(child[\"slug\"])\n\n if \"children\" in child:\n results.extend(fetch_children(child[\"children\"]))\n\n return results\n\n response = self._get(\n urljoin(self.base_url, \"informationobjects/tree/{}\".format(resource_id))\n )\n tree = response.json()\n return fetch_children(tree[\"children\"])", "response": "Fetches a list of slugs representing descriptions within the specified parent description."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_resource_component_and_children(\n self,\n resource_id,\n resource_type=\"collection\",\n level=1,\n sort_data={},\n recurse_max_level=False,\n sort_by=None,\n **kwargs\n ):\n \"\"\"\n Fetch detailed metadata for the specified resource_id and all of its children.\n\n :param str resource_id: The slug for which to fetch description metadata.\n :param str resource_type: no-op; not required or used in this implementation.\n :param int recurse_max_level: The maximum depth level to fetch when fetching children.\n Default is to fetch all of the resource's children, descending as deeply as necessary.\n Pass 1 to fetch no children.\n\n :return: A dict containing detailed metadata about both the requested resource and its children.\n Consult ArchivistsToolkitClient.get_resource_component_and_children for the output format.\n :rtype dict:\n \"\"\"\n return self._get_resources(\n resource_id, recurse_max_level=recurse_max_level, sort_by=sort_by\n )", "response": "This function returns detailed metadata about the specified resource and all of its children."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngiving the slug of a description returns the parent description s slug.", "response": "def find_parent_id_for_component(self, slug):\n \"\"\"\n Given the slug of a description, returns the parent description's slug.\n\n :param string slug: The slug of a description.\n :return: The URL of the parent record.\n :rtype: string\n \"\"\"\n response = self.get_record(slug)\n\n if \"parent\" in response:\n return response[\"parent\"]\n # resource was passed in, which has no higher-up record;\n # return the same ID\n else:\n return slug"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef find_collection_ids(self, search_pattern=\"\", identifier=\"\", fetched=0, page=1):\n response = self._collections_search_request(search_pattern, identifier, page)\n hits = response.json()\n results = [r[\"slug\"] for r in hits[\"results\"]]\n\n results_so_far = fetched + len(results)\n if hits[\"total\"] > results_so_far:\n results.extend(\n self.find_collection_ids(fetched=results_so_far, page=page + 1)\n )\n\n return results", "response": "Fetches a list of resource URLs for every top - level description in the database."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _collections_search_request(\n self, search_pattern=\"\", identifier=\"\", page=1, page_size=50, sort_by=None\n ):\n \"\"\"\n Fetches a list of resource URLs for every top-level description in the database.\n\n :param string search_pattern: A search pattern to use in looking up resources by title or resourceid.\n The search will match any title containing this string;\n for example, \"text\" will match \"this title has this text in it\".\n If omitted, then all resources will be fetched.\n :param string identifier: Only records containing this identifier will be returned.\n Substring matching will not be performed; however, wildcards are supported.\n For example, searching \"F1\" will only return records with the identifier \"F1\", while searching \"F*\" will return \"F1\", \"F2\", etc.\n\n :return: A list containing every matched resource's URL.\n :rtype list:\n \"\"\"\n skip = (page - 1) * page_size\n\n params = {\n \"limit\": page_size,\n \"skip\": skip,\n \"topLod\": \"1\",\n \"sf0\": \"_all\",\n \"sq0\": \"\",\n }\n\n if search_pattern:\n params[\"sq0\"] = '\"' + self._escape_lucene_query(search_pattern) + '\"'\n\n if identifier != \"\":\n params[\"sf1\"] = \"identifier\"\n params[\"sq1\"] = self._escape_lucene_query(identifier)\n\n if sort_by is not None:\n params[\"sort\"] = \"alphabetic\"\n if sort_by == \"desc\":\n params[\"reverse\"] = True\n\n return self._get(urljoin(self.base_url, \"informationobjects\"), params=params)", "response": "This method is used to make a request to retrieve a list of resource URLs for every top - level description in the database."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfetching a list of all resource IDs for every resource in the database.", "response": "def find_collections(\n self,\n search_pattern=\"\",\n identifier=\"\",\n fetched=0,\n page=1,\n page_size=30,\n sort_by=None,\n ):\n \"\"\"\n Fetches a list of all resource IDs for every resource in the database.\n\n :param string search_pattern: A search pattern to use in looking up resources by title or resourceid.\n The search will match any title or resourceid containing this string;\n for example, \"text\" will match \"this title has this text in it\".\n If omitted, then all resources will be fetched.\n :param string identifier: Restrict records to only those with this identifier.\n This refers to the human-assigned record identifier, not the automatically generated internal ID.\n This value can contain wildcards.\n\n :return: A list containing every matched resource's ID.\n :rtype: list\n \"\"\"\n\n def format_record(record):\n # Get record details\n full_record = self.get_record(record[\"slug\"])\n dates = self._fetch_dates_from_record(record)\n date_expression = self._fetch_date_expression_from_record(record)\n\n # Determine whether descendents exist\n url = urljoin(\n self.base_url, \"informationobjects/tree/{}\".format(record[\"slug\"])\n )\n tree = self._get(url).json()\n if \"children\" in tree:\n has_children = len(self._get(url).json()[\"children\"]) > 0\n else:\n has_children = False\n\n formatted = {\n \"id\": record[\"slug\"],\n \"type\": \"resource\",\n \"sortPosition\": 1,\n \"identifier\": record.get(\"reference_code\", \"\"),\n \"title\": record.get(\"title\", \"\"),\n \"dates\": dates,\n \"date_expression\": date_expression,\n \"children\": [] if has_children else False,\n \"has_children\": has_children,\n \"notes\": full_record.get(\"notes\", []),\n }\n\n if \"level_of_description\" in record:\n formatted[\"levelOfDescription\"] = record[\"level_of_description\"]\n\n return formatted\n\n response = self._collections_search_request(\n search_pattern, identifier, page, page_size, sort_by\n )\n hits = response.json()\n return [format_record(r) for r in hits[\"results\"]]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngive a list of resource IDs, returns a list of dicts containing detailed information about the specified resources and their children. This function recurses to a maximum of two levels when fetching children from the specified resources. Consult the documentation of ArchivistsToolkitClient.get_resource_component_children for the format of the returned dicts. :param list resource_ids: A list of one or more resource IDs. :return: A list containing metadata dicts. :rtype list:", "response": "def augment_resource_ids(self, resource_ids):\n \"\"\"\n Given a list of resource IDs, returns a list of dicts containing detailed information about the specified resources and their children.\n\n This function recurses to a maximum of two levels when fetching children from the specified resources.\n Consult the documentation of ArchivistsToolkitClient.get_resource_component_children for the format of the returned dicts.\n\n :param list resource_ids: A list of one or more resource IDs.\n :return: A list containing metadata dicts.\n :rtype list:\n \"\"\"\n resources_augmented = []\n for id in resource_ids:\n # resource_data = self.get_resource_component_and_children(id, recurse_max_level=2)\n # resources_augmented.append(resource_data)\n resources_augmented.append(\n self.get_resource_component_and_children(id, recurse_max_level=2)\n )\n\n return resources_augmented"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nadds a digital object to the current object.", "response": "def add_digital_object(\n self,\n information_object_slug,\n identifier=None,\n title=None,\n uri=None,\n location_of_originals=None,\n object_type=None,\n xlink_show=\"embed\",\n xlink_actuate=\"onLoad\",\n restricted=False,\n use_statement=\"\",\n use_conditions=None,\n access_conditions=None,\n size=None,\n format_name=None,\n format_version=None,\n format_registry_key=None,\n format_registry_name=None,\n file_uuid=None,\n aip_uuid=None,\n inherit_dates=False,\n usage=None,\n ):\n \"\"\" Creates a new digital object. \"\"\"\n\n new_object = {\"information_object_slug\": information_object_slug}\n\n if title is not None:\n new_object[\"name\"] = title\n\n if uri is not None:\n new_object[\"uri\"] = uri\n\n if size is not None:\n new_object[\"byte_size\"] = size\n\n if object_type is not None:\n new_object[\"media_type\"] = object_type\n\n if usage is not None:\n new_object[\"usage\"] = usage\n\n if file_uuid is not None:\n new_object[\"file_uuid\"] = file_uuid\n if aip_uuid is not None:\n new_object[\"aip_uuid\"] = aip_uuid\n\n if format_name is not None:\n new_object[\"format_name\"] = format_name\n if format_version is not None:\n new_object[\"format_version\"] = format_version\n if format_registry_key is not None:\n new_object[\"format_registry_key\"] = format_registry_key\n if format_registry_name is not None:\n new_object[\"format_registry_name\"] = format_registry_name\n\n new_object[\"slug\"] = self._post(\n urljoin(self.base_url, \"digitalobjects\"),\n data=json.dumps(new_object),\n expected_response=201,\n ).json()[\"slug\"]\n\n return new_object"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_child(\n self,\n parent_slug=None,\n title=\"\",\n level=\"\",\n start_date=None,\n end_date=None,\n date_expression=None,\n notes=[],\n ):\n \"\"\"\n Adds a new resource component parented within `parent`.\n\n :param str parent_slug: The parent's slug.\n :param str title: A title for the record.\n :param str level: The level of description.\n\n :return: The ID of the newly-created record.\n \"\"\"\n\n new_object = {\"title\": title, \"level_of_description\": level}\n\n if parent_slug is not None:\n new_object[\"parent_slug\"] = parent_slug\n\n # Optionally add date specification\n new_date = {}\n\n if start_date is not None:\n new_date[\"start_date\"] = start_date\n\n if end_date is not None:\n new_date[\"end_date\"] = end_date\n\n if date_expression is not None:\n new_date[\"date\"] = date_expression\n\n if new_date != {}:\n new_object[\"dates\"] = [new_date]\n\n # Optionally add notes\n new_object[\"notes\"] = []\n for note in notes:\n note_type = note.get(\"type\", \"General note\")\n # If there is a note, but it's an empty string, skip this;\n content = note.get(\"content\")\n if not content:\n continue\n new_note = {\"content\": content, \"type\": note_type}\n new_object[\"notes\"].append(new_note)\n\n return self._post(\n urljoin(self.base_url, \"informationobjects\"),\n data=json.dumps(new_object),\n expected_response=201,\n ).json()[\"slug\"]", "response": "Adds a new child record to the current record."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndeleting a record with record_id.", "response": "def delete_record(self, record_id):\n \"\"\"\n Delete a record with record_id.\n \"\"\"\n self._delete(\n urljoin(self.base_url, \"informationobjects/{}\".format(record_id)),\n expected_response=204,\n )\n return {\"status\": \"Deleted\"}"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting a section with information of priorities to a string dict.", "response": "def get_string_dict(self, plugin_info):\n \"\"\"\n Convert a section with information of priorities to a string dict.\n \n To avoid typos we make all letters lower case when comparing\n \n Arguments:\n plugin_info (dict): A dictionary with plugin information\n \n Return:\n string_dict (dict): A dictionary with strings as keys and integer\n that specifies their priorities as values\n \"\"\"\n string_info = []\n string_dict = {}\n \n for key in plugin_info:\n try:\n string_info.append(dict(plugin_info[key]))\n except ValueError:\n pass\n \n string_rules = {}\n \n for raw_info in string_info:\n try:\n string = raw_info['string']\n except KeyError:\n raise ValidateError(\"String information has to have a 'string'\")\n try:\n priority = raw_info['priority']\n except KeyError:\n raise ValidateError(\"String information has to have a 'priority'\")\n try:\n priority = int(priority)\n except ValueError:\n raise ValidateError(\"'priority' has to be an integer\")\n \n string_dict[string] = priority\n\n if len(string_dict) == 0:\n raise ValidateError(\"'string' entrys must have string rules defined\")\n \n return string_dict"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef version_check(self):\n try:\n version_info = self['Version']\n except KeyError:\n raise ValidateError('Config file has to have a Version section')\n try:\n float(version_info['version'])\n except KeyError:\n raise ValidateError('Config file has to have a version section')\n except ValueError:\n raise ValidateError('Version has to be a float.')\n \n try:\n version_info['name']\n except KeyError:\n raise ValidateError(\"Config file has to have a name\")\n return", "response": "Check if the version entry is in the proper format."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef check_plugin(self, plugin):\n \n vcf_section = self[plugin]\n \n try:\n vcf_field = vcf_section['field']\n if not vcf_field in self.vcf_columns:\n raise ValidateError(\n \"field has to be in {0}\\n\"\n \"Wrong field name in plugin: {1}\".format(\n self.vcf_columns, plugin\n ))\n if vcf_field == 'INFO':\n try:\n info_key = vcf_section['info_key']\n\n if info_key == 'CSQ':\n try:\n csq_key = vcf_section['csq_key']\n except KeyError:\n raise ValidateError(\n \"CSQ entrys has to refer to an csq field.\\n\"\n \"Refer with keyword 'csq_key'\\n\"\n \"csq_key is missing in section: {0}\".format(\n plugin\n )\n )\n\n\n except KeyError:\n raise ValidateError(\n \"INFO entrys has to refer to an INFO field.\\n\"\n \"Refer with keyword 'info_key'\\n\"\n \"info_key is missing in section: {0}\".format(\n plugin\n )\n )\n except KeyError:\n raise ValidateError(\n \"Vcf entrys have to refer to a field in the VCF with keyword\"\n \" 'field'.\\nMissing keyword 'field' in plugin: {0}\".format(\n plugin\n ))\n\n try:\n data_type = vcf_section['data_type']\n if not data_type in self.data_types:\n raise ValidateError(\n \"data_type has to be in {0}\\n\"\n \"Wrong data_type in plugin: {1}\".format(\n self.data_types, plugin)\n )\n except KeyError:\n raise ValidateError(\n \"Vcf entrys have to refer to a data type in the VCF with \"\n \"keyword 'data_type'.\\n\"\n \"Missing data_type in plugin: {0}\".format(plugin)\n )\n\n \n separators = vcf_section.get('separators', None)\n if separators:\n if len(separators) == 1:\n self[plugin]['separators'] = list(separators)\n else:\n if data_type != 'flag':\n raise ValidateError(\n \"If data_type != flag the separators have to be defined\"\n \"Missing separators in plugin: {0}\".format(plugin)\n )\n \n \n record_rule = vcf_section.get('record_rule', None)\n \n if record_rule:\n if not record_rule in ['min', 'max']:\n raise ValidateError(\n \"Record rules have to be in {0}\\n\"\n \"Wrong record_rule in plugin: {1}\".format(\n ['min', 'max'], plugin)\n )\n else:\n self.logger.info(\"Setting record rule to default: 'max'\")\n \n return True", "response": "Check if the section is in the proper format."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef dict_to_nx(graph, oriented=False):\n nxg = nx.DiGraph() if oriented else nx.Graph()\n for node, succs in graph.items():\n for succ in succs:\n nxg.add_edge(node, succ)\n return nxg", "response": "Return a nx. DiGraph equivalent of given dict."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef values():\n result = {}\n try:\n # Check if there's a sub-span in progress, otherwise use the main span:\n span = g.get(\"subspan\") if \"subspan\" in g else g\n for header in b3_headers:\n result[header] = span.get(header)\n except RuntimeError:\n # We're probably working outside the Application Context at this point, likely on startup:\n # https://stackoverflow.com/questions/31444036/runtimeerror-working-outside-of-application-context\n # We return a dict of empty values so the expected keys are present.\n for header in b3_headers:\n result[header] = None\n\n return result", "response": "Get the full current set of B3 values."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef start_span(request_headers=None):\n global debug\n try:\n headers = request_headers if request_headers else request.headers\n except RuntimeError:\n # We're probably working outside the Application Context at this point, likely on startup:\n # https://stackoverflow.com/questions/31444036/runtimeerror-working-outside-of-application-context\n # We return a dict of empty values so the expected keys are present.\n headers = {}\n\n trace_id = headers.get(b3_trace_id)\n parent_span_id = headers.get(b3_parent_span_id)\n span_id = headers.get(b3_span_id)\n sampled = headers.get(b3_sampled)\n flags = headers.get(b3_flags)\n root_span = not trace_id\n\n # Collect (or generate) a trace ID\n setattr(g, b3_trace_id, trace_id or _generate_identifier())\n\n # Parent span, if present\n setattr(g, b3_parent_span_id, parent_span_id)\n\n # Collect (or set) the span ID\n setattr(g, b3_span_id, span_id or g.get(b3_trace_id))\n\n # Collect the \"sampled\" flag, if present\n # We'll propagate the sampled value unchanged if it's set.\n # We're not currently recording traces to Zipkin, so if it's present, follow the standard and propagate it,\n # otherwise it's better to leave it out, rather than make it \"0\".\n # This allows downstream services to make a decision if they need to.\n setattr(g, b3_sampled, sampled)\n\n # Set or update the debug setting\n # We'll set it to \"1\" if debug=True, otherwise we'll propagate it if present.\n setattr(g, b3_flags, \"1\" if debug else flags)\n\n _info(\"Server receive. Starting span\" if trace_id else \"Root span\")\n _log.debug(\"Resolved B3 values: {values}\".format(values=values()))", "response": "Starts a new Zipkin Trace ID and span ID."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _start_subspan(headers=None):\n b3 = values()\n g.subspan = {\n\n # Propagate the trace ID\n b3_trace_id: b3[b3_trace_id],\n\n # Start a new span for the outgoing request\n b3_span_id: _generate_identifier(),\n\n # Set the current span as the parent span\n b3_parent_span_id: b3[b3_span_id],\n\n b3_sampled: b3[b3_sampled],\n b3_flags: b3[b3_flags],\n }\n\n # Set up headers\n # NB dict() ensures we don't alter the value passed in. Maybe that's too conservative?\n result = dict(headers or {})\n result.update({\n b3_trace_id: g.subspan[b3_trace_id],\n b3_span_id: g.subspan[b3_span_id],\n b3_parent_span_id: g.subspan[b3_parent_span_id],\n })\n\n # Propagate only if set:\n if g.subspan[b3_sampled]:\n result[b3_sampled] = g.subspan[b3_sampled]\n if g.subspan[b3_flags]:\n result[b3_flags] = g.subspan[b3_flags]\n\n _info(\"Client start. Starting sub-span\")\n _log.debug(\"B3 values for sub-span: {b3_headers}\".format(b3_headers=values()))\n _log.debug(\"All headers for downstream request: {b3_headers}\".format(b3_headers=result))\n\n return result", "response": "Start a new sub - span for a downstream service call."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _generate_identifier():\n bit_length = 64\n byte_length = int(bit_length / 8)\n identifier = os.urandom(byte_length)\n return hexlify(identifier).decode('ascii')", "response": "Generates a new 64 - bit random identifier in B3 format."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef pub_view(request, docid, configuration):\n if 'autodeclare' in settings.CONFIGURATIONS[configuration]:\n for annotationtype, set in settings.CONFIGURATIONS['configuration']['autodeclare']:\n try:\n r = flat.comm.query(request, \"USE pub/\" + docid + \" DECLARE \" + annotationtype + \" OF \" + set)\n except Exception as e:\n return fatalerror(request,e)\n\n return initdoc(request, 'pub',docid, 'editor', 'editor.html', configuration=configuration)", "response": "The initial view of the pub document"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef checkversion(version):\n try:\n for refversion, responseversion in zip([int(x) for x in REQUIREFOLIADOCSERVE.split('.')], [int(x) for x in version.split('.')]):\n if responseversion > refversion:\n return 1 #response is newer than library\n elif responseversion < refversion:\n return -1 #response is older than library\n return 0 #versions are equal\n except ValueError:\n raise ValueError(\"Unable to parse version, invalid syntax\")", "response": "Checks the foliadocserve version returns 1 if the document is newer than the library 0 if it is older"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ninitialises a document (not invoked directly)", "response": "def initdoc(request, namespace, docid, mode, template, context=None, configuration=None):\n \"\"\"Initialise a document (not invoked directly)\"\"\"\n perspective = request.GET.get('perspective','document')\n if context is None: context = {}\n if 'configuration' in request.session:\n configuration = request.session['configuration']\n elif configuration is None:\n return fatalerror(request, \"No configuration specified\")\n if configuration not in settings.CONFIGURATIONS:\n return fatalerror(request, \"Specified configuration does not exist\")\n flatargs = {\n 'setdefinitions': True,\n 'declarations': True, #implies provenance as well\n 'metadata': True,\n 'toc': True,\n 'slices': request.GET.get('slices',settings.CONFIGURATIONS[configuration].get('slices','p:25,s:100')), #overriden either by configuration or by user\n 'customslicesize': 0, #disabled for initial probe\n 'textclasses': True,\n }\n error = False\n try:\n doc = flat.comm.query(request, \"USE \" + namespace + \"/\" + docid + \" PROBE\", **flatargs) #retrieves only the meta information, not document content\n context.update(getcontext(request,namespace,docid, doc, mode, configuration))\n except Exception as e:\n context.update(docserveerror(e))\n error = True\n\n if not error:\n dorequiredeclaration = 'requiredeclaration' in settings.CONFIGURATIONS[configuration] and settings.CONFIGURATIONS[configuration]['requiredeclaration']\n if dorequiredeclaration:\n if not 'declarations' in doc:\n return fatalerror(request, \"Refusing to load document, missing expected declarations, none declared\")\n declarations = doc['declarations']\n for annotationtype, annotationset in settings.CONFIGURATIONS[configuration]['requiredeclaration']:\n found = False\n for d in declarations:\n if annotationtype == d['annotationtype'] and (not annotationset or annotationset == d['set']):\n found = True\n break\n if not found:\n if annotationset:\n return fatalerror(request, \"Refusing to load document, missing expected declaration for annotation type \" + annotationtype + \"/\" + annotationset)\n else:\n return fatalerror(request, \"Refusing to load document, missing expected declaration for annotation type \" + annotationtype)\n\n dometadataindex = 'metadataindex' in settings.CONFIGURATIONS[configuration] and settings.CONFIGURATIONS[configuration]['metadataindex']\n if dometadataindex:\n metadata = json.loads(context['metadata'])\n for metakey in settings.CONFIGURATIONS[configuration]['metadataindex']:\n if metakey in metadata:\n MetadataIndex.objects.update_or_create(namespace=namespace,docid=docid, key=metakey,defaults={'value':metadata[metakey]})\n response = render(request, template, context)\n if 'fatalerror' in context:\n response.status_code = 500\n return response"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef query_helper(request,namespace, docid, configuration=None):\n flatargs = {\n 'customslicesize': request.POST.get('customslicesize',settings.CONFIGURATIONS[configuration].get('customslicesize','50')), #for pagination of search results\n }\n #stupid compatibility stuff\n if sys.version < '3':\n if hasattr(request, 'body'):\n data = json.loads(unicode(request.body,'utf-8')) #pylint: disable=undefined-variable\n else: #older django\n data = json.loads(unicode(request.raw_post_data,'utf-8')) #pylint: disable=undefined-variable\n else:\n if hasattr(request, 'body'):\n data = json.loads(str(request.body,'utf-8'))\n else: #older django\n data = json.loads(str(request.raw_post_data,'utf-8'))\n\n if not data['queries']:\n return HttpResponseForbidden(\"No queries to run\")\n\n for query in data['queries']:\n #get document selector and check it doesn't violate the namespace\n docselector, query = getdocumentselector(query)\n if not docselector:\n return HttpResponseForbidden(\"Query does not start with a valid document selector (USE keyword)!\")\n elif docselector[0] != namespace:\n return HttpResponseForbidden(\"Query would affect a different namespace than your current one, forbidden!\")\n\n if query != \"GET\" and query[:4] != \"CQL \" and query[:4] != \"META\":\n #parse query on this end to catch syntax errors prior to sending, should be fast enough anyway\n #first resolve variables to dummies (real ones will be handled server-side) as it won't be valid FQL otherwise\n query = query.replace(\"$FOLIADOCSERVE_PROCESSOR\", \"PROCESSOR name \\\"foliadocserve\\\"\")\n query = query.replace(\"$FLAT_PROCESSOR\", \"PROCESSOR name \\\"FLAT\\\" version \\\"\" + VERSION + \"\\\" host \\\"\" + request.get_host() + \"\\\" src \\\"\" + request.build_absolute_uri(\"/\") + \"\\\"\") #also another instance in comm.py\n try:\n query = fql.Query(query)\n except fql.SyntaxError as e:\n return HttpResponseForbidden(\"FQL Syntax Error: \" + str(e))\n needwritepermission = query.declarations or query.action and query.action.action != \"SELECT\"\n else:\n needwritepermission = False\n\n if configuration != \"pub\":\n if needwritepermission and not flat.users.models.haswritepermission(request.user.username, namespace, request):\n return HttpResponseForbidden(\"Permission denied, no write access\")\n\n query = \"\\n\".join(data['queries']) #throw all queries on a big pile to transmit\n try:\n d = flat.comm.query(request, query,**flatargs)\n except Exception as e:\n if sys.version < '3':\n errmsg = docserveerror(e)['fatalerror_text']\n return HttpResponseForbidden(\"FoLiA Document Server error: \".encode('utf-8') + errmsg.encode('utf-8'))\n else:\n return HttpResponseForbidden(\"FoLiA Document Server error: \" + docserveerror(e)['fatalerror_text'])\n return HttpResponse(json.dumps(d).encode('utf-8'), content_type='application/json')", "response": "This function is used to make the actual query"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef pub_poll(request, docid):\n try:\n r = flat.comm.get(request, '/poll/pub/' + docid + '/', False)\n except URLError:\n return HttpResponseForbidden(\"Unable to connect to the document server [viewer/poll]\")\n return HttpResponse(r, content_type='application/json')", "response": "Get the content of a single document"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nhooks after prepare and set id disabled.", "response": "def prepare(_next, self):\n \"\"\"Hook after prepare and set 'id' disabled.\n \"\"\"\n _next(self)\n if not self.autoincrement_support:\n return\n id_field = self.form['id']\n del id_field.attrs['required']\n id_field.attrs['disabled'] = 'disabled'\n id_field.getter = _('auto_incremented', default='auto incremented')"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef delete_user_action(model, request):\n try:\n users = model.parent.backend\n uid = model.model.name\n del users[uid]\n users()\n model.parent.invalidate()\n localizer = get_localizer(request)\n message = localizer.translate(_(\n 'delete_user_from_database',\n default=\"Deleted user '${uid}' from database.\",\n mapping={'uid': uid}\n ))\n return {\n 'success': True,\n 'message': message\n }\n except Exception as e:\n return {\n 'success': False,\n 'message': str(e)\n }", "response": "Delete user from database."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef user_add_to_group_action(model, request):\n group_id = request.params.get('id')\n if not group_id:\n group_ids = request.params.getall('id[]')\n else:\n group_ids = [group_id]\n try:\n user = model.model\n validate_add_users_to_groups(model, [user.id], group_ids)\n groups = user.root.groups\n for group_id in group_ids:\n groups[group_id].add(user.name)\n groups()\n model.parent.invalidate(user.name)\n localizer = get_localizer(request)\n message = localizer.translate(_(\n 'added_user_to_group',\n default=\"Added user '${uid}' to group '${gid}'.\",\n mapping={\n 'uid': user.id,\n 'gid': ', '.join(group_ids)\n }\n ))\n return {\n 'success': True,\n 'message': message\n }\n except ManageMembershipError as e:\n if e.reason is not LM_TARGET_GID_NOT_ALLOWED:\n raise Exception(u\"Unknown ManageMembershipError reason.\")\n localizer = get_localizer(request)\n message = localizer.translate(_(\n 'lm_add_target_gid_not_allowed',\n default=(\n \"Failed adding user '${uid}' to group '${gid}'. \"\n \"Manage membership denied for target group.\"\n ),\n mapping={\n 'uid': user.id,\n 'gid': e.data\n }\n ))\n return {\n 'success': False,\n 'message': message\n }\n except Exception as e:\n return {\n 'success': False,\n 'message': str(e)\n }", "response": "Add user to group."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndeletes group from database.", "response": "def delete_group_action(model, request):\n \"\"\"Delete group from database.\n \"\"\"\n try:\n groups = model.parent.backend\n uid = model.model.name\n del groups[uid]\n groups()\n model.parent.invalidate()\n except Exception as e:\n return {\n 'success': False,\n 'message': str(e)\n }\n localizer = get_localizer(request)\n message = localizer.translate(_(\n 'deleted_group',\n default='Deleted group from database'\n ))\n return {\n 'success': True,\n 'message': message\n }"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef group_add_user_action(model, request):\n user_id = request.params.get('id')\n if not user_id:\n user_ids = request.params.getall('id[]')\n else:\n user_ids = [user_id]\n try:\n group = model.model\n validate_add_users_to_groups(model, user_ids, [group.id])\n for user_id in user_ids:\n group.add(user_id)\n group()\n model.parent.invalidate(group.name)\n localizer = get_localizer(request)\n message = localizer.translate(_(\n 'added_user_to_group',\n default=\"Added user '${uid}' to group '${gid}'.\",\n mapping={\n 'uid': ', '.join(user_ids),\n 'gid': group.id\n }\n ))\n return {\n 'success': True,\n 'message': message\n }\n except ManageMembershipError as e:\n if e.reason is not LM_TARGET_UID_NOT_ALLOWED:\n raise Exception(u\"Unknown ManageMembershipError reason.\")\n localizer = get_localizer(request)\n message = localizer.translate(_(\n 'lm_add_target_uid_not_allowed',\n default=(\n \"Failed adding user '${uid}' to group '${gid}'. \"\n \"Manage membership denied for user.\"\n ),\n mapping={\n 'uid': e.data,\n 'gid': group.id\n }\n ))\n return {\n 'success': False,\n 'message': message\n }\n except Exception as e:\n return {\n 'success': False,\n 'message': str(e)\n }", "response": "Add user to group."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef prepare(_next, self):\n _next(self)\n if not self.roles_support:\n return\n if not self.request.has_permission('manage', self.model.parent):\n # XXX: yafowil selection display renderer\n return\n value = []\n if self.action_resource == 'edit':\n value = self.model.model.roles\n roles_widget = factory(\n 'field:label:select',\n name='principal_roles',\n value=value,\n props={\n 'label': _('roles', default='Roles'),\n 'multivalued': True,\n 'vocabulary': self.roles_vocab,\n 'format': 'single',\n 'listing_tag': 'ul',\n 'listing_label_position': 'after',\n })\n save_widget = self.form['save']\n self.form.insertbefore(roles_widget, save_widget)", "response": "Hook after prepare and set principal_roles as selection to\n form."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nextracting expiration information from widget.", "response": "def expiration_extractor(widget, data):\n \"\"\"Extract expiration information.\n\n - If active flag not set, Account is disabled (value 0).\n - If active flag set and value is UNSET, account never expires.\n - If active flag set and datetime choosen, account expires at given\n datetime.\n - Timestamp in seconds since epoch is returned.\n \"\"\"\n active = int(data.request.get('%s.active' % widget.name, '0'))\n if not active:\n return 0\n expires = data.extracted\n if expires:\n return time.mktime(expires.utctimetuple())\n return UNSET"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef prepare(_next, self):\n _next(self)\n cfg = ugm_general(self.model)\n if cfg.attrs['users_account_expiration'] != 'True':\n return\n mode = 'edit'\n if not self.request.has_permission(\n 'manage_expiration', self.model.parent):\n mode = 'display'\n if self.action_resource == 'edit':\n attr = cfg.attrs['users_expires_attr']\n unit = int(cfg.attrs['users_expires_unit'])\n value = int(self.model.attrs.get(attr, 0))\n # if format days, convert to seconds\n if unit == 0:\n value *= 86400\n else:\n value = UNSET\n expires_widget = factory(\n 'field:label:expiration',\n name='active',\n value=value,\n props={\n 'label': _('active', default='Active')\n },\n mode=mode\n )\n save_widget = self.form['save']\n self.form.insertbefore(expires_widget, save_widget)", "response": "Hook after prepare and set expiration widget to\n self. form."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef portrait_image(model, request):\n response = Response()\n cfg = ugm_general(model)\n response.body = model.attrs[cfg.attrs['users_portrait_attr']]\n response.headers['Content-Type'] = 'image/jpeg'\n response.headers['Cache-Control'] = 'max-age=0'\n return response", "response": "XXX: needs polishing. Return configured default portrait if not set\n on user."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef prepare(_next, self):\n _next(self)\n if not self.portrait_support:\n return\n model = self.model\n request = self.request\n if request.has_permission('edit_user', model.parent):\n mode = 'edit'\n else:\n mode = 'display'\n cfg = ugm_general(model)\n image_attr = cfg.attrs['users_portrait_attr']\n image_accept = cfg.attrs['users_portrait_accept']\n image_width = int(cfg.attrs['users_portrait_width'])\n image_height = int(cfg.attrs['users_portrait_height'])\n image_data = model.attrs.get(image_attr)\n if image_data:\n image_value = {\n 'file': BytesIO(image_data),\n 'mimetype': 'image/jpeg',\n }\n image_url = make_url(request, node=model,\n resource='portrait_image')\n else:\n image_value = UNSET\n resource = 'cone.ugm.static/images/default_portrait.jpg'\n image_url = make_url(request, node=model.root, resource=resource)\n portrait_widget = factory(\n 'field:label:error:image',\n name='portrait',\n value=image_value,\n props={\n 'label': _('portrait', default='Portrait'),\n 'src': image_url,\n 'alt': _('portrait', default='Portrait'),\n 'accept': image_accept,\n 'minsize': (image_width, image_height),\n 'crop': {\n 'size': (image_width, image_height),\n 'fitting': True,\n }\n },\n mode=mode)\n save_widget = self.form['save']\n self.form.insertbefore(portrait_widget, save_widget)", "response": "Hook after prepare and set portrait as image widget to\n self. form."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nflag whether local manager ACL should be considered for current authenticated user.", "response": "def local_manager_consider_for_user(self):\n \"\"\"Flag whether local manager ACL should be considered for current\n authenticated user.\n \"\"\"\n if not self.local_management_enabled:\n return False\n request = get_current_request()\n if authenticated_userid(request) == security.ADMIN_USER:\n return False\n roles = security.authenticated_user(request).roles\n if 'admin' in roles or 'manager' in roles:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef local_manager_gid(self):\n config = self.root['settings']['ugm_localmanager'].attrs\n user = security.authenticated_user(get_current_request())\n if not user:\n return None\n gids = user.group_ids\n adm_gids = list()\n for gid in gids:\n rule = config.get(gid)\n if rule:\n adm_gids.append(gid)\n if len(adm_gids) == 0:\n return None\n if len(adm_gids) > 1:\n msg = (u\"Authenticated member defined in local manager \"\n u\"groups %s but only one management group allowed for \"\n u\"each user. Please contact System Administrator in \"\n u\"order to fix this problem.\")\n exc = msg % ', '.join([\"'%s'\" % gid for gid in adm_gids])\n raise Exception(exc)\n return adm_gids[0]", "response": "Return the local manager group id of current authenticated member."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef local_manager_rule(self):\n adm_gid = self.local_manager_gid\n if not adm_gid:\n return None\n config = self.root['settings']['ugm_localmanager'].attrs\n return config[adm_gid]", "response": "Return rule for local manager."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ntargeting uid s for local manager.", "response": "def local_manager_target_uids(self):\n \"\"\"Target uid's for local manager.\n \"\"\"\n groups = self.root['groups'].backend\n managed_uids = set()\n for gid in self.local_manager_target_gids:\n group = groups.get(gid)\n if group:\n managed_uids.update(group.member_ids)\n return list(managed_uids)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef local_manager_is_default(self, adm_gid, gid):\n config = self.root['settings']['ugm_localmanager'].attrs\n rule = config[adm_gid]\n if gid not in rule['target']:\n raise Exception(u\"group '%s' not managed by '%s'\" % (gid, adm_gid))\n return gid in rule['default']", "response": "Check whether the local manager group is default group."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nhook optional_login extractor if necessary for form defaults.", "response": "def form_field_definitions(self):\n \"\"\"Hook optional_login extractor if necessary for form defaults.\n \"\"\"\n schema = copy.deepcopy(form_field_definitions.user)\n uid, login = self._get_auth_attrs()\n if uid != login:\n field = schema.get(login, schema['default'])\n if field['chain'].find('*optional_login') == -1:\n field['chain'] = '%s:%s' % (\n '*optional_login', field['chain'])\n if not field.get('custom'):\n field['custom'] = dict()\n field['custom']['optional_login'] = \\\n (['context.optional_login'], [], [], [], [])\n schema[login] = field\n return schema"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nadds user via remote service.", "response": "def remote_add_user(model, request):\n \"\"\"Add user via remote service.\n\n Returns a JSON response containing success state and a message indicating\n what happened::\n\n {\n success: true, // respective false\n message: 'message'\n }\n\n Expected request parameters:\n\n id\n New user id.\n\n password\n User password to be set initially (optional).\n\n roles\n Comma seperated role names the user initially has.\n\n groups\n Comma seperated groups names the user should initially be member of.\n\n attr.*\n User attributes to be set. I.e. ``attr.mail`` would set the mail\n attribute for newly created user. All request parameters prefixed with\n ``attr`` get checked against user attribute attrmap from settings.\n\n Restrictions - All values, whether single or multi valued, are passed\n as string or list of strings to the create function.\n \"\"\"\n params = request.params\n uid = params.get('id')\n\n if not uid:\n return {\n 'success': False,\n 'message': u\"No user ID given.\",\n }\n\n users = model.backend\n if uid in users:\n return {\n 'success': False,\n 'message': u\"User with given ID already exists.\",\n }\n\n password = params.get('password')\n\n add_roles = params.get('roles', '')\n add_roles = [val.strip() for val in add_roles.split(',') if val]\n\n add_groups = params.get('groups', '')\n add_groups = [val.strip() for val in add_groups.split(',') if val]\n\n attrs = dict()\n for key, val in params.items():\n if not key.startswith('attr.'):\n continue\n key = key[key.find('.') + 1:]\n attrs[key] = val\n\n settings = ugm_users(model)\n attrmap = settings.attrs.users_form_attrmap\n exposed = settings.attrs.users_exposed_attributes\n if not exposed:\n exposed = list()\n valid_attrs = attrmap.keys() + exposed\n checked_attrs = dict()\n for key in valid_attrs:\n val = attrs.get(key)\n if not val:\n continue\n checked_attrs[key] = val\n\n try:\n user = users.create(uid, **checked_attrs)\n message = u\"\"\n\n from cone.app.security import DEFAULT_ROLES\n available_roles = [role[0] for role in DEFAULT_ROLES]\n for role in add_roles:\n if role not in available_roles:\n message += u\"Role '%s' given but inexistent. \" % role\n continue\n user.add_role(role)\n\n groups = users.parent.groups\n for group in add_groups:\n if group not in groups:\n message += u\"Group '%s' given but inexistent. \" % group\n continue\n groups[group].add(uid)\n\n users.parent()\n\n if password is not None:\n users.passwd(uid, None, password)\n\n message += u\"Created user with ID '%s'.\" % uid\n return {\n 'success': True,\n 'message': message,\n }\n except Exception as e:\n return {\n 'success': False,\n 'message': str(e),\n }\n finally:\n model.invalidate()"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nremoves user via remote service.", "response": "def remote_delete_user(model, request):\n \"\"\"Remove user via remote service.\n\n Returns a JSON response containing success state and a message indicating\n what happened::\n\n {\n success: true, // respective false\n message: 'message'\n }\n\n Expected request parameters:\n\n id\n Id of user to delete.\n \"\"\"\n params = request.params\n uid = params.get('id')\n\n if not uid:\n return {\n 'success': False,\n 'message': u\"No user ID given.\",\n }\n\n users = model.backend\n if uid not in users:\n return {\n 'success': False,\n 'message': u\"User with given ID not exists.\",\n }\n\n try:\n del users[uid]\n users.parent()\n\n message = u\"Deleted user with ID '%s'.\" % uid\n return {\n 'success': True,\n 'message': message,\n }\n except Exception as e:\n return {\n 'success': False,\n 'message': str(e),\n }\n finally:\n model.invalidate()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_formset(self, request, obj=None, **kwargs):\n if obj is not None:\n try:\n # this is enough to load the new schema\n obj.external\n except LayerExternal.DoesNotExist:\n pass\n return super(LayerExternalInline, self).get_formset(request, obj=None, **kwargs)", "response": "Load Synchronizer schema to display specific fields in administered by the user."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef public_broadcaster():\n while __websocket_server_running__:\n pipein = open(PUBLIC_PIPE, 'r')\n line = pipein.readline().replace('\\n', '').replace('\\r', '')\n if line != '':\n WebSocketHandler.broadcast(line)\n print line\n \n remaining_lines = pipein.read()\n pipein.close()\n pipeout = open(PUBLIC_PIPE, 'w')\n pipeout.write(remaining_lines)\n pipeout.close()\n else:\n pipein.close()\n \n time.sleep(0.05)", "response": "A thread which checks for new messages in the public pipe and broadcasts them to all connected clients."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nthread which checks for new messages in the private pipe and sends them to the specific client.", "response": "def private_messenger():\n \"\"\"\n Thread which runs in parallel and constantly checks for new messages\n in the private pipe and sends them to the specific client.\n If client is not connected the message is discarded.\n \"\"\"\n while __websocket_server_running__:\n pipein = open(PRIVATE_PIPE, 'r')\n line = pipein.readline().replace('\\n', '').replace('\\r', '')\n if line != '':\n message = json.loads(line)\n WebSocketHandler.send_private_message(user_id=message['user_id'],\n message=message)\n print line\n \n remaining_lines = pipein.read()\n pipein.close()\n pipeout = open(PRIVATE_PIPE, 'w')\n pipeout.write(remaining_lines)\n pipeout.close()\n else:\n pipein.close()\n \n time.sleep(0.05)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef save(self, *args, **kwargs):\n if not self.pk and self.model.polarization:\n self.polarization = self.model.polarization\n super(Antenna, self).save(*args, **kwargs)", "response": "Save the antenna record."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns list of open 311 services", "response": "def get(self, request, *args, **kwargs):\n \"\"\" return list of open 311 services \"\"\"\n # init django rest framework specific stuff\n serializer_class = self.get_serializer_class()\n context = self.get_serializer_context()\n\n # init empty list\n services = []\n\n # loop over each service\n for service_type in SERVICES.keys():\n # initialize serializers for layer\n services.append(\n serializer_class(\n object(),\n context=context,\n service_type=service_type\n ).data\n )\n\n return Response(services)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_serializer(self, instance=None, data=None,\n many=False, partial=False):\n \"\"\"\n Return the serializer instance that should be used for validating and\n deserializing input, and for serializing output.\n \"\"\"\n serializers = {\n 'node': NodeRequestListSerializer,\n 'vote': VoteRequestListSerializer,\n 'comment': CommentRequestListSerializer,\n 'rate': RatingRequestListSerializer,\n }\n context = self.get_serializer_context()\n service_code = context['request'].query_params.get('service_code', 'node')\n\n if service_code not in serializers.keys():\n serializer_class = self.get_serializer_class()\n else:\n serializer_class = serializers[service_code]\n\n return serializer_class(instance, many=many, partial=partial, context=context)", "response": "Returns the serializer instance that should be used for validating and\n deserializing input and serializing output."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nretrieves list of service requests", "response": "def get(self, request, *args, **kwargs):\n \"\"\" Retrieve list of service requests \"\"\"\n if 'service_code' not in request.GET.keys():\n return Response({ 'detail': _('A service code must be inserted') }, status=404)\n\n service_code = request.GET['service_code']\n\n if service_code not in SERVICES.keys():\n return Response({ 'detail': _('Service not found') }, status=404)\n\n start_date = None\n end_date = None\n status = None\n layer = None\n\n STATUSES = {}\n for status_type in ('open', 'closed'):\n STATUSES[status_type] = [k for k, v in STATUS.items() if v == status_type]\n\n if 'start_date' in request.GET.keys():\n start_date = request.GET['start_date']\n if iso8601_REGEXP.match(start_date) is None:\n return Response({ 'detail': _('Invalid date inserted') }, status=404)\n\n if 'end_date' in request.GET.keys():\n end_date = request.GET['end_date']\n if iso8601_REGEXP.match(end_date) is None:\n return Response({ 'detail': _('Invalid date inserted') }, status=404)\n\n if 'status' in request.GET.keys():\n if request.GET['status'] not in ('open','closed'):\n return Response({ 'detail': _('Invalid status inserted') }, status=404)\n status = request.GET['status']\n\n if 'layer' in request.GET.keys():\n layer = request.GET['layer']\n node_layer = get_object_or_404(Layer, slug=layer)\n\n service_model = MODELS[service_code]\n if service_code in ('vote', 'comment', 'rate'):\n self.queryset = service_model.objects.none()\n else:\n self.queryset = service_model.objects.all()\n\n # Filter by layer\n if layer is not None:\n self.queryset = self.queryset.filter(layer = node_layer)\n\n # Check of date parameters\n if start_date is not None and end_date is not None:\n self.queryset = self.queryset.filter(added__gte = start_date).filter(added__lte = end_date)\n\n if start_date is not None and end_date is None:\n self.queryset = self.queryset.filter(added__gte = start_date)\n\n if start_date is None and end_date is not None:\n self.queryset = self.queryset.filter(added__lte = end_date)\n\n # Check of status parameter\n if status is not None:\n q_list = [Q(status__slug__exact = s) for s in STATUSES[status]]\n self.queryset = self.queryset.filter(reduce(operator.or_, q_list))\n return self.list(request, *args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\npost a new object in a service", "response": "def post(self, request, *args, **kwargs):\n \"\"\" Post a service request (requires authentication) \"\"\"\n service_code = request.data['service_code']\n\n if service_code not in SERVICES.keys():\n return Response({ 'detail': _('Service not found') }, status=404)\n\n serializers = {\n 'node': NodeRequestSerializer,\n 'vote': VoteRequestSerializer,\n 'comment': CommentRequestSerializer,\n 'rate': RatingRequestSerializer,\n }\n\n # init right serializer\n kwargs['service_code'] = service_code\n kwargs['serializer'] = serializers[service_code]\n\n user=self.get_custom_data()\n request.UPDATED = request.data.copy()\n request.UPDATED['user'] = user['user']\n if service_code == 'node':\n for checkPOSTdata in ('layer','name','lat','long'):\n # Check if mandatory parameters key exists\n if checkPOSTdata not in request.data.keys():\n return Response({ 'detail': _('Mandatory parameter not found') }, status=400)\n else:\n # Check if mandatory parameters values have been inserted\n if not request.data[checkPOSTdata] :\n return Response({ 'detail': _('Mandatory parameter not found') }, status=400)\n # Get layer id\n layer = Layer.objects.get(slug=request.UPDATED['layer'])\n request.UPDATED['layer'] = layer.id\n\n # Transform coords in wkt geometry\n lat = float(request.UPDATED['lat'])\n long = float(request.UPDATED['long'])\n point = Point((long, lat))\n request.UPDATED['geometry'] = point.wkt\n request.UPDATED['slug'] = slugify(request.UPDATED['name'])\n\n return self.create(request, *args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nadd user info to ExtensibleNodeSerializer", "response": "def ready(self):\n \"\"\" Add user info to ExtensibleNodeSerializer \"\"\"\n from nodeshot.core.nodes.base import ExtensibleNodeSerializer\n from .models import Vote\n from .serializers import (CommentRelationSerializer,\n ParticipationSerializer)\n\n ExtensibleNodeSerializer.add_relationship(\n 'comments',\n serializer=CommentRelationSerializer,\n many=True,\n queryset=lambda obj, request: obj.comment_set.all()\n )\n\n ExtensibleNodeSerializer.add_relationship(\n 'counts',\n serializer=ParticipationSerializer,\n queryset=lambda obj, request: obj.noderatingcount\n )\n\n ExtensibleNodeSerializer.add_relationship(\n 'votes_url',\n view_name='api_node_votes',\n lookup_field='slug'\n )\n\n ExtensibleNodeSerializer.add_relationship(\n 'ratings_url',\n view_name='api_node_ratings',\n lookup_field='slug'\n )\n\n ExtensibleNodeSerializer.add_relationship(\n 'comments_url',\n view_name='api_node_comments',\n lookup_field='slug'\n )\n\n def voted(obj, request):\n \"\"\"\n Determines if current logged-in user has already voted on a node\n returns 1 if user has already liked\n returns -1 if user has already disliked\n returns False if user hasn't voted or if not authenticated\n \"\"\"\n if request.user.is_authenticated():\n v = Vote.objects.filter(node_id=obj.id, user_id=request.user.id)\n if len(v) > 0:\n return v[0].vote\n # hasn't voted yet or not authenticated\n return False\n\n ExtensibleNodeSerializer.add_relationship(\n 'voted',\n function=voted\n )\n\n ExtensibleNodeSerializer.add_relationship(\n 'voting_allowed',\n function=lambda obj, request: obj.voting_allowed\n )\n\n ExtensibleNodeSerializer.add_relationship(\n 'rating_allowed',\n function=lambda obj, request: obj.rating_allowed\n )\n\n ExtensibleNodeSerializer.add_relationship(\n 'comments_allowed',\n function=lambda obj, request: obj.comments_allowed\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef node_created_handler(sender, **kwargs):\n if kwargs['created']:\n obj = kwargs['instance']\n queryset = exclude_owner_of_node(obj)\n create_notifications.delay(**{\n \"users\": queryset,\n \"notification_model\": Notification,\n \"notification_type\": \"node_created\",\n \"related_object\": obj\n })", "response": "send notification when a new node is created according to users s settings"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsend notification when the status of a node changes according to users s settings", "response": "def node_status_changed_handler(**kwargs):\n \"\"\" send notification when the status of a node changes according to users's settings \"\"\"\n obj = kwargs['instance']\n obj.old_status = kwargs['old_status'].name\n obj.new_status = kwargs['new_status'].name\n queryset = exclude_owner_of_node(obj)\n create_notifications.delay(**{\n \"users\": queryset,\n \"notification_model\": Notification,\n \"notification_type\": \"node_status_changed\",\n \"related_object\": obj\n })\n\n # if node has owner send a different notification to him\n if obj.user is not None:\n create_notifications.delay(**{\n \"users\": [obj.user],\n \"notification_model\": Notification,\n \"notification_type\": \"node_own_status_changed\",\n \"related_object\": obj\n })"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef diff(self):\n latest = self.latest\n current = NetJsonParser(self.json())\n return diff(current, latest)", "response": "returns the diff of the current and latest entries"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef json(self):\n nodes = []\n links = []\n\n for link in self.link_set.all():\n if self.is_layer2:\n source = link.interface_a.mac\n destination = link.interface_b.mac\n else:\n source = str(link.interface_a.ip_set.first().address)\n destination = str(link.interface_b.ip_set.first().address)\n nodes.append({\n 'id': source\n })\n nodes.append({\n 'id': destination\n })\n links.append(OrderedDict((\n ('source', source),\n ('target', destination),\n ('cost', link.metric_value)\n )))\n\n return OrderedDict((\n ('type', 'NetworkGraph'),\n ('protocol', self.parser.protocol),\n ('version', self.parser.version),\n ('metric', self.parser.metric),\n ('nodes', nodes),\n ('links', links)\n ))", "response": "returns a dict that represents a NetJSON NetworkGraph object"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nupdating the topology with the current state of the entries in the diff table.", "response": "def update(self):\n \"\"\"\n Updates topology\n Links are not deleted straightaway but set as \"disconnected\"\n \"\"\"\n from .link import Link # avoid circular dependency\n diff = self.diff()\n\n status = {\n 'added': 'active',\n 'removed': 'disconnected',\n 'changed': 'active'\n }\n\n for section in ['added', 'removed', 'changed']:\n # section might be empty\n if not diff[section]:\n continue\n for link_dict in diff[section]['links']:\n try:\n link = Link.get_or_create(source=link_dict['source'],\n target=link_dict['target'],\n cost=link_dict['cost'],\n topology=self)\n except (LinkDataNotFound, ValidationError) as e:\n msg = 'Exception while updating {0}'.format(self.__repr__())\n logger.exception(msg)\n print('{0}\\n{1}\\n'.format(msg, e))\n continue\n link.ensure(status=status[section],\n cost=link_dict['cost'])"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef clean(self, *args, **kwargs):\n if self.synchronizer_path != 'None' and self.config:\n # call synchronizer custom clean\n try:\n self.synchronizer.load_config(self.config)\n self.synchronizer.clean()\n except ImproperlyConfigured as e:\n raise ValidationError(e.message)", "response": "Call self. synchronizer. clean method"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsaving the object to the database", "response": "def save(self, *args, **kwargs):\n \"\"\"\n call synchronizer \"after_external_layer_saved\" method\n for any additional operation that must be executed after save\n \"\"\"\n after_save = kwargs.pop('after_save', True)\n super(LayerExternal, self).save(*args, **kwargs)\n # call after_external_layer_saved method of synchronizer\n if after_save:\n try:\n synchronizer = self.synchronizer\n except ImproperlyConfigured:\n pass\n else:\n if synchronizer:\n synchronizer.after_external_layer_saved(self.config)\n # reload schema\n self._reload_schema()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_can_edit(self, obj):\n view = self.context.get('view')\n request = copy(self.context.get('request'))\n request._method = 'PUT'\n try:\n view.check_object_permissions(request, obj)\n except (PermissionDenied, NotAuthenticated):\n return False\n else:\n return True", "response": "returns true if user has permission to edit the object"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning uri of API image resource", "response": "def get_details(self, obj):\n \"\"\" returns uri of API image resource \"\"\"\n args = {\n 'slug': obj.node.slug,\n 'pk': obj.pk\n }\n return reverse('api_node_image_detail',\n kwargs=args,\n request=self.context.get('request', None))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn True if the user has permission to view the social - link - list.", "response": "def has_permission(self, request, view):\n \"\"\" applies to social-link-list \"\"\"\n if request.method == 'POST':\n user = Profile.objects.only('id', 'username').get(username=view.kwargs['username'])\n return request.user.id == user.id\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef delete(self, *args, **kwargs):\n try:\n os.remove(self.file.file.name)\n # image does not exist\n except (OSError, IOError):\n pass\n super(Image, self).delete(*args, **kwargs)", "response": "delete image when an image record is deleted"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nuses to determine color of links", "response": "def get_quality(self, type='etx'):\n \"\"\" used to determine color of links\"\"\"\n if type == 'etx':\n if 0 < self.etx < 1.5:\n quality = 1\n elif self.etx < 3:\n quality = 2\n else:\n quality = 3\n elif type == 'dbm':\n if -83 < self.dbm < 0:\n quality = 1\n elif self.dbm > -88:\n quality = 2\n else:\n quality = 3\n return quality"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nensure new nodes are allowed for this layer", "response": "def new_nodes_allowed_for_layer(self):\n \"\"\"\n ensure new nodes are allowed for this layer\n \"\"\"\n if not self.pk and self.layer and not self.layer.new_nodes_allowed:\n raise ValidationError(_('New nodes are not allowed for this layer'))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nvalidating that the minimum distance of a node is less than the minimum distance of other nodes.", "response": "def nodes_minimum_distance_validation(self):\n \"\"\"\n if minimum distance is specified, ensure node is not too close to other nodes;\n \"\"\"\n if self.layer and self.layer.nodes_minimum_distance:\n minimum_distance = self.layer.nodes_minimum_distance\n # TODO - lower priority: do this check only when coordinates are changing\n near_nodes = Node.objects.exclude(pk=self.id).filter(geometry__distance_lte=(self.geometry, D(m=minimum_distance))).count()\n if near_nodes > 0:\n raise ValidationError(_('Distance between nodes cannot be less than %s meters') % minimum_distance)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef node_contained_in_layer_area_validation(self):\n # if area is a polygon ensure it contains the node\n if self.layer and isinstance(self.layer.area, Polygon) and not self.layer.area.contains(self.geometry):\n raise ValidationError(_('Node must be inside layer area'))", "response": "Ensure that the node coordinates are contained in the area of the layer."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef save(self, *args, **kwargs):\n super(Layer, self).save(*args, **kwargs)\n\n # if is_published of an existing layer changes\n if self.pk and self.is_published != self._current_is_published:\n # send django signal\n layer_is_published_changed.send(\n sender=self.__class__,\n instance=self,\n old_is_published=self._current_is_published,\n new_is_published=self.is_published\n )\n # unpublish nodes\n self.update_nodes_published()\n\n # update _current_is_published\n self._current_is_published = self.is_published", "response": "intercepts changes to is_published and fires layer_is_published_changed signal"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef update_nodes_published(self):\n if self.pk:\n self.node_set.all().update(is_published=self.is_published)", "response": "Update published nodes of current layer"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of unread notifications and mark as read", "response": "def get_unread(self, request, notifications, mark_as_read):\n \"\"\"\n return unread notifications and mark as read\n (unless read=false param is passed)\n \"\"\"\n notifications = notifications.filter(is_read=False)\n serializer = UnreadNotificationSerializer(list(notifications), # evaluate queryset\n many=True,\n context=self.get_serializer_context())\n # retrieve unread notifications as read (default behaviour)\n if mark_as_read:\n notifications.update(is_read=True)\n return Response(serializer.data)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the count of unread notifications", "response": "def get_count(self, request, notifications, mark_as_read=False):\n \"\"\" return count of unread notification \"\"\"\n return Response({'count': notifications.filter(is_read=False).count()})"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_all(self, request, notifications, mark_as_read=False):\n return self.list(request, notifications)", "response": "Returns all the notifications"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets object from queryset", "response": "def get_object(self, queryset=None):\n \"\"\" get privacy settings of current user \"\"\"\n try:\n obj = self.get_queryset()\n except self.model.DoesNotExist:\n raise Http404()\n self.check_object_permissions(self.request, obj)\n return obj"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfind a link between source and target.", "response": "def get_link(cls, source, target, topology=None):\n \"\"\"\n Find link between source and target, (or vice versa, order is irrelevant).\n :param source: ip or mac addresses\n :param target: ip or mac addresses\n :param topology: optional topology relation\n :returns: Link object\n :raises: LinkNotFound\n \"\"\"\n a = source\n b = target\n # ensure parameters are coherent\n if not (valid_ipv4(a) and valid_ipv4(b)) and not (valid_ipv6(a) and valid_ipv6(b)) and not (valid_mac(a) and valid_mac(b)):\n raise ValueError('Expecting valid ipv4, ipv6 or mac address')\n # get interfaces\n a = cls._get_link_interface(a)\n b = cls._get_link_interface(b)\n # raise LinkDataNotFound if an interface is not found\n not_found = []\n if a is None:\n not_found.append(source)\n if b is None:\n not_found.append(target)\n if not_found:\n msg = 'the following interfaces could not be found: {0}'.format(', '.join(not_found))\n raise LinkDataNotFound(msg)\n # find link with interfaces\n # inverse order is also ok\n q = (Q(interface_a=a, interface_b=b) | Q(interface_a=b, interface_b=a))\n # add topology to lookup\n if topology:\n q = q & Q(topology=topology)\n link = Link.objects.filter(q).first()\n if link is None:\n raise LinkNotFound('Link matching query does not exist',\n interface_a=a,\n interface_b=b,\n topology=topology)\n return link"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_or_create(cls, source, target, cost, topology=None):\n try:\n return cls.get_link(source, target, topology)\n except LinkNotFound as e:\n pass\n # create link\n link = Link(interface_a=e.interface_a,\n interface_b=e.interface_b,\n status=LINK_STATUS['active'],\n metric_value=cost,\n topology=topology)\n link.full_clean()\n link.save()\n return link", "response": "Tries to find a link with get_link creates a new link if it doesn t exist."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef ensure(self, status, cost):\n changed = False\n status_id = LINK_STATUS[status]\n if self.status != status_id:\n self.status = status_id\n changed = True\n if self.metric_value != cost:\n self.metric_value = cost\n changed = True\n if changed:\n self.save()", "response": "ensure link properties correspond to the specified ones\n perform save operation only if necessary\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsend the message to the admin.", "response": "def send(self, request, queryset):\n \"\"\"\n Send action available in change outward list\n \"\"\"\n send_outward_mails.delay(queryset)\n # show message in the admin\n messages.info(request, _('Message sent successfully'), fail_silently=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef post(self, request, *args, **kwargs):\n try:\n self.get_object(**kwargs)\n except Http404:\n return Response({'detail': _('Not Found')}, status=404)\n\n if not self.is_contactable():\n return Response({'detail': _('This resource cannot be contacted')}, status=400)\n\n content_type = ContentType.objects.only('id', 'model').get(model=self.content_type)\n # shortcut\n data = request.data\n # init inward message\n message = Inward()\n # required fields\n message.content_type = content_type\n message.object_id = self.recipient.id\n message.message = data.get('message')\n # user info if authenticated\n if request.user.is_authenticated():\n message.user = request.user\n else:\n message.from_name = data.get('from_name')\n message.from_email = data.get('from_email')\n # additional user info\n message.ip = request.META.get('REMOTE_ADDR', '')\n message.user_agent = request.META.get('HTTP_USER_AGENT', '')\n message.accept_language = request.META.get('HTTP_ACCEPT_LANGUAGE', '')\n try:\n message.full_clean()\n except ValidationError, e:\n return Response(e.message_dict, status=400)\n message.save()\n\n if message.status >= 1:\n return Response({'details': _('Message sent successfully')}, status=201)\n else:\n return Response({'details': _('Something went wrong. The email was not sent.')}, status=500)", "response": "This is the HTTP POST method."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the profile of current user if authenticated otherwise 401", "response": "def get(self, request, *args, **kwargs):\n \"\"\" return profile of current user if authenticated otherwise 401 \"\"\"\n serializer = self.serializer_reader_class\n if request.user.is_authenticated():\n return Response(serializer(request.user, context=self.get_serializer_context()).data)\n else:\n return Response({'detail': _('Authentication credentials were not provided')}, status=401)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef post(self, request, format=None):\n serializer_class = self.get_serializer_class()\n serializer = serializer_class(data=request.data, instance=request.user)\n\n if serializer.is_valid():\n serializer.save()\n return Response({'detail': _(u'Password successfully changed')})\n\n return Response(serializer.errors, status=400)", "response": "Validate password change operation and return result"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sync(self):\n self.before_start()\n self.retrieve_data()\n self.parse()\n\n # TRICK: disable new_nodes_allowed_for_layer validation\n try:\n Node._additional_validation.remove('new_nodes_allowed_for_layer')\n except ValueError as e:\n print \"WARNING! got exception: %s\" % e\n # avoid sending zillions of notifications\n pause_disconnectable_signals()\n\n self.save()\n\n # Re-enable new_nodes_allowed_for_layer validation\n try:\n Node._additional_validation.insert(0, 'new_nodes_allowed_for_layer')\n except ValueError as e:\n print \"WARNING! got exception: %s\" % e\n # reconnect signals\n resume_disconnectable_signals()\n\n self.after_complete()\n\n # return message as a list because more than one messages might be returned\n return [self.message]", "response": "This method is used to synchronize the internal state of the internal state of the internal state of the external source with the internal state of the external source."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef retrieve_data(self):\n # shortcuts for readability\n url = self.config.get('url')\n timeout = float(self.config.get('timeout', 10))\n # perform HTTP request and store content\n self.data = requests.get(url, verify=self.verify_ssl, timeout=timeout).content", "response": "retrieve data from an HTTP URL"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_text(item, tag, default=False):\n try:\n xmlnode = item.getElementsByTagName(tag)[0].firstChild\n except IndexError as e:\n if default is not False:\n return default\n else:\n raise IndexError(e)\n\n if xmlnode is not None:\n return unicode(xmlnode.nodeValue)\n # empty tag\n else:\n return ''", "response": "returns text content of an xml tag"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts a parsed item as input and returns a python dictionary containing the keys of the node model that can be used to store the item in the hstore. data field.", "response": "def _convert_item(self, item):\n \"\"\"\n take a parsed item as input and returns a python dictionary\n the keys will be saved into the Node model\n either in their respective fields or in the hstore \"data\" field\n\n :param item: object representing parsed item\n \"\"\"\n item = self.parse_item(item)\n\n # name is required\n if not item['name']:\n raise Exception('Expected property %s not found in item %s.' % (self.keys['name'], item))\n elif len(item['name']) > 75:\n item['name'] = item['name'][:75]\n\n if not item['status']:\n item['status'] = self.default_status\n\n # get status or get default status or None\n try:\n item['status'] = Status.objects.get(slug__iexact=item['status'])\n except Status.DoesNotExist:\n try:\n item['status'] = Status.objects.create(name=item['status'],\n slug=slugify(item['status']),\n description=item['status'],\n is_default=False)\n except Exception as e:\n logger.exception(e)\n item['status'] = None\n\n # slugify slug\n item['slug'] = slugify(item['name'])\n\n if not item['address']:\n item['address'] = ''\n\n if not item['is_published']:\n item['is_published'] = ''\n\n User = get_user_model()\n # get user or None\n try:\n item['user'] = User.objects.get(username=item['user'])\n except User.DoesNotExist:\n item['user'] = None\n\n if not item['elev']:\n item['elev'] = None\n\n if not item['description']:\n item['description'] = ''\n\n if not item['notes']:\n item['notes'] = ''\n\n if item['added']:\n # convert dates to python datetime\n try:\n item['added'] = parser.parse(item['added'])\n except Exception as e:\n print \"Exception while parsing 'added' date: %s\" % e\n if item['updated']:\n try:\n item['updated'] = parser.parse(item['updated'])\n except Exception as e:\n print \"Exception while parsing 'updated' date: %s\" % e\n\n result = {\n \"name\": item['name'],\n \"slug\": item['slug'],\n \"status\": item['status'],\n \"address\": item['address'],\n \"is_published\": item['is_published'],\n \"user\": item['user'],\n \"geometry\": item['geometry'],\n \"elev\": item['elev'],\n \"description\": item['description'],\n \"notes\": item['notes'],\n \"added\": item['added'],\n \"updated\": item['updated'],\n \"data\": {}\n }\n\n # ensure all additional data items are strings\n for key, value in item['data'].items():\n result[\"data\"][key] = value\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef save(self):\n self.key_mapping()\n # retrieve all items\n items = self.parsed_data\n\n # init empty lists\n added_nodes = []\n changed_nodes = []\n unmodified_nodes = []\n\n # retrieve a list of all the slugs of this layer\n layer_nodes_slug_list = Node.objects.filter(layer=self.layer).values_list('slug', flat=True)\n # keep a list of all the nodes of other layers\n other_layers_slug_list = Node.objects.exclude(layer=self.layer).values_list('slug', flat=True)\n # init empty list of slug of external nodes that will be needed to perform delete operations\n processed_slug_list = []\n deleted_nodes_count = 0\n\n # loop over every item\n for item in items:\n\n item = self._convert_item(item)\n\n number = 1\n original_name = item['name']\n needed_different_name = False\n\n while True:\n # items might have the same name... so we add a number..\n if item['slug'] in processed_slug_list or item['slug'] in other_layers_slug_list:\n needed_different_name = True\n number = number + 1\n item['name'] = \"%s - %d\" % (original_name, number)\n item['slug'] = slugify(item['name'])\n else:\n if needed_different_name:\n self.verbose('needed a different name for %s, trying \"%s\"' % (original_name, item['name']))\n break\n\n # default values\n added = False\n changed = False\n\n try:\n # edit existing node\n node = Node.objects.get(slug=item['slug'], layer=self.layer)\n except Node.DoesNotExist:\n # add a new node\n node = Node()\n node.layer = self.layer\n added = True\n\n # loop over fields and store data only if necessary\n for field in Node._meta.fields:\n # geometry is a special case, skip\n if field.name == 'geometry':\n continue\n # skip if field is not present in values\n if field.name not in item.keys():\n continue\n # shortcut for value\n value = item[field.name]\n # if value is different than what we have\n if value is not None and getattr(node, field.name) != value:\n # set value\n setattr(node, field.name, value)\n # indicates that a DB query is necessary\n changed = True\n\n if added or (node.geometry.equals(item['geometry']) is False\n and node.geometry.equals_exact(item['geometry']) is False):\n node.geometry = item['geometry']\n changed = True\n\n node.data = node.data or {}\n\n # store any additional key/value in HStore data field\n for key, value in item['data'].items():\n if node.data[key] != value:\n node.data[key] = value\n changed = True\n\n # perform save or update only if necessary\n if added or changed:\n try:\n node.full_clean()\n if None not in [node.added, node.updated]:\n node.save(auto_update=False)\n else:\n node.save()\n except Exception as e:\n raise Exception('error while processing \"%s\": %s' % (node.name, e))\n\n if added:\n added_nodes.append(node)\n self.verbose('new node saved with name \"%s\"' % node.name)\n elif changed:\n changed_nodes.append(node)\n self.verbose('node \"%s\" updated' % node.name)\n else:\n unmodified_nodes.append(node)\n self.verbose('node \"%s\" unmodified' % node.name)\n\n # fill node list container\n processed_slug_list.append(node.slug)\n\n # delete old nodes\n for local_node in layer_nodes_slug_list:\n # if local node not found in external nodes\n if local_node not in processed_slug_list:\n # retrieve from DB and delete\n node = Node.objects.get(slug=local_node)\n # store node name to print it later\n node_name = node.name\n node.delete()\n # then increment count that will be included in message\n deleted_nodes_count = deleted_nodes_count + 1\n self.verbose('node \"%s\" deleted' % node_name)\n\n # message that will be returned\n self.message = \"\"\"\n %s nodes added\n %s nodes changed\n %s nodes deleted\n %s nodes unmodified\n %s total external records processed\n %s total local nodes for this layer\n \"\"\" % (\n len(added_nodes),\n len(changed_nodes),\n deleted_nodes_count,\n len(unmodified_nodes),\n len(items),\n Node.objects.filter(layer=self.layer).count()\n )", "response": "save data into DB"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_queryset(self):\n return self.queryset.all().accessible_to(user=self.request.user)", "response": "Returns only objects which are accessible to the current user."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef disconnect():\n post_save.disconnect(node_created_handler, sender=Node)\n node_status_changed.disconnect(node_status_changed_handler)\n pre_delete.disconnect(node_deleted_handler, sender=Node)", "response": "Disconnects from the node."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreconnect the node to the next update.", "response": "def reconnect():\n \"\"\" reconnect signals \"\"\"\n post_save.connect(node_created_handler, sender=Node)\n node_status_changed.connect(node_status_changed_handler)\n pre_delete.connect(node_deleted_handler, sender=Node)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef clean(self, *args, **kwargs):\n if self.from_user and self.from_user_id == self.to_user_id:\n raise ValidationError(_('A user cannot send a notification to herself/himself'))", "response": "Validate that the from_user and to_user are the same."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef save(self, *args, **kwargs):\n created = self.pk is None\n\n # save notification to database only if user settings allow it\n if self.check_user_settings(medium='web'):\n super(Notification, self).save(*args, **kwargs)\n\n if created:\n # send notifications through other mediums according to user settings\n self.send_notifications()", "response": "custom save method to send email and push notifications through other mediums"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsending email notification according to user settings", "response": "def send_email(self):\n \"\"\" send email notification according to user settings \"\"\"\n # send only if user notification setting is set to true\n if self.check_user_settings():\n send_mail(_(self.type), self.email_message, settings.DEFAULT_FROM_EMAIL, [self.to_user.email])\n return True\n else:\n # return false otherwise\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchecks if the user is ok with receiving this notification through the specified medium.", "response": "def check_user_settings(self, medium='email'):\n \"\"\"\n Ensure user is ok with receiving this notification through the specified medium.\n Available mediums are 'web' and 'email', while 'mobile' notifications will\n hopefully be implemented in the future.\n \"\"\"\n # custom notifications are always sent\n if self.type == 'custom':\n return True\n\n try:\n user_settings = getattr(self.to_user, '%s_notification_settings' % medium)\n except ObjectDoesNotExist:\n # user has no settings specified\n # TODO: it would be better to create the settings with default values\n return False\n\n user_setting_type = getattr(user_settings.__class__, self.type).user_setting_type\n\n if user_setting_type == 'boolean':\n return getattr(user_settings, self.type, True)\n elif user_setting_type == 'distance':\n value = getattr(user_settings, self.type, 0)\n # enabled for all related objects\n if value is 0:\n return True\n # disabled for all related objects\n elif value < 0:\n return False\n # enabled for related objects comprised in specified distance range in km\n else:\n Model = self.related_object.__class__\n geo_field = getattr(user_settings.__class__, self.type).geo_field\n geo_value = getattr(self.related_object, geo_field)\n km = value * 1000\n queryset = Model.objects.filter(**{\n \"user_id\": self.to_user_id,\n geo_field + \"__distance_lte\": (geo_value, km)\n })\n # if user has related object in a distance range less than or equal to\n # his prefered range (specified in number of km), return True and send the notification\n return queryset.count() >= 1"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncomposes complete email message text", "response": "def email_message(self):\n \"\"\" compose complete email message text \"\"\"\n url = settings.SITE_URL\n hello_text = __(\"Hi %s,\" % self.to_user.get_full_name())\n action_text = __(\"\\n\\nMore details here: %s\") % url\n explain_text = __(\n \"This is an automatic notification sent from from %s.\\n\"\n \"If you want to stop receiving this notification edit your\"\n \"email notification settings here: %s\") % (settings.SITE_NAME, 'TODO')\n\n return \"%s\\n\\n%s%s\\n\\n%s\" % (hello_text, self.text, action_text, explain_text)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef generate_key(self, email):\n salt = sha1(str(random())).hexdigest()[:5]\n return sha1(salt + email).hexdigest()", "response": "Generate a new email confirmation key and return it."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating an email confirmation obj from the given email address obj", "response": "def create_emailconfirmation(self, email_address):\n \"Create an email confirmation obj from the given email address obj\"\n confirmation_key = self.generate_key(email_address.email)\n confirmation = self.create(\n email_address=email_address,\n created_at=now(),\n key=confirmation_key,\n )\n return confirmation"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef user_loggedin(sender, **kwargs):\n values = {\n 'value': 1,\n 'path': kwargs['request'].path,\n 'user_id': str(kwargs['user'].pk),\n 'username': kwargs['user'].username,\n }\n write('user_logins', values=values)", "response": "collect metrics about user logins"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncollecting metrics about new users signing up", "response": "def user_deleted(sender, **kwargs):\n \"\"\" collect metrics about new users signing up \"\"\"\n if kwargs.get('created'):\n write('user_variations', {'variation': 1}, tags={'action': 'created'})\n write('user_count', {'total': User.objects.count()})"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning node_rating_count record or create it usage", "response": "def _node_rating_count(self):\n \"\"\"\n Return node_rating_count record\n or create it if it does not exist\n\n usage:\n node = Node.objects.get(pk=1)\n node.rating_count\n \"\"\"\n try:\n return self.noderatingcount\n except ObjectDoesNotExist:\n node_rating_count = NodeRatingCount(node=self)\n node_rating_count.save()\n return node_rating_count"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _node_participation_settings(self):\n try:\n return self.node_participation_settings\n except ObjectDoesNotExist:\n node_participation_settings = NodeParticipationSettings(node=self)\n node_participation_settings.save()\n return node_participation_settings", "response": "Return node_participation_settings record if it does not exist"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncheck if participation actions can be performed on the node level or per node basis.", "response": "def _action_allowed(self, action):\n \"\"\"\n participation actions can be disabled on layer level, or disabled on a per node basis\n \"\"\"\n if getattr(self.layer.participation_settings, '{0}_allowed'.format(action)) is False:\n return False\n else:\n return getattr(self.participation_settings, '{0}_allowed'.format(action))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef create_node_rating_counts_settings(sender, **kwargs):\n created = kwargs['created']\n node = kwargs['instance']\n if created:\n # create node_rating_count and settings\n # task will be executed in background unless settings.CELERY_ALWAYS_EAGER is True\n # if CELERY_ALWAYS_EAGER is False celery worker must be running otherwise task won't be executed\n create_related_object.delay(NodeRatingCount, {'node': node})\n create_related_object.delay(NodeParticipationSettings, {'node': node})", "response": "create node rating count and settings"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating layer rating settings", "response": "def create_layer_rating_settings(sender, **kwargs):\n \"\"\" create layer rating settings \"\"\"\n created = kwargs['created']\n layer = kwargs['instance']\n if created:\n # create layer participation settings\n # task will be executed in background unless settings.CELERY_ALWAYS_EAGER is True\n # if CELERY_ALWAYS_EAGER is False celery worker must be running otherwise task won't be executed\n create_related_object.delay(LayerParticipationSettings, {'layer': layer})"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef save(self, *args, **kwargs):\n created = self.pk is None\n sync_emailaddress = kwargs.pop('sync_emailaddress', True)\n # ensure usable password\n if created and self.has_usable_password() is False:\n self.set_password(self.password)\n # save\n super(Profile, self).save(*args, **kwargs)\n # add user to default group\n if created and self.groups.count() < 1:\n # TODO: make default group configurable in settings\n try:\n default_group = Group.objects.get(name='registered')\n self.groups.add(default_group)\n except Group.DoesNotExist:\n pass\n # keep in sync with EmailAddress model\n if (EMAIL_CONFIRMATION and sync_emailaddress\n and self.email # noqa\n and self.email_set.filter(email=self.email).count() < 1): # noqa\n self.email_set.add_email(self, email=self.email)\n self.email_set.last().set_as_primary()", "response": "Saves the instance to the database."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns True if the user needs email confirmation", "response": "def needs_confirmation(self):\n \"\"\"\n set is_active to False if needs email confirmation\n \"\"\"\n if EMAIL_CONFIRMATION:\n self.is_active = False\n self.save()\n return True\n else:\n return False"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nchange password and sends a signal password_changed signal", "response": "def change_password(self, new_password):\n \"\"\"\n Changes password and sends a signal\n \"\"\"\n self.set_password(new_password)\n self.save()\n password_changed.send(sender=self.__class__, user=self)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef initial(self, request, *args, **kwargs):\n super(NodeRelationViewMixin, self).initial(request, *args, **kwargs)\n self.node = get_object_or_404(Node, **{'slug': self.kwargs['slug']})\n self.queryset = self.model.objects.filter(node_id=self.node.id)", "response": "Custom initial method that creates a node and sets the queryset to only comments of current node"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_notifications(myclass):\n for key, value in TEXTS.items():\n # custom notifications cannot be disabled\n if 'custom' in [key, value]:\n continue\n\n field_type = USER_SETTING[key]['type']\n\n if field_type == 'boolean':\n field = models.BooleanField(_(key), default=DEFAULT_BOOLEAN)\n elif field_type == 'distance':\n field = models.IntegerField(\n _(key),\n default=DEFAULT_DISTANCE,\n help_text=_('-1 (less than 0): disabled; 0: enabled for all;\\\n 1 (less than 0): enabled for those in the specified distance range (km)')\n )\n field.geo_field = USER_SETTING[key]['geo_field']\n\n field.name = field.column = field.attname = key\n field.user_setting_type = field_type\n setattr(myclass, key, field)\n myclass.add_to_class(key, field)\n\n return myclass", "response": "This decorator adds the User Notification Settings fields dynamically to the User Notification Settings models."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchecking if login credentials are correct", "response": "def validate(self, attrs):\n \"\"\" checks if login credentials are correct \"\"\"\n user = authenticate(**self.user_credentials(attrs))\n\n if user:\n if user.is_active:\n self.instance = user\n else:\n raise serializers.ValidationError(_(\"This account is currently inactive.\"))\n else:\n error = _(\"Invalid login credentials.\")\n raise serializers.ValidationError(error)\n return attrs"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_details(self, obj):\n return reverse('api_user_social_links_detail',\n args=[obj.user.username, obj.pk],\n request=self.context.get('request'),\n format=self.context.get('format'))", "response": "Get details url for a user"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_location(self, obj):\n if not obj.city and not obj.country:\n return None\n elif obj.city and obj.country:\n return '%s, %s' % (obj.city, obj.country)\n elif obj.city or obj.country:\n return obj.city or obj.country", "response": "get user s location"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef validate_email(self, value):\n if EMAIL_CONFIRMATION:\n queryset = EmailAddress.objects.filter(email__iexact=value, verified=True)\n else:\n queryset = User.objects.get(email__iexact=value, is_active=True).count() == 0\n if queryset.count() < 1:\n raise serializers.ValidationError(_(\"Email address not found\"))\n return queryset.first().email", "response": "ensure email is in the database"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef update(self, instance, validated_data):\n instance.user.set_password(validated_data[\"password1\"])\n instance.user.full_clean()\n instance.user.save()\n # mark password reset object as reset\n instance.reset = True\n instance.full_clean()\n instance.save()\n return instance", "response": "update user with new password"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef save(self, *args, **kwargs):\n custom_checks = kwargs.pop('custom_checks', True)\n\n super(Device, self).save(*args, **kwargs)\n\n if custom_checks is False:\n return\n\n changed = False\n\n if not self.location:\n self.location = self.node.point\n changed = True\n\n if not self.elev and self.node.elev:\n self.elev = self.node.elev\n changed = True\n\n original_user = self.shortcuts.get('user')\n\n if self.node.user:\n self.shortcuts['user'] = self.node.user\n\n if original_user != self.shortcuts.get('user'):\n changed = True\n\n if 'nodeshot.core.layers' in settings.INSTALLED_APPS:\n original_layer = self.shortcuts.get('layer')\n self.shortcuts['layer'] = self.node.layer\n\n if original_layer != self.shortcuts.get('layer'):\n changed = True\n\n if changed:\n self.save(custom_checks=False)", "response": "Custom save method that automatically inherit node coordinates and elevation and shortcuts if HSTORE is enabled."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create_for_user(self, user):\n # support passing email address too\n if type(user) is unicode:\n from .profile import Profile as User\n user = User.objects.get(email=user)\n\n temp_key = token_generator.make_token(user)\n\n # save it to the password reset model\n password_reset = PasswordReset(user=user, temp_key=temp_key)\n password_reset.save()\n\n # send the password reset email\n subject = _(\"Password reset email sent\")\n message = render_to_string(\"profiles/email_messages/password_reset_key_message.txt\", {\n \"user\": user,\n \"uid\": int_to_base36(user.id),\n \"temp_key\": temp_key,\n \"site_url\": settings.SITE_URL,\n \"site_name\": settings.SITE_NAME\n })\n send_mail(subject, message, settings.DEFAULT_FROM_EMAIL, [user.email])\n\n return password_reset", "response": "create password reset for specified user"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nprompt user what to do when an old node is contained in more than one layer.", "response": "def prompt_layer_selection(self, node, layers):\n \"\"\"Ask user what to do when an old node is contained in more than one layer.\n Possible answers are:\n * use default layer (default answer if pressing enter)\n * choose layer\n * discard node\n \"\"\"\n valid = {\n \"default\": \"default\",\n \"def\": \"default\",\n \"discard\": \"discard\",\n \"dis\": \"discard\",\n }\n question = \"\"\"Cannot automatically determine layer for node \"%s\" because there \\\nare %d layers available in that area, what do you want to do?\\n\\n\"\"\" % (node.name, len(layers))\n\n available_layers = \"\"\n for layer in layers:\n available_layers += \"%d (%s)\\n\" % (layer.id, layer.name)\n valid[str(layer.id)] = layer.id\n\n prompt = \"\"\"\\\nchoose (enter the number of) one of the following layers:\n%s\n\"default\" use default layer (if no default layer specified in settings node will be discarded)\n\"discard\" discard node\n\n(default action is to use default layer)\\n\\n\"\"\" % available_layers\n\n sys.stdout.write(question + prompt)\n\n while True:\n if self.options.get('noinput') is True:\n answer = 'default'\n break\n\n answer = raw_input().lower()\n if answer == '':\n answer = \"default\"\n\n if answer in valid:\n answer = valid[answer]\n break\n else:\n sys.stdout.write(\"Please respond with one of the valid answers\\n\")\n\n sys.stdout.write(\"\\n\")\n return answer"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nensure status map does not contain status values which are not present in DB", "response": "def check_status_mapping(self):\n \"\"\" ensure status map does not contain status values which are not present in DB \"\"\"\n self.verbose('checking status mapping...')\n\n if not self.status_mapping:\n self.message('no status mapping found')\n return\n\n for old_val, new_val in self.status_mapping.iteritems():\n try:\n # look up by slug if new_val is string\n if isinstance(new_val, basestring):\n lookup = {'slug': new_val}\n # lookup by primary key otherwise\n else:\n lookup = {'pk': new_val}\n status = Status.objects.get(**lookup)\n self.status_mapping[old_val] = status.id\n except Status.DoesNotExist:\n raise ImproperlyConfigured('Error! Status with slug %s not found in the database' % new_val)\n\n self.verbose('status map correct')"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef retrieve_nodes(self):\n self.verbose('retrieving nodes from old mysql DB...')\n\n self.old_nodes = list(OldNode.objects.all())\n self.message('retrieved %d nodes' % len(self.old_nodes))", "response": "retrieve nodes from old mysql DB"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef extract_users(self):\n email_set = set()\n users_dict = {}\n\n self.verbose('going to extract user information from retrieved nodes...')\n\n for node in self.old_nodes:\n email_set.add(node.email)\n\n if node.email not in users_dict:\n users_dict[node.email] = {\n 'owner': node.owner\n }\n\n self.email_set = email_set\n self.users_dict = users_dict\n\n self.verbose('%d users extracted' % len(email_set))", "response": "extract user info from retrieved nodes"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsaving admins into local DB", "response": "def import_admins(self):\n \"\"\" save admins to local DB \"\"\"\n self.message('saving admins into local DB')\n\n saved_admins = []\n\n for olduser in OldUser.objects.all():\n try:\n user = User.objects.get(Q(username=olduser.username) | Q(email=olduser.email))\n except User.DoesNotExist:\n user = User()\n except User.MultipleObjectsReturned:\n continue\n\n user.username = olduser.username\n user.password = olduser.password\n user.first_name = olduser.first_name\n user.last_name = olduser.last_name\n user.email = olduser.email\n user.is_active = olduser.is_active\n user.is_staff = olduser.is_staff\n user.is_superuser = olduser.is_superuser\n user.date_joined = olduser.date_joined\n user.full_clean()\n user.save(sync_emailaddress=False)\n saved_admins.append(user)\n\n # mark email address as confirmed if feature is enabled\n if EMAIL_CONFIRMATION and EmailAddress.objects.filter(email=user.email).count() is 0:\n try:\n email_address = EmailAddress(user=user, email=user.email, verified=True, primary=True)\n email_address.full_clean()\n email_address.save()\n except Exception:\n tb = traceback.format_exc()\n self.message('Could not save email address for user %s, got exception:\\n\\n%s' % (user.username, tb))\n\n self.message('saved %d admins into local DB' % len(saved_admins))\n self.saved_admins = saved_admins"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nimport users into DB", "response": "def import_users(self):\n \"\"\" save users to local DB \"\"\"\n self.message('saving users into local DB')\n\n saved_users = self.saved_admins\n\n # loop over all extracted unique email addresses\n for email in self.email_set:\n owner = self.users_dict[email].get('owner')\n\n # if owner is not specified, build username from email\n if owner.strip() == '':\n owner, domain = email.split('@')\n # replace any points with a space\n owner = owner.replace('.', ' ')\n\n # if owner has a space, assume he specified first and last name\n if ' ' in owner:\n owner_parts = owner.split(' ')\n first_name = owner_parts[0]\n last_name = owner_parts[1]\n else:\n first_name = owner\n last_name = ''\n\n # username must be slugified otherwise won't get into the DB\n username = slugify(owner)\n\n # check if user exists first\n try:\n # try looking by email\n user = User.objects.get(email=email)\n except User.DoesNotExist:\n # otherwise init new\n user = User()\n user.username = username\n # generate new password only for new users\n user.password = self.generate_random_password()\n user.is_active = True\n\n # we'll create one user for each unique email address we've got\n user.first_name = first_name.capitalize()\n user.last_name = last_name.capitalize()\n user.email = email\n\n # extract date joined from old nodes\n # find the oldest node of this user\n oldest_node = OldNode.objects.filter(email=email).order_by('added')[0]\n user.date_joined = oldest_node.added\n\n # be sure username is unique\n counter = 1\n original_username = username\n while True:\n # do this check only if user is new\n if not user.pk and User.objects.filter(username=user.username).count() > 0:\n counter += 1\n user.username = '%s%d' % (original_username, counter)\n else:\n break\n\n try:\n # validate data and save\n user.full_clean()\n user.save(sync_emailaddress=False)\n except Exception:\n # if user already exists use that instance\n if(User.objects.filter(email=email).count() == 1):\n user = User.objects.get(email=email)\n # otherwise report error\n else:\n tb = traceback.format_exc()\n self.message('Could not save user %s, got exception:\\n\\n%s' % (user.username, tb))\n continue\n\n # if we got a user to add\n if user:\n # store id\n self.users_dict[email]['id'] = user.id\n # append to saved users\n saved_users.append(user)\n self.verbose('Saved user %s (%s) with email <%s>' % (user.username, user.get_full_name(), user.email))\n\n # mark email address as confirmed if feature is enabled\n if EMAIL_CONFIRMATION and EmailAddress.objects.filter(email=user.email).count() is 0:\n try:\n email_address = EmailAddress(user=user, email=user.email, verified=True, primary=True)\n email_address.full_clean()\n email_address.save()\n except Exception:\n tb = traceback.format_exc()\n self.message('Could not save email address for user %s, got exception:\\n\\n%s' % (user.username, tb))\n\n self.message('saved %d users into local DB' % len(saved_users))\n self.saved_users = saved_users"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef import_nodes(self):\n self.message('saving nodes into local DB...')\n\n saved_nodes = []\n\n # loop over all old node and create new nodes\n for old_node in self.old_nodes:\n # if this old node is unconfirmed skip to next cycle\n if old_node.status == 'u':\n continue\n\n try:\n node = Node.objects.get(pk=old_node.id)\n except Node.DoesNotExist:\n node = Node(id=old_node.id)\n node.data = {}\n\n node.user_id = self.users_dict[old_node.email]['id']\n node.name = old_node.name\n node.slug = old_node.slug\n node.geometry = Point(old_node.lng, old_node.lat)\n node.elev = old_node.alt\n node.description = old_node.description\n node.notes = old_node.notes\n node.added = old_node.added\n node.updated = old_node.updated\n node.data['imported'] = 'true'\n\n intersecting_layers = node.intersecting_layers\n # if more than one intersecting layer\n if len(intersecting_layers) > 1:\n # prompt user\n answer = self.prompt_layer_selection(node, intersecting_layers)\n if isinstance(answer, int):\n node.layer_id = answer\n elif answer == 'default' and self.default_layer is not False:\n node.layer_id = self.default_layer\n else:\n self.message('Node %s discarded' % node.name)\n continue\n # if one intersecting layer select that\n elif 2 > len(intersecting_layers) > 0:\n node.layer = intersecting_layers[0]\n # if no intersecting layers\n else:\n if self.default_layer is False:\n # discard node if no default layer specified\n self.message(\"\"\"Node %s discarded because is not contained\n in any specified layer and no default layer specified\"\"\" % node.name)\n continue\n else:\n node.layer_id = self.default_layer\n\n if old_node.postal_code:\n # additional info\n node.data['postal_code'] = old_node.postal_code\n\n # is it a hotspot?\n if old_node.status in ['h', 'ah']:\n node.data['is_hotspot'] = 'true'\n\n # determine status according to settings\n if self.status_mapping:\n node.status_id = self.get_status(old_node.status)\n\n try:\n node.full_clean()\n node.save(auto_update=False)\n saved_nodes.append(node)\n self.verbose('Saved node %s in layer %s with status %s' % (node.name, node.layer, node.status.name))\n except Exception:\n tb = traceback.format_exc()\n self.message('Could not save node %s, got exception:\\n\\n%s' % (node.name, tb))\n\n self.message('saved %d nodes into local DB' % len(saved_nodes))\n self.saved_nodes = saved_nodes", "response": "import nodes into local DB"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef check_deleted_nodes(self):\n imported_nodes = Node.objects.filter(data__contains=['imported'])\n deleted_nodes = []\n\n for node in imported_nodes:\n if OldNode.objects.filter(pk=node.pk).count() == 0:\n user = node.user\n deleted_nodes.append(node)\n node.delete()\n # delete user if there are no other nodes assigned to her\n if user.node_set.count() == 0:\n user.delete()\n\n if len(deleted_nodes) > 0:\n self.message('deleted %d imported nodes from local DB' % len(deleted_nodes))\n self.deleted_nodes = deleted_nodes", "response": "delete imported nodes that are not present in the old database"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef create_settings(sender, **kwargs):\n created = kwargs['created']\n user = kwargs['instance']\n if created:\n UserWebNotificationSettings.objects.create(user=user)\n UserEmailNotificationSettings.objects.create(user=user)", "response": "create user notification settings on user creation"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef ready(self):\n from nodeshot.core.nodes.base import ExtensibleNodeSerializer\n from .serializers import ProfileRelationSerializer\n\n ExtensibleNodeSerializer.add_relationship(\n name='user',\n serializer=ProfileRelationSerializer,\n queryset=lambda obj, request: obj.user\n )", "response": "Add user info to ExtensibleNodeSerializer"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nparsing geojson and ensure is collection", "response": "def parse(self):\n \"\"\" parse geojson and ensure is collection \"\"\"\n try:\n self.parsed_data = json.loads(self.data)\n except UnicodeError as e:\n self.parsed_data = json.loads(self.data.decode('latin1'))\n except Exception as e:\n raise Exception('Error while converting response from JSON to python. %s' % e)\n\n if self.parsed_data.get('type', '') != 'FeatureCollection':\n raise Exception('GeoJson synchronizer expects a FeatureCollection object at root level')\n\n self.parsed_data = self.parsed_data['features']"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_relationship(_class, name,\n view_name=None, lookup_field=None,\n serializer=None, many=False, queryset=None,\n function=None):\n \"\"\" adds a relationship to serializer\n :param name: relationship name (dictionary key)\n :type name: str\n :param view_name: view name as specified in urls.py\n :type view_name: str\n :param lookup_field: lookup field, usually slug or id/pk\n :type lookup_field: str\n :param serializer: Serializer class to use for relationship\n :type serializer: Serializer\n :param many: indicates if it's a list or a single element, defaults to False\n :type many: bool\n :param queryset: queryset string representation to use for the serializer\n :type queryset: QuerySet\n :param function: function that returns the value to display (dict, list or str)\n :type function: function(obj, request)\n :returns: None\n \"\"\"\n if view_name is not None and lookup_field is not None:\n _class._relationships[name] = {\n 'type': 'link',\n 'view_name': view_name,\n 'lookup_field': lookup_field\n }\n elif serializer is not None and queryset is not None:\n _class._relationships[name] = {\n 'type': 'serializer',\n 'serializer': serializer,\n 'many': many,\n 'queryset': queryset\n }\n elif function is not None:\n _class._relationships[name] = {\n 'type': 'function',\n 'function': function\n }\n else:\n raise ValueError('missing arguments, either pass view_name and lookup_field or serializer and queryset')", "response": "Adds a relationship to the given serializer."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef retrieve_old_notifications(self):\n\n date = ago(days=DELETE_OLD)\n\n return Notification.objects.filter(added__lte=date)", "response": "Retrieve old notifications from the database."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef send_message(message, pipe='public'):\n if pipe not in ['public', 'private']:\n raise ValueError('pipe argument can be only \"public\" or \"private\"')\n else:\n pipe = pipe.upper()\n\n pipe_path = PUBLIC_PIPE if pipe == 'PUBLIC' else PRIVATE_PIPE\n\n # create file if it doesn't exist, append contents\n pipeout = open(pipe_path, 'a')\n\n pipeout.write('%s\\n' % message)\n pipeout.close()", "response": "writes message to the pipe"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating user in the database.", "response": "def create_user(backend, details, response, uid, username, user=None, *args, **kwargs):\n \"\"\"\n Creates user. Depends on get_username pipeline.\n \"\"\"\n if user:\n return {'user': user}\n if not username:\n return None\n email = details.get('email')\n original_email = None\n # email is required\n if not email:\n message = _(\"\"\"your social account needs to have a verified email address in order to proceed.\"\"\")\n raise AuthFailed(backend, message)\n # Avoid hitting field max length\n if email and len(email) > 75:\n original_email = email\n email = ''\n return {\n 'user': UserSocialAuth.create_user(username=username,\n email=email,\n sync_emailaddress=False),\n 'original_email': original_email,\n 'is_new': True\n }"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load_extra_data(backend, details, response, uid, user, social_user=None, *args, **kwargs):\n social_user = social_user or UserSocialAuth.get_social_auth(backend.name, uid)\n # create verified email address\n if kwargs['is_new'] and EMAIL_CONFIRMATION:\n from ..models import EmailAddress\n # check if email exist before creating it\n # we might be associating an exisiting user\n if EmailAddress.objects.filter(email=user.email).count() < 1:\n EmailAddress.objects.create(user=user,\n email=user.email,\n verified=True,\n primary=True)\n\n if social_user:\n extra_data = backend.extra_data(user, uid, response, details)\n if kwargs.get('original_email') and 'email' not in extra_data:\n extra_data['email'] = kwargs.get('original_email')\n # update extra data if anything has changed\n if extra_data and social_user.extra_data != extra_data:\n if social_user.extra_data:\n social_user.extra_data.update(extra_data)\n else:\n social_user.extra_data = extra_data\n social_user.save()\n # fetch additional data from facebook on creation\n if backend.name == 'facebook' and kwargs['is_new']:\n response = json.loads(requests.get('https://graph.facebook.com/%s?access_token=%s' % (extra_data['id'], extra_data['access_token'])).content)\n try:\n user.city, user.country = response.get('hometown').get('name').split(', ')\n except (AttributeError, TypeError):\n pass\n try:\n user.birth_date = datetime.strptime(response.get('birthday'), '%m/%d/%Y').date()\n except (AttributeError, TypeError):\n pass\n user.save()\n return {'social_user': social_user}", "response": "Load extra data from provider and store it on current UserSocialAuth extra_data field."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef clean(self, *args, **kwargs):\n if not self.user and (not self.from_name or not self.from_email):\n raise ValidationError(_('If sender is not specified from_name and from_email must be filled in.'))\n # fill name and email\n if self.user:\n self.from_name = self.user.get_full_name()\n self.from_email = self.user.email", "response": "custom validation for the from_name and from_email attributes"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef send(self):\n if self.content_type.name == 'node':\n to = [self.to.user.email]\n elif self.content_type.name == 'layer':\n to = [self.to.email]\n # layer case is slightly special, mantainers need to be notified as well\n # TODO: consider making the mantainers able to switch off notifications\n for mantainer in self.to.mantainers.all().only('email'):\n to += [mantainer.email]\n else:\n to = [self.to.email]\n\n context = {\n 'sender_name': self.from_name,\n 'sender_email': self.from_email,\n 'message': self.message,\n 'site': settings.SITE_NAME,\n 'object_type': self.content_type.name,\n 'object_name': str(self.to)\n }\n message = render_to_string('mailing/inward_message.txt', context)\n\n email = EmailMessage(\n # subject\n _('Contact request from %(sender_name)s - %(site)s') % context,\n # message\n message,\n # from\n settings.DEFAULT_FROM_EMAIL,\n # to\n to,\n # reply-to header\n headers={'Reply-To': self.from_email}\n )\n\n import socket\n # try sending email\n try:\n email.send()\n self.status = 1\n # if error\n except socket.error as e:\n # log the error\n import logging\n log = logging.getLogger(__name__)\n error_msg = 'nodeshot.community.mailing.models.inward.send(): %s' % e\n log.error(error_msg)\n # set status of the instance as \"error\"\n self.status = -1", "response": "Sends the email to the recipient"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns n objects according to specified ordering", "response": "def slice(self, order_by='pk', n=None):\n \"\"\" return n objects according to specified ordering \"\"\"\n if n is not None and n < 0:\n raise ValueError('slice parameter cannot be negative')\n \n queryset = self.order_by(order_by)\n \n if n is None:\n return queryset[0]\n else:\n return queryset[0:n]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning all items that have an access level lower than or lower than the specified value", "response": "def access_level_up_to(self, access_level):\n \"\"\" returns all items that have an access level equal or lower than the one specified \"\"\"\n # if access_level is number\n if isinstance(access_level, int):\n value = access_level\n # else if is string get the numeric value\n else:\n value = ACCESS_LEVELS.get(access_level)\n # return queryset\n return self.filter(access_level__lte=value)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns all the items that are accessible to the specified user", "response": "def accessible_to(self, user):\n \"\"\"\n returns all the items that are accessible to the specified user\n if user is not authenticated will return public items\n \n :param user: an user instance\n \"\"\"\n if user.is_superuser:\n try:\n queryset = self.get_queryset()\n except AttributeError:\n queryset = self\n elif user.is_authenticated():\n # get user group (higher id)\n group = user.groups.all().order_by('-id')[0]\n queryset = self.filter(access_level__lte=ACCESS_LEVELS.get(group.name))\n else:\n queryset = self.filter(access_level__lte=ACCESS_LEVELS.get('public'))\n return queryset"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nintercepts changes to is_default", "response": "def save(self, *args, **kwargs):\n \"\"\" intercepts changes to is_default \"\"\"\n ignore_default_check = kwargs.pop('ignore_default_check', False)\n # if making this status the default one\n if self.is_default != self._current_is_default and self.is_default is True:\n # uncheck other default statuses first\n for status in self.__class__.objects.filter(is_default=True):\n status.is_default = False\n status.save(ignore_default_check=True)\n\n super(Status, self).save(*args, **kwargs)\n\n # in case there are no default statuses, make this one as the default one\n if self.__class__.objects.filter(is_default=True).count() == 0 and not ignore_default_check:\n self.is_default = True\n self.save()\n # update __current_status\n self._current_is_default = self.is_default"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef ready(self):\n from .models import LayerExternal\n from nodeshot.core.layers.views import LayerNodeListMixin\n\n def get_nodes(self, request, *args, **kwargs):\n try:\n external = self.layer.external\n except LayerExternal.DoesNotExist:\n external = False\n # override view get_nodes method if we have a custom one\n if external and self.layer.is_external and hasattr(external, 'get_nodes'):\n return external.get_nodes(self.__class__.__name__, request.query_params)\n # otherwise return the standard one\n else:\n return (self.list(request, *args, **kwargs)).data\n\n LayerNodeListMixin.get_nodes = get_nodes", "response": "patch LayerNodesList view to support external layers"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a list of recipients for the current node and the current level of the user.", "response": "def get_recipients(self):\n \"\"\"\n Determine recipients depending on selected filtering which can be either:\n * group based\n * layer based\n * user based\n\n Choosing \"group\" and \"layer\" filtering together has the effect of sending the message\n only to users for which the following conditions are both true:\n * have a node assigned to one of the selected layers\n * are part of any of the specified groups (eg: registered, community, trusted)\n\n The user based filtering has instead the effect of translating in an **OR** query. Here's a practical example:\n if selecting \"group\" and \"user\" filtering the message will be sent to all the users for which ANY of the following conditions is true:\n * are part of any of the specified groups (eg: registered, community, trusted)\n * selected users\n \"\"\"\n # user model\n User = get_user_model()\n\n # prepare email list\n emails = []\n\n # the following code is a bit ugly. Considering the titanic amount of work required to build all\n # the cools functionalities that I have in my mind, I can't be bothered to waste time on making it nicer right now.\n # if you have ideas on how to improve it to make it cleaner and less cluttered, please join in\n # this method has unit tests written for it, therefore if you try to change it be sure to check unit tests do not fail after your changes\n # python manage.py test mailing\n\n # send to all case\n if not self.is_filtered:\n # retrieve only email DB column of all active users\n users = User.objects.filter(is_active=True).only('email')\n # loop over users list\n for user in users:\n # add email to the recipient list if not already there\n if user.email not in emails:\n emails += [user.email]\n else:\n # selected users\n if FILTERS.get('users') in self.filters:\n # retrieve selected users\n users = self.users.all().only('email')\n # loop over selected users\n for user in users:\n # add email to the recipient list if not already there\n if user.email not in emails:\n emails += [user.email]\n\n # Q is a django object for \"complex\" filtering queries (not that complex in this case)\n # init empty Q object that will be needed in case of group filtering\n q = Q()\n q2 = Q()\n\n # if group filtering is checked\n if FILTERS.get('groups') in self.filters:\n # loop over each group\n for group in self.groups:\n # if not superusers\n if group != '0':\n # add the group to the Q object\n # this means that the query will look for users of that specific group\n q = q | Q(groups=int(group))\n q2 = q2 | Q(user__groups=int(group))\n else:\n # this must be done manually because superusers is not a group but an attribute of the User model\n q = q | Q(is_superuser=True)\n q2 = q2 | Q(user__is_superuser=True)\n\n # plus users must be active\n q = q & Q(is_active=True)\n\n # if layer filtering is checked\n if FILTERS.get('layers') in self.filters:\n # retrieve non-external layers\n layers = self.layers.all().only('id')\n # init empty q3\n q3 = Q()\n # loop over layers to form q3 object\n for layer in layers:\n q3 = q3 | Q(layer=layer)\n # q2: user group if present\n # q3: layers\n # retrieve nodes\n nodes = Node.objects.filter(q2 & q3)\n # loop over nodes of a layer and get their email\n for node in nodes:\n # add email to the recipient list if not already there\n if node.user.email not in emails:\n emails += [node.user.email]\n # else if group filterins is checked but not layers\n elif FILTERS.get('groups') in self.filters and not FILTERS.get('layers') in self.filters:\n # retrieve only email DB column of all active users\n users = User.objects.filter(q).only('email')\n # loop over users list\n for user in users:\n # add email to the recipient list if not already there\n if user.email not in emails:\n emails += [user.email]\n return emails"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsends the email to the recipients and returns True if the email was successfully sent False otherwise.", "response": "def send(self):\n \"\"\"\n Sends the email to the recipients\n \"\"\"\n # if it has already been sent don't send again\n if self.status is OUTWARD_STATUS.get('sent'):\n return False\n # determine recipients\n recipients = self.get_recipients()\n # init empty list that will contain django's email objects\n emails = []\n\n # prepare text plain if necessary\n if OUTWARD_HTML:\n # store plain text in var\n html_content = self.message\n # set EmailClass to EmailMultiAlternatives\n EmailClass = EmailMultiAlternatives\n else:\n EmailClass = EmailMessage\n\n # default message is plain text\n message = strip_tags(self.message)\n\n # loop over recipients and fill \"emails\" list\n for recipient in recipients:\n msg = EmailClass(\n # subject\n self.subject,\n # message\n message,\n # from\n settings.DEFAULT_FROM_EMAIL,\n # to\n [recipient],\n )\n if OUTWARD_HTML:\n msg.attach_alternative(html_content, \"text/html\")\n # prepare email object\n emails.append(msg)\n\n # try sending email\n try:\n # counter will count how many emails have been sent\n counter = 0\n for email in emails:\n # if step reached\n if counter == OUTWARD_STEP:\n # reset counter\n counter = 0\n # sleep\n time.sleep(OUTWARD_DELAY)\n # send email\n email.send()\n # increase counter\n counter += 1\n # if error (connection refused, SMTP down)\n except socket.error as e:\n # log the error\n from logging import error\n error('nodeshot.core.mailing.models.outward.send(): %s' % e)\n # set status of the instance as \"error\"\n self.status = OUTWARD_STATUS.get('error')\n # change status\n self.status = OUTWARD_STATUS.get('sent')\n # save\n self.save()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef save(self, *args, **kwargs):\n # geometry collection check\n if isinstance(self.geometry, GeometryCollection) and 0 < len(self.geometry) < 2:\n self.geometry = self.geometry[0]\n # if no status specified\n if not self.status and not self.status_id:\n try:\n self.status = Status.objects.filter(is_default=True)[0]\n except IndexError:\n pass\n super(Node, self).save(*args, **kwargs)\n # if status of a node changes\n if (self.status and self._current_status and self.status.id != self._current_status) or\\\n (self.status_id and self._current_status and self.status_id != self._current_status):\n # send django signal\n node_status_changed.send(\n sender=self.__class__,\n instance=self,\n old_status=Status.objects.get(pk=self._current_status),\n new_status=self.status\n )\n # update _current_status\n self._current_status = self.status_id", "response": "Custom save method for the node object."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nextending validation of a Node by adding a function to the _additional_validation list.", "response": "def add_validation_method(cls, method):\n \"\"\"\n Extend validation of Node by adding a function to the _additional_validation list.\n The additional validation function will be called by the clean method\n\n :method function: function to be added to _additional_validation\n \"\"\"\n method_name = method.func_name\n\n # add method name to additional validation method list\n cls._additional_validation.append(method_name)\n\n # add method to this class\n setattr(cls, method_name, method)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef point(self):\n if not self.geometry:\n raise ValueError('geometry attribute must be set before trying to get point property')\n if self.geometry.geom_type == 'Point':\n return self.geometry\n else:\n try:\n # point_on_surface guarantees that the point is within the geometry\n return self.geometry.point_on_surface\n except GEOSException:\n # fall back on centroid which may not be within the geometry\n # for example, a horseshoe shaped polygon\n return self.geometry.centroid", "response": "returns location of node in the geometry"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a Google Elevation profile for the specified resource.", "response": "def elevation_profile(request, format=None):\n \"\"\"\n Proxy to google elevation API but returns GeoJSON\n (unless \"original\" parameter is passed, in which case the original response is returned).\n\n For input parameters read:\n https://developers.google.com/maps/documentation/elevation/\n \"\"\"\n if format is None:\n format = 'json'\n\n path = request.query_params.get('path')\n if not path:\n return Response({'detail': _('missing required path argument')}, status=400)\n\n return Response(elevation(path,\n api_key=ELEVATION_API_KEY,\n sampling=ELEVATION_DEFAULT_SAMPLING))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndetermine user when node is added", "response": "def perform_create(self, serializer):\n \"\"\" determine user when node is added \"\"\"\n if serializer.instance is None:\n serializer.save(user=self.request.user)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_queryset(self):\n # retrieve all nodes which are published and accessible to current user\n # and use joins to retrieve related fields\n queryset = super(NodeList, self).get_queryset().select_related('status', 'user', 'layer')\n # query string params\n search = self.request.query_params.get('search', None)\n layers = self.request.query_params.get('layers', None)\n if search is not None:\n search_query = (\n Q(name__icontains=search) |\n Q(slug__icontains=search) |\n Q(description__icontains=search) |\n Q(address__icontains=search)\n )\n # add instructions for search to queryset\n queryset = queryset.filter(search_query)\n if layers is not None:\n # look for nodes that are assigned to the specified layers\n queryset = queryset.filter(Q(layer__slug__in=layers.split(',')))\n return queryset", "response": "Returns a queryset of all nodes which are published and accessible to current user."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates notifications in a background job to avoid slowing down users", "response": "def create_notifications(users, notification_model, notification_type, related_object):\n \"\"\"\n create notifications in a background job to avoid slowing down users\n \"\"\"\n # shortcuts for readability\n Notification = notification_model\n\n # text\n additional = related_object.__dict__ if related_object else ''\n notification_text = TEXTS[notification_type] % additional\n\n # loop users, notification settings check is done in Notification model\n for user in users:\n\n n = Notification(\n to_user=user,\n type=notification_type,\n text=notification_text\n )\n # attach related object if present\n if related_object:\n n.related_object = related_object\n # create notification and send according to user settings\n n.save()"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a list of all the available resources of this RESTful API.", "response": "def root_endpoint(request, format=None):\n \"\"\"\n List of all the available resources of this RESTful API.\n \"\"\"\n endpoints = []\n # loop over url modules\n for urlmodule in urlpatterns:\n # is it a urlconf module?\n if hasattr(urlmodule, 'urlconf_module'):\n is_urlconf_module = True\n else:\n is_urlconf_module = False\n # if url is really a urlmodule\n if is_urlconf_module:\n # loop over urls of that module\n for url in urlmodule.urlconf_module.urlpatterns:\n # TODO: configurable skip url in settings\n # skip api-docs url\n if url.name in ['django.swagger.resources.view']:\n continue\n # try adding url to list of urls to show\n try:\n endpoints.append({\n 'name': url.name.replace('api_', ''),\n 'url': reverse(url.name, request=request, format=format)\n })\n # urls of object details will fail silently (eg: /nodes/<slug>/)\n except NoReverseMatch:\n pass\n return Response(endpoints)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef save(self, *args, **kwargs):\n if 'node' not in self.shortcuts:\n self.shortcuts['node'] = self.device.node\n\n if 'user' not in self.shortcuts and self.device.node.user:\n self.shortcuts['user'] = self.device.node.user\n\n if 'layer' not in self.shortcuts and 'nodeshot.core.layers' in settings.INSTALLED_APPS:\n self.shortcuts['layer'] = self.device.node.layer\n\n super(Interface, self).save(*args, **kwargs)", "response": "Custom save method does the following :"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nset the ip_addresses field", "response": "def ip_addresses(self, value):\n \"\"\" :param value: a list of ip addresses \"\"\"\n if not isinstance(value, list):\n raise ValueError('ip_addresses value must be a list')\n # in soem cases self.data might be none, so let's instantiate an empty dict\n if self.data is None:\n self.data = {}\n # update field\n self.data['ip_addresses'] = ', '.join(value)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef clean(self, *args, **kwargs):\n # check done only for new nodes!\n if not self.pk:\n node = self.node\n # ensure comments for this node are allowed\n if node.participation_settings.comments_allowed is False:\n raise ValidationError(\"Comments not allowed for this node\")\n # ensure comments for this layer are allowed\n if 'nodeshot.core.layers' in settings.INSTALLED_APPS:\n layer = node.layer\n if layer.participation_settings.comments_allowed is False:\n raise ValidationError(\"Comments not allowed for this layer\")", "response": "Check if comments can be inserted for this node or layer"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef retrieve_layers(self, *args, **options):\n # init empty Q object\n queryset = Q()\n\n # if no layer specified\n if len(args) < 1:\n # cache queryset\n all_layers = Layer.objects.published().external()\n # check if there is any layer to exclude\n if options['exclude']:\n # convert comma separated string in python list, ignore spaces\n exclude_list = options['exclude'].replace(' ', '').split(',')\n # retrieve all layers except the ones specified in exclude list\n return all_layers.exclude(slug__in=exclude_list)\n else:\n # nothing to exclude, retrieve all layers\n self.verbose('no layer specified, will retrieve all layers!')\n return all_layers\n\n # otherwise loop over args and retrieve each specified layer\n for layer_slug in args:\n queryset = queryset | Q(slug=layer_slug)\n # verify existence\n try:\n # retrieve layer\n layer = Layer.objects.get(slug=layer_slug)\n # raise exception if layer is not external\n if not layer.is_external:\n raise CommandError('Layer \"%s\" is not an external layer\\n\\r' % layer_slug)\n # raise exception if layer is not published\n if not layer.is_published:\n raise CommandError('Layer \"%s\" is not published. Why are you trying to work on an unpublished layer?\\n\\r' % layer_slug)\n # raise exception if one of the layer looked for doesn't exist\n except Layer.DoesNotExist:\n raise CommandError('Layer \"%s\" does not exist\\n\\r' % layer_slug)\n\n # return published external layers\n return Layer.objects.published().external().select_related().filter(queryset)", "response": "Retrieve specified layers or all external layers if no layer specified."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nensure users cannot vote the same node multiple times and let users change their votes", "response": "def save(self, *args, **kwargs):\n \"\"\"\n ensure users cannot vote the same node multiple times\n but let users change their votes\n \"\"\"\n if not self.pk:\n old_votes = Vote.objects.filter(user=self.user, node=self.node)\n for old_vote in old_votes:\n old_vote.delete()\n super(Vote, self).save(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nupdating likes and dislikes count", "response": "def update_count(self):\n \"\"\" updates likes and dislikes count \"\"\"\n node_rating_count = self.node.rating_count\n node_rating_count.likes = self.node.vote_set.filter(vote=1).count()\n node_rating_count.dislikes = self.node.vote_set.filter(vote=-1).count()\n node_rating_count.save()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchecks if votes can be inserted for this node or parent layer", "response": "def clean(self, *args, **kwargs):\n \"\"\"\n Check if votes can be inserted for parent node or parent layer\n \"\"\"\n if not self.pk:\n # ensure voting for this node is allowed\n if self.node.participation_settings.voting_allowed is not True:\n raise ValidationError(\"Voting not allowed for this node\")\n\n if 'nodeshot.core.layers' in settings.INSTALLED_APPS:\n layer = self.node.layer\n\n # ensure voting for this layer is allowed\n if layer.participation_settings.voting_allowed is not True:\n raise ValidationError(\"Voting not allowed for this layer\")"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef whois_list(request, format=None):\n results = []\n # layer3 results\n for ip in Ip.objects.select_related().all():\n interface = ip.interface\n user = interface.device.node.user\n device = interface.device\n results.append({\n 'address': str(ip.address),\n 'user': user.username,\n 'name': user.get_full_name(),\n 'device': device.name,\n 'node': device.node.name\n })\n # layer2 results\n for interface in Interface.objects.select_related().all():\n if interface.mac is None:\n continue\n user = interface.device.node.user\n device = interface.device\n results.append({\n 'address': str(interface.mac).replace('-', ':'),\n 'user': user.username,\n 'name': user.get_full_name(),\n 'device': device.name,\n 'node': device.node.name\n })\n return Response(results)", "response": "Retrieve basic whois information related to a layer2 or layer3 network address."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nretrieve basic whois information related to a network address.", "response": "def whois_detail(request, address, format=None):\n \"\"\"\n Retrieve basic whois information related to a layer2 or layer3 network address.\n \"\"\"\n address_obj = None\n try:\n address_obj = IPAddress(address)\n except AddrFormatError:\n try:\n address_obj = EUI(address)\n except AddrFormatError:\n pass\n if not address_obj:\n return Response({'detail': 'invalid address'}, status=400)\n elif isinstance(address_obj, IPAddress):\n try:\n ip = Ip.objects.get(address=address)\n except Ip.DoesNotExist:\n return Response({'detail': 'address not found'}, status=404)\n else:\n interface = ip.interface\n else:\n try:\n interface = Interface.objects.get(mac=address)\n except Interface.DoesNotExist:\n return Response({'detail': 'address not found'}, status=404)\n # prepare response\n user = interface.device.node.user\n device = interface.device\n data = {\n 'address': address,\n 'user': user.username,\n 'name': user.get_full_name(),\n 'device': device.name,\n 'node': device.node.name\n }\n return Response(data)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a queryset of all devices which are published and accessible to current user.", "response": "def get_queryset(self):\n \"\"\"\n Optionally restricts the returned devices\n by filtering against a `search` query parameter in the URL.\n \"\"\"\n # retrieve all devices which are published and accessible to current user\n # and use joins to retrieve related fields\n queryset = super(DeviceList, self).get_queryset()#.select_related('layer', 'status', 'user')\n\n # retrieve value of querystring parameter \"search\"\n search = self.request.query_params.get('search', None)\n\n if search is not None:\n search_query = (\n Q(name__icontains=search) |\n Q(description__icontains=search)\n )\n # add instructions for search to queryset\n queryset = queryset.filter(search_query)\n\n return queryset"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef initial(self, request, *args, **kwargs):\n super(BaseInterfaceList, self).initial(request, *args, **kwargs)\n\n # ensure device exists\n try:\n self.device = Device.objects.accessible_to(request.user)\\\n .get(pk=self.kwargs.get('pk', None))\n except Device.DoesNotExist:\n raise Http404(_('Device not found.'))\n\n # check permissions on device (for interface creation)\n self.check_object_permissions(request, self.device)\n\n # return only interfaces of current device\n self.queryset = self.model.objects.filter(device_id=self.device.id)\\\n .accessible_to(self.request.user)", "response": "Custom initial method for base class base interfaces."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngives a dict of kwargs pull out the group and bridge from the database and return the group and bridge.", "response": "def group_and_bridge(kwargs):\n \"\"\"\n Given kwargs from the view (with view specific keys popped) pull out the\n bridge and fetch group from database.\n \"\"\"\n\n bridge = kwargs.pop(\"bridge\", None)\n\n if bridge:\n try:\n group = bridge.get_group(**kwargs)\n except ObjectDoesNotExist:\n raise Http404\n else:\n group = None\n\n return group, bridge"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsave the object to the database.", "response": "def save(self, *args, **kwargs):\n \"\"\"\n automatically update updated date field\n \"\"\"\n # auto fill updated field with current time unless explicitly disabled\n auto_update = kwargs.get('auto_update', True)\n if auto_update:\n self.updated = now()\n\n # remove eventual auto_update\n if 'auto_update' in kwargs:\n kwargs.pop('auto_update')\n\n super(BaseDate, self).save(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef save(self, *args, **kwargs):\n if self.order == '' or self.order is None:\n try:\n self.order = self.get_auto_order_queryset().order_by(\"-order\")[0].order + 1\n except IndexError:\n self.order = 0\n super(BaseOrdered, self).save()", "response": "if order left blank"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef check_dependencies(dependencies, module):\n if type(dependencies) == str:\n dependencies = [dependencies]\n elif type(dependencies) != list:\n raise TypeError('dependencies argument must be of type list or string')\n\n for dependency in dependencies:\n if dependency not in settings.INSTALLED_APPS:\n raise DependencyError('%s depends on %s, which should be in settings.INSTALLED_APPS' % (module, dependency))", "response": "Checks that the dependencies of a module are listed in settings. INSTALLED_APPS"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef choicify(dictionary):\n # get order of the fields\n ordered_fields = sorted(dictionary, key=dictionary.get)\n choices = []\n # loop over each field\n for field in ordered_fields:\n # build tuple (value, i18n_key)\n row = (dictionary[field], _(field.replace('_', ' ')))\n # append tuple to choices\n choices.append(row)\n # return django sorted choices\n return choices", "response": "Converts a readable python dictionary into a django model / form\n choice structure ordered based on the values of each key in the dictionary"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsearch a value in a dicionary and returns the key of the first occurrence in the dicionary and returns the key of the first occurrence in the dicionary", "response": "def get_key_by_value(dictionary, search_value):\n \"\"\"\n searchs a value in a dicionary and returns the key of the first occurrence\n\n :param dictionary: dictionary to search in\n :param search_value: value to search for\n \"\"\"\n for key, value in dictionary.iteritems():\n if value == search_value:\n return ugettext(key)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nretrieves layer from DB", "response": "def get_layer(self):\n \"\"\" retrieve layer from DB \"\"\"\n if self.layer:\n return\n try:\n self.layer = Layer.objects.get(slug=self.kwargs['slug'])\n except Layer.DoesNotExist:\n raise Http404(_('Layer not found'))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nextending parent class queryset by filtering nodes of the specified layer", "response": "def get_queryset(self):\n \"\"\" extend parent class queryset by filtering nodes of the specified layer \"\"\"\n self.get_layer()\n return super(LayerNodeListMixin, self).get_queryset().filter(layer_id=self.layer.id)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get(self, request, *args, **kwargs):\n self.get_layer()\n # get nodes of layer\n nodes = self.get_nodes(request, *args, **kwargs)\n return Response(nodes)", "response": "Retrieve list of nodes of the specified layer"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsave the IP record in the cache.", "response": "def save(self, *args, **kwargs):\n \"\"\"\n Determines ip protocol version automatically.\n Stores address in interface shortcuts for convenience.\n \"\"\"\n self.protocol = 'ipv%d' % self.address.version\n # save\n super(Ip, self).save(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef allow_relation(self, obj1, obj2, **hints):\n if obj1._meta.app_label != 'oldimporter' and obj2._meta.app_label != 'oldimporter':\n return True\n return None", "response": "Allow relation between objects between nodeshot2 objects only"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nallow migrations of old_nodeshot models to old_nodeshot models", "response": "def allow_migrate(self, db, model):\n \"\"\"\n Make sure the old_nodeshot app only appears in the 'old_nodeshot' database\n \"\"\"\n if db != 'old_nodeshot' or model._meta.app_label != 'oldimporter':\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef clean_email(self):\n if EMAIL_CONFIRMATION:\n from .models import EmailAddress\n condition = EmailAddress.objects.filter(\n email__iexact=self.cleaned_data[\"email\"],\n verified=True\n ).count() == 0\n else:\n condition = User.objects.get(\n email__iexact=self.cleaned_data[\"email\"],\n is_active=True\n ).count() == 0\n\n if condition is True:\n raise forms.ValidationError(\n _(\"Email address not verified for any user account\")\n )\n return self.cleaned_data[\"email\"]", "response": "ensure email is in the database"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting or write metric values", "response": "def metric_details(request, pk, format=None):\n \"\"\"\n Get or write metric values\n \"\"\"\n metric = get_object_or_404(Metric, pk=pk)\n # get\n if request.method == 'GET':\n try:\n results = metric.select(q=request.query_params.get('q', metric.query))\n except InfluxDBClientError as e:\n return Response({'detail': e.content}, status=e.code)\n return Response(list(results.get_points(metric.name)))\n # post\n else:\n if not request.data:\n return Response({'detail': 'expected values in POST data or JSON payload'},\n status=400)\n data = request.data.copy()\n # try converting strings to floats when sending form-data\n if request.content_type != 'application/json':\n for key, value in data.items():\n try:\n data[key] = float(value) if '.' in value else int(value)\n except ValueError:\n pass\n # write\n metric.write(data)\n return Response({'detail': 'ok'})"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nadds current instance to public or private channel.", "response": "def add_client(self, user_id=None):\n \"\"\"\n Adds current instance to public or private channel.\n If user_id is specified it will be added to the private channel,\n If user_id is not specified it will be added to the public one instead.\n \"\"\"\n if user_id is None:\n # generate a random uuid if it's an unauthenticated client\n self.channel = 'public'\n user_id = uuid.uuid1().hex\n else:\n self.channel = 'private'\n \n self.id = user_id\n self.channels[self.channel][self.id] = self\n print 'Client connected to the %s channel.' % self.channel"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nbroadcast a message to all connected clients", "response": "def broadcast(cls, message):\n \"\"\" broadcast message to all connected clients \"\"\"\n clients = cls.get_clients()\n # loop over every client and send message\n for id, client in clients.iteritems():\n client.send_message(message)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsends a message to a specific client. Returns True if successful False otherwise.", "response": "def send_private_message(self, user_id, message):\n \"\"\"\n Send a message to a specific client.\n Returns True if successful, False otherwise\n \"\"\"\n try:\n client = self.channels['private'][str(user_id)]\n except KeyError:\n print '====debug===='\n print self.channels['private']\n print 'client with id %s not found' % user_id\n return False\n \n client.send_message(message)\n print 'message sent to client #%s' % user_id\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_clients(self):\n public = self.channels['public']\n private = self.channels['private']\n return dict(public.items() + private.items())", "response": "return a merge of public and private clients"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_db():\n return client.InfluxDBClient(\n settings.INFLUXDB_HOST,\n settings.INFLUXDB_PORT,\n settings.INFLUXDB_USER,\n settings.INFLUXDB_PASSWORD,\n settings.INFLUXDB_DATABASE,\n )", "response": "Returns an InfluxDBClient instance."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef query(query, params={}, epoch=None,\n expected_response_code=200, database=None):\n \"\"\"Wrapper around ``InfluxDBClient.query()``.\"\"\"\n db = get_db()\n database = database or settings.INFLUXDB_DATABASE\n return db.query(query, params, epoch, expected_response_code, database=database)", "response": "Wrapper around InfluxDBClient. query"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating database if necessary", "response": "def create_database():\n \"\"\" creates database if necessary \"\"\"\n db = get_db()\n response = db.query('SHOW DATABASES')\n items = list(response.get_points('databases'))\n databases = [database['name'] for database in items]\n # if database does not exists, create it\n if settings.INFLUXDB_DATABASE not in databases:\n db.create_database(settings.INFLUXDB_DATABASE)\n print('Created inlfuxdb database {0}'.format(settings.INFLUXDB_DATABASE))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nensuring users cannot rate the same node multiple times and let users change their rating", "response": "def save(self, *args, **kwargs):\n \"\"\"\n ensure users cannot rate the same node multiple times\n but let users change their rating\n \"\"\"\n if not self.pk:\n old_ratings = Rating.objects.filter(user=self.user, node=self.node)\n for old_rating in old_ratings:\n old_rating.delete()\n super(Rating, self).save(*args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nupdates the count of the node and the rating average", "response": "def update_count(self):\n \"\"\" updates rating count and rating average \"\"\"\n node_rating_count = self.node.rating_count\n node_rating_count.rating_count = self.node.rating_set.count()\n node_rating_count.rating_avg = self.node.rating_set.aggregate(rate=Avg('value'))['rate']\n\n # if all ratings are deleted the value will be None!\n if node_rating_count.rating_avg is None:\n # set to 0 otherwise we'll get an exception\n node_rating_count.rating_avg = 0\n\n node_rating_count.save()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef clean(self, *args, **kwargs):\n if not self.pk:\n node = self.node\n layer = Layer.objects.get(pk=node.layer_id)\n if layer.participation_settings.rating_allowed is not True:\n raise ValidationError(\"Rating not allowed for this layer\")\n if node.participation_settings.rating_allowed is not True:\n raise ValidationError(\"Rating not allowed for this node\")", "response": "Check if rating can be inserted for this node or parent layer"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef save(self, *args, **kwargs):\n # the following lines determines if the comment is being created or not\n # in case the comment exists the pk attribute is an int\n created = type(self.pk) is not int\n\n super(UpdateCountsMixin, self).save(*args, **kwargs)\n\n # this operation must be performed after the parent save\n if created:\n self.update_count()", "response": "custom save method to update counts"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef update_topology():\n for topology in Topology.objects.all():\n try:\n topology.update()\n except Exception as e:\n msg = 'Failed to update {}'.format(topology.__repr__())\n logger.exception(msg)\n print('{0}: {1}\\n'\n 'see networking.log for more information\\n'.format(msg, e.__class__))", "response": "Updates all the topology and sends logs to the nodeshot. networking logger"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef save(self, *args, **kwargs):\n self.host = self.host.strip()\n\n if not self.id:\n self.device = self.__create_device()\n\n if self.store is True:\n super(DeviceConnector, self).save(*args, **kwargs)", "response": "Custom save method for device connector."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nclean up the cache", "response": "def clean(self, *args, **kwargs):\n \"\"\" validation \"\"\"\n self._validate_backend()\n self._validate_config()\n self._validate_netengine()\n self._validate_duplicates()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn python netengine backend class importing it if needed", "response": "def backend_class(self):\n \"\"\"\n returns python netengine backend class, importing it if needed\n \"\"\"\n if not self.backend:\n return None\n\n if not self.__backend_class:\n self.__backend_class = self._get_netengine_backend()\n\n return self.__backend_class"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nensuring backend string representation is correct", "response": "def _validate_backend(self):\n \"\"\" ensure backend string representation is correct \"\"\"\n try:\n self.backend_class\n # if we get an import error the specified path is wrong\n except (ImportError, AttributeError) as e:\n raise ValidationError(_('No valid backend found, got the following python exception: \"%s\"') % e)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _validate_config(self):\n # exit if no backend specified\n if not self.backend:\n return\n # exit if no required config keys\n if len(self.REQUIRED_CONFIG_KEYS) < 1:\n return\n\n self.config = self.config or {} # default to empty dict of no config\n required_keys_set = set(self.REQUIRED_CONFIG_KEYS)\n config_keys_set = set(self.config.keys())\n missing_required_keys = required_keys_set - config_keys_set\n unrecognized_keys = config_keys_set - required_keys_set\n\n # if any missing required key raise ValidationError\n if len(missing_required_keys) > 0:\n # converts list in comma separated string\n missing_keys_string = ', '.join(missing_required_keys)\n # django error\n raise ValidationError(_('Missing required config keys: \"%s\"') % missing_keys_string)\n elif len(unrecognized_keys) > 0:\n # converts list in comma separated string\n unrecognized_keys_string = ', '.join(unrecognized_keys)\n # django error\n raise ValidationError(_('Unrecognized config keys: \"%s\"') % unrecognized_keys_string)", "response": "validate that the config is well formed"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncall netengine validate method", "response": "def _validate_netengine(self):\n \"\"\"\n call netengine validate() method\n verifies connection parameters are correct\n \"\"\"\n if self.backend:\n try:\n self.netengine.validate()\n except NetEngineError as e:\n raise ValidationError(e)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchecks that we don t create a device that already exists and that we don t have any duplicate interfaces.", "response": "def _validate_duplicates(self):\n \"\"\"\n Ensure we're not creating a device that already exists\n Runs only when the DeviceConnector object is created, not when is updated\n \"\"\"\n # if connector is being created right now\n if not self.id:\n duplicates = []\n self.netengine_dict = self.netengine.to_dict()\n # loop over interfaces and check mac address\n for interface in self.netengine_dict['interfaces']:\n # avoid checking twice for the same interface (often ifconfig returns duplicates)\n if interface['mac_address'] in duplicates:\n continue\n # check in DB\n if Interface.objects.filter(mac__iexact=interface['mac_address']).count() > 0:\n duplicates.append(interface['mac_address'])\n\n # if we have duplicates raise validation error\n if len(duplicates) > 0:\n mac_address_string = ', '.join(duplicates)\n raise ValidationError(_('interfaces with the following mac addresses already exist: %s') % mac_address_string)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_netengine_arguments(self, required=False):\n # inspect netengine class\n backend_class = self._get_netengine_backend()\n argspec = inspect.getargspec(backend_class.__init__)\n # store args\n args = argspec.args\n # remove known arguments\n for argument_name in ['self', 'host', 'port']:\n args.remove(argument_name)\n\n if required:\n # list of default values\n default_values = list(argspec.defaults)\n # always remove last default value, which is port number\n default_values = default_values[0:-1]\n\n # remove an amount of arguments equals to number of default values, starting from right\n args = args[0:len(args)-len(default_values)]\n\n return args", "response": "returns list of available config params"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_netengine_backend(self):\n # extract backend class name, eg: AirOS or OpenWRT\n backend_class_name = self.backend.split('.')[-1]\n # convert to lowercase to get the path\n backend_path = self.backend.lower()\n # import module by its path\n module = import_module(backend_path)\n # get netengine backend class\n BackendClass = getattr(module, backend_class_name)\n\n return BackendClass", "response": "returns the netengine backend specified in self. backend\n for internal use only\n "} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nbuilds a python dictionary representing the arguments that will be passed to netengine backend for internal use only", "response": "def _build_netengine_arguments(self):\n \"\"\"\n returns a python dictionary representing arguments\n that will be passed to a netengine backend\n for internal use only\n \"\"\"\n arguments = {\n \"host\": self.host\n }\n\n if self.config is not None:\n for key, value in self.config.iteritems():\n arguments[key] = value\n\n if self.port:\n arguments[\"port\"] = self.port\n\n return arguments"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate a new device object based on the current state of the node.", "response": "def __create_device(self):\n \"\"\"\n creates device, internal use only\n \"\"\"\n # retrieve netengine dictionary from memory or from network\n device_dict = getattr(self, 'netengine_dict', self.netengine.to_dict())\n device = Device()\n device.node_id = self.node_id\n device.name = device_dict['name']\n device.type = device_dict['type']\n device.status = DEVICE_STATUS.get('reachable')\n device.os = device_dict['os']\n device.os_version = device_dict['os_version']\n # this is the first time the device is seen by the system because we are just adding it\n device.first_seen = now()\n # and is also the latest\n device.last_seen = now()\n device.full_clean()\n device.save()\n\n # add routing protocols\n for routing_protocol in device_dict['routing_protocols']:\n # retrieve routing protocol from DB\n try:\n rp = RoutingProtocol.objects.filter(\n name__iexact=routing_protocol['name'],\n version__iexact=routing_protocol['version']\n )[0]\n # create if doesn't exist yet\n except IndexError:\n rp = RoutingProtocol(\n name=routing_protocol['name'],\n version=routing_protocol['version']\n )\n rp.full_clean()\n rp.save()\n # add to device\n device.routing_protocols.add(rp)\n\n for interface in device_dict['interfaces']:\n interface_object = False\n vap_object = False\n # create interface depending on type\n if interface['type'] == 'ethernet':\n interface_object = Ethernet(**{\n 'device': device,\n 'name': interface['name'],\n 'mac': interface['mac_address'],\n 'mtu': interface['mtu'],\n 'standard': interface['standard'],\n 'duplex': interface['duplex'],\n 'tx_rate': interface['tx_rate'],\n 'rx_rate': interface['rx_rate']\n })\n elif interface['type'] == 'wireless':\n interface_object = Wireless(**{\n 'device': device,\n 'name': interface['name'],\n 'mac': interface['mac_address'],\n 'mtu': interface['mtu'],\n 'mode': interface['mode'],\n 'standard': interface['standard'],\n 'channel': interface['channel'],\n 'channel_width': interface['channel_width'],\n 'output_power': interface['output_power'],\n 'dbm': interface['dbm'],\n 'noise': interface['noise'],\n 'tx_rate': interface['tx_rate'],\n 'rx_rate': interface['rx_rate']\n })\n\n for vap in interface['vap']:\n vap_object = Vap(\n essid=vap['essid'],\n bssid=vap['bssid'],\n encryption=vap['encryption']\n )\n\n if interface_object:\n interface_object.full_clean()\n interface_object.save()\n\n if vap_object:\n vap_object.interface = interface_object\n vap_object.full_clean()\n vap_object.save()\n\n for ip in interface['ip']:\n ip_object = Ip(**{\n 'interface': interface_object,\n 'address': ip['address'],\n })\n ip_object.full_clean()\n ip_object.save()\n\n if HARDWARE_INSTALLED:\n # try getting device model from db\n try:\n device_model = DeviceModel.objects.filter(name__iexact=device_dict['model'])[0]\n # if it does not exist create it\n except IndexError as e:\n # try getting manufacturer from DB\n try:\n manufacturer = Manufacturer.objects.filter(name__iexact=device_dict['manufacturer'])[0]\n # or create\n except IndexError as e:\n manufacturer = Manufacturer(name=device_dict['manufacturer'])\n manufacturer.full_clean()\n manufacturer.save()\n\n device_model = DeviceModel(\n manufacturer=manufacturer,\n name=device_dict['model']\n )\n device_model.ram = device_dict['RAM_total']\n\n device_model.full_clean()\n device_model.save()\n\n # create relation between device model and device\n rel = DeviceToModelRel(device=device, model=device_model)\n rel.full_clean()\n rel.save()\n\n return device"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncache view response by group.", "response": "def cache_by_group(view_instance, view_method, request, args, kwargs):\n \"\"\"\n Cache view response by media type and user group.\n The cache_key is constructed this way: \"{view_name:path.group.media_type}\"\n EG: \"MenuList:/api/v1/menu/.public.application/json\"\n Possible groups are:\n * public\n * superuser\n * the rest are retrieved from DB (registered, community, trusted are the default ones)\n \"\"\"\n if request.user.is_anonymous():\n group = 'public'\n elif request.user.is_superuser:\n group = 'superuser'\n else:\n try:\n group = request.user.groups.all().order_by('-id').first().name\n except IndexError:\n group = 'public'\n\n key = '%s:%s.%s.%s' % (\n view_instance.__class__.__name__,\n request.META['PATH_INFO'],\n group,\n request.accepted_media_type\n )\n\n return key"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ngenerate a model class based on the provided JSON schema.", "response": "def model_factory(schema, resolver=None, base_class=model.Model, name=None):\n \"\"\"Generate a model class based on the provided JSON Schema\n\n :param schema: dict representing valid JSON schema\n :param name: A name to give the class, if `name` is not in `schema`\n \"\"\"\n schema = copy.deepcopy(schema)\n resolver = resolver\n\n class Model(base_class):\n def __init__(self, *args, **kwargs):\n self.__dict__['schema'] = schema\n self.__dict__['resolver'] = resolver\n base_class.__init__(self, *args, **kwargs)\n\n if resolver is not None:\n Model.resolver = resolver\n\n if name is not None:\n Model.__name__ = name\n elif 'name' in schema:\n Model.__name__ = str(schema['name'])\n return Model"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a jsonpatch object representing the delta", "response": "def patch(self):\n \"\"\"Return a jsonpatch object representing the delta\"\"\"\n original = self.__dict__['__original__']\n return jsonpatch.make_patch(original, dict(self)).to_string()"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef changes(self):\n deprecation_msg = 'Model.changes will be removed in warlock v2'\n warnings.warn(deprecation_msg, DeprecationWarning, stacklevel=2)\n return copy.deepcopy(self.__dict__['changes'])", "response": "A list of changes in this object."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef validate(self, obj):\n try:\n if self.resolver is not None:\n jsonschema.validate(obj, self.schema, resolver=self.resolver)\n else:\n jsonschema.validate(obj, self.schema)\n except jsonschema.ValidationError as exc:\n raise exceptions.ValidationError(str(exc))", "response": "Apply a JSON schema to an object"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef isSupportedContent(cls, fileContent):\n magic = bytearray(fileContent)[:4]\n return magic == p('>I', 0xfeedface) or magic == p('>I', 0xfeedfacf) or magic == p('<I', 0xfeedface) or magic == p('<I', 0xfeedfacf)", "response": "Returns if the file content is valid for this filetype"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the OatHeaderData object from the data", "response": "def _parseOatHeader(self, data):\n \"\"\"Returns the OatHeader\"\"\"\n header = OatHeader.from_buffer(data)\n if header.magic != b'oat\\n':\n raise BinaryError('No valid OAT file')\n\n key_value_store_bytes = (c_ubyte * header.keyValueStoreSize).from_buffer(data, sizeof(OatHeader))\n key_value_store = self.__parseKeyValueStore(key_value_store_bytes)\n\n return OatHeaderData(header=header, keyValueStoreRaw=key_value_store_bytes, keyValueStore=key_value_store)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef __parseKeyValueStore(self, data):\n offset = 0\n key_value_store = {}\n while offset != len(data):\n key = get_str(data, offset)\n offset += len(key)+1\n\n value = get_str(data, offset)\n offset += len(value)+1\n\n key_value_store[key] = value\n\n return key_value_store", "response": "Parses the key value store from the data"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef to_raw_address(addr, section):\n return addr - section.header.VirtualAddress + section.header.PointerToRawData", "response": "Converts the addr from a rva to a pointer to raw data in the file"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the class which holds the suitable classes for the loaded file", "response": "def _getSuitableClasses(self, data, imageDosHeader):\n \"\"\"Returns the class which holds the suitable classes for the loaded file\"\"\"\n classes = None\n machine = IMAGE_FILE_MACHINE[c_ushort.from_buffer(data,imageDosHeader.header.e_lfanew+4).value]\n\n if machine == IMAGE_FILE_MACHINE.I386:\n classes = PE32\n elif machine == IMAGE_FILE_MACHINE.AMD64:\n classes = PE64\n\n return classes"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the ImageDosHeader data", "response": "def _parseImageDosHeader(self, data):\n \"\"\"Returns the ImageDosHeader\"\"\"\n ioh = IMAGE_DOS_HEADER.from_buffer(data)\n if ioh.e_magic != b'MZ':\n raise BinaryError('No valid PE/COFF file')\n\n return ImageDosHeaderData(header=ioh)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _parseImageNtHeaders(self, data, imageDosHeader):\n inth = self._classes.IMAGE_NT_HEADERS.from_buffer(data, imageDosHeader.header.e_lfanew)\n\n if inth.Signature != b'PE':\n raise BinaryError('No valid PE/COFF file')\n\n return ImageNtHeaderData(header=inth)", "response": "Returns the ImageNtHeaders object from the data."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _parseSections(self, data, imageDosHeader, imageNtHeaders, parse_header_only=False):\n sections = []\n\n optional_header_offset = imageDosHeader.header.e_lfanew + 4 + sizeof(IMAGE_FILE_HEADER)\n offset = optional_header_offset + imageNtHeaders.header.FileHeader.SizeOfOptionalHeader # start reading behind the dos- and ntheaders\n\n image_section_header_size = sizeof(IMAGE_SECTION_HEADER)\n\n for sectionNo in range(imageNtHeaders.header.FileHeader.NumberOfSections):\n ishdr = IMAGE_SECTION_HEADER.from_buffer(data, offset)\n\n if parse_header_only:\n raw = None\n bytes_ = bytearray()\n else:\n size = ishdr.SizeOfRawData\n raw = (c_ubyte * size).from_buffer(data, ishdr.PointerToRawData)\n bytes_ = bytearray(raw)\n\n sections.append(SectionData(header=ishdr, name=ishdr.Name.decode('ASCII', errors='ignore'), bytes=bytes_, raw=raw))\n\n offset += image_section_header_size\n\n return sections", "response": "Parses the sections in the memory and returns a list of them"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the section which contains the data of the DataDirectory entry.", "response": "def _getSectionForDataDirectoryEntry(self, data_directory_entry, sections):\n \"\"\"Returns the section which contains the data of DataDirectory\"\"\"\n for section in sections:\n if data_directory_entry.VirtualAddress >= section.header.VirtualAddress and \\\n data_directory_entry.VirtualAddress < section.header.VirtualAddress + section.header.SizeOfRawData :\n\n return section"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _parseDataDirectory(self, data, sections, imageNtHeaders):\n data_directory_data_list = [None for i in range(15)]\n\n # parse DataDirectory[Export]\n export_data_directory = imageNtHeaders.header.OptionalHeader.DataDirectory[ImageDirectoryEntry.EXPORT]\n export_section = self._getSectionForDataDirectoryEntry(export_data_directory, sections)\n export_data_directory_data = self._parseDataDirectoryExport(data, export_data_directory, export_section)\n data_directory_data_list[ImageDirectoryEntry.EXPORT] = export_data_directory_data\n\n # parse DataDirectory[Import]\n import_data_directory = imageNtHeaders.header.OptionalHeader.DataDirectory[ImageDirectoryEntry.IMPORT]\n import_section = self._getSectionForDataDirectoryEntry(import_data_directory, sections)\n import_data_directory_data = self._parseDataDirectoryImport(import_data_directory, import_section)\n data_directory_data_list[ImageDirectoryEntry.IMPORT] = import_data_directory_data\n\n # parse DataDirectory[LOAD_CONFIG]\n loadconfig_data_directory = imageNtHeaders.header.OptionalHeader.DataDirectory[ImageDirectoryEntry.LOAD_CONFIG]\n loadconfig_section = self._getSectionForDataDirectoryEntry(loadconfig_data_directory, sections)\n loadconfig_data = self._parseLoadConfig(loadconfig_data_directory, loadconfig_section)\n data_directory_data_list[ImageDirectoryEntry.LOAD_CONFIG] = loadconfig_data\n\n return data_directory_data_list", "response": "Parses the entries of the DataDirectory and returns a list of the content"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nparses the EmportDataDirectory and returns an instance of ExportDirectoryData", "response": "def _parseDataDirectoryExport(self, data, dataDirectoryEntry, exportSection):\n \"\"\"Parses the EmportDataDirectory and returns an instance of ExportDirectoryData\"\"\"\n if not exportSection:\n return\n functions = []\n export_directory = IMAGE_EXPORT_DIRECTORY.from_buffer(exportSection.raw, to_offset(dataDirectoryEntry.VirtualAddress, exportSection))\n offset = to_offset(export_directory.Name, exportSection)\n\n checkOffset(offset, exportSection)\n name = get_str(exportSection.raw, offset)\n\n offsetOfNames = to_offset(export_directory.AddressOfNames, exportSection)\n offsetOfAddress = to_offset(export_directory.AddressOfFunctions, exportSection)\n offsetOfNameOrdinals = to_offset(export_directory.AddressOfNameOrdinals, exportSection)\n for i in range(export_directory.NumberOfNames):\n name_address = c_uint.from_buffer(exportSection.raw, offsetOfNames).value\n name_offset = to_offset(name_address, exportSection)\n\n checkOffset(name_offset, exportSection)\n func_name = get_str(exportSection.raw, name_offset)\n ordinal = c_ushort.from_buffer(exportSection.raw, offsetOfNameOrdinals).value\n func_addr = c_uint.from_buffer(exportSection.raw, offsetOfAddress).value\n\n offsetOfNames += 4\n offsetOfAddress += 4\n offsetOfNameOrdinals += 2\n functions.append(FunctionData(name=func_name, rva=func_addr, ordinal=ordinal))\n\n return ExportDirectoryData(header=export_directory, name=name, functions=functions)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nparse the ImportDataDirectory and returns a list of ImportDescriptorData s.", "response": "def _parseDataDirectoryImport(self, dataDirectoryEntry, importSection):\n \"\"\"Parses the ImportDataDirectory and returns a list of ImportDescriptorData\"\"\"\n if not importSection:\n return\n\n\n raw_bytes = (c_ubyte * dataDirectoryEntry.Size).from_buffer(importSection.raw, to_offset(dataDirectoryEntry.VirtualAddress, importSection))\n offset = 0\n import_descriptors = []\n while True:\n import_descriptor = IMAGE_IMPORT_DESCRIPTOR.from_buffer(raw_bytes, offset)\n\n\n if import_descriptor.OriginalFirstThunk == 0:\n break\n else:\n nameOffset = to_offset(import_descriptor.Name, importSection)\n\n checkOffset(nameOffset, importSection)\n dllName = get_str(importSection.raw, nameOffset)\n\n import_name_table = self.__parseThunks(import_descriptor.OriginalFirstThunk, importSection)\n import_address_table = self.__parseThunks(import_descriptor.FirstThunk, importSection)\n\n import_descriptors.append(ImportDescriptorData(header=import_descriptor, dllName=dllName, importNameTable=import_name_table, importAddressTable=import_address_table))\n offset += sizeof(IMAGE_IMPORT_DESCRIPTOR)\n return import_descriptors"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nparsing the thunks and returns a list of ThunkData objects", "response": "def __parseThunks(self, thunkRVA, importSection):\n \"\"\"Parses the thunks and returns a list\"\"\"\n offset = to_offset(thunkRVA, importSection)\n table_offset = 0\n thunks = []\n while True:\n thunk = IMAGE_THUNK_DATA.from_buffer(importSection.raw, offset)\n offset += sizeof(IMAGE_THUNK_DATA)\n if thunk.Ordinal == 0:\n break\n thunkData = ThunkData(header=thunk, rva=table_offset+thunkRVA,ordinal=None, importByName=None)\n if to_offset(thunk.AddressOfData, importSection) > 0 and to_offset(thunk.AddressOfData, importSection) < len(self._bytes):\n self.__parseThunkData(thunkData, importSection)\n thunks.append(thunkData)\n table_offset += 4\n return thunks"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef __parseThunkData(self, thunk,importSection):\n offset = to_offset(thunk.header.AddressOfData, importSection)\n if 0xf0000000 & thunk.header.AddressOfData == 0x80000000:\n thunk.ordinal = thunk.header.AddressOfData & 0x0fffffff\n else:\n ibn = IMAGE_IMPORT_BY_NAME.from_buffer(importSection.raw, offset)\n\n checkOffset(offset+2, importSection)\n name = get_str(importSection.raw, offset+2)\n thunk.importByName = ImportByNameData(header=ibn, hint=ibn.Hint, name=name)", "response": "Parses the data of a thunk and sets the data"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a void pointer to the data", "response": "def get_ptr(data, offset=None, ptr_type=ctypes.c_void_p):\n \"\"\"Returns a void pointer to the data\"\"\"\n ptr = ctypes.cast(ctypes.pointer(data), ctypes.c_void_p)\n\n if offset:\n ptr = ctypes.c_void_p(ptr.value + offset)\n\n if ptr_type != ctypes.c_void_p:\n ptr = ctypes.cast(ptr, ptr_type)\n\n return ptr"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a c_ubyte_array filled with the given data of a bytearray or bytes", "response": "def to_ubyte_array(barray):\n \"\"\"Returns a c_ubyte_array filled with the given data of a bytearray or bytes\"\"\"\n bs = (ctypes.c_ubyte * len(barray))()\n pack_into('%ds' % len(barray), bs, 0, barray)\n\n return bs"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreads the file and returns the bytes of the file.", "response": "def _readFile(self, fileName):\n \"\"\"\n Returns the bytes of the file.\n \"\"\"\n with open(fileName, 'rb') as binFile:\n b = binFile.read()\n return to_ubyte_array(b)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the class which holds the suitable classes for the loaded file", "response": "def _getSuitableClasses(self, data):\n \"\"\"Returns the class which holds the suitable classes for the loaded file\"\"\"\n classes = None\n if data[EI.CLASS] == ELFCLASS.BITS_32:\n if data[EI.DATA] == ELFDATA.LSB:\n classes = LSB_32\n elif data[EI.DATA] == ELFDATA.MSB:\n classes = MSB_32\n\n elif data[EI.CLASS] == ELFCLASS.BITS_64:\n if data[EI.DATA] == ELFDATA.LSB:\n classes = LSB_64\n elif data[EI.DATA] == ELFDATA.MSB:\n classes = MSB_64\n\n return classes"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _parseElfHeader(self, data):\n ehdr = self.__classes.EHDR.from_buffer(data)\n return EhdrData(header=ehdr)", "response": "Returns the elf header"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _parseSegments(self, data, elfHeader):\n offset = elfHeader.header.e_phoff\n segments = []\n for i in range(elfHeader.header.e_phnum):\n phdr = self.__classes.PHDR.from_buffer(data, offset)\n segment_bytes = (c_ubyte * phdr.p_filesz).from_buffer(data, phdr.p_offset)\n\n phdrData = PhdrData(header=phdr, raw=segment_bytes, bytes=bytearray(segment_bytes), type=PT[phdr.p_type], vaddr=phdr.p_vaddr, offset=phdr.p_offset)\n segments.append(phdrData)\n\n offset += elfHeader.header.e_phentsize\n\n return segments", "response": "Parse the segments from the data."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _parseSections(self, data, elfHeader):\n offset = elfHeader.header.e_shoff\n shdrs = []\n for i in range(elfHeader.header.e_shnum):\n shdr = self.__classes.SHDR.from_buffer(data, offset)\n section_bytes = None\n ba_section_bytes = None\n if shdr.sh_type != SHT.NOBITS:\n section_bytes = (c_ubyte * shdr.sh_size).from_buffer(data, shdr.sh_offset)\n ba_section_bytes = bytearray(section_bytes)\n shdrs.append(ShdrData(name=None,header=shdr, raw=section_bytes, bytes=ba_section_bytes))\n offset += elfHeader.header.e_shentsize\n\n if elfHeader.header.e_shstrndx != SHN.UNDEF:\n strtab = shdrs[elfHeader.header.e_shstrndx]\n strtab_offset = strtab.header.sh_offset\n\n for section in shdrs:\n section.name = get_str(strtab.raw, section.header.sh_name)\n\n return shdrs", "response": "Returns a list of sections"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _parseSymbols(self, sections):\n for section in sections:\n strtab = sections[section.header.sh_link]\n if section.header.sh_type in (int(SHT.DYNSYM), int(SHT.SYMTAB)):\n section.symbols = self.__parseSymbolEntriesForSection(section, strtab)", "response": "Sets a list of symbols in each DYNSYM and SYMTAB section"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _parseRelocations(self, sections):\n for section in sections:\n if section.header.sh_link != SHN.UNDEF and section.header.sh_type in (SHT.REL, SHT.RELA):\n symbols = sections[section.header.sh_link].symbols\n relocations = self.__parseRelocationEntries(section, symbols)\n section.relocations = relocations", "response": "Parses the relocations and adds those to the section"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nrun the model forward yielding to user code at specified intervals.", "response": "def run_with_snapshots(self, tsnapstart=0., tsnapint=432000.):\n \"\"\"Run the model forward, yielding to user code at specified intervals.\n\n Parameters\n ----------\n\n tsnapstart : int\n The timestep at which to begin yielding.\n tstapint : int\n The interval at which to yield.\n \"\"\"\n\n tsnapints = np.ceil(tsnapint/self.dt)\n\n while(self.t < self.tmax):\n self._step_forward()\n if self.t>=tsnapstart and (self.tc%tsnapints)==0:\n yield self.t\n return"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncalculating the vertical modes of the standard matrix S.", "response": "def vertical_modes(self):\n \"\"\" Calculate standard vertical modes. Simply\n the eigenvectors of the stretching matrix S \"\"\"\n\n evals,evecs = np.linalg.eig(-self.S)\n\n asort = evals.argsort()\n\n # deformation wavenumbers and radii\n self.kdi2 = evals[asort]\n self.radii = np.zeros_like(self.kdi2)\n self.radii[0] = self.g*self.H/np.abs(self.f) # barotropic def. radius\n self.radii[1:] = 1./np.sqrt(self.kdi2[1:])\n\n # eigenstructure\n self.pmodes = evecs[:,asort]\n\n # normalize to have unit L2-norm\n Ai = (self.H / (self.Hi[:,np.newaxis]*(self.pmodes**2)).sum(axis=0))**0.5\n self.pmodes = Ai[np.newaxis,:]*self.pmodes"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef modal_projection(self,p,forward=True):\n\n if forward:\n pt = np.linalg.solve(self.pmodes[np.newaxis,np.newaxis],p.T).T\n else:\n pt = np.einsum(\"ik,k...->i...\",self.pmodes,p)\n\n return pt", "response": "Performs a field p into modal amplitudes pn\n using the basis [ pmodes ]. The inverse\n transform calculates p from pn."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _initialize_background(self):\n\n # background vel.\n if len(np.shape(self.U)) == 0:\n self.U = (self.U * np.ones((self.ny)))\n print(np.shape(self.U))\n self.set_U(self.U)\n\n # the meridional PV gradients in each layer\n self.Qy = (self.beta + np.gradient(np.gradient(self.U, self.dy), self.dy))[np.newaxis,...]\n\n\n # complex versions, multiplied by k, speeds up computations to pre-compute\n self.ikQy = np.expand_dims(self.Qy, axis=2) * 1j * self.k\n\n self.ilQx = 0.", "response": "Set up background state."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _initialize_inversion_matrix(self):\n # The sqg model is diagonal. The inversion is simply qh = -kappa**2 ph\n self.a = np.asarray(self.Nb*np.sqrt(self.wv2i))[np.newaxis, np.newaxis, :, :]", "response": "initialize the inversion matrix"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef set_U(self, U):\n self.Ubg = np.asarray(U)[np.newaxis,...]", "response": "Set background zonal flow"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef step_forward_with_function(self, uv0fun, uv1fun, dt):\n \n dx, dy = self._rk4_integrate(self.x, self.y, uv0fun, uv1fun, dt)\n self.x = self._wrap_x(self.x + dx)\n self.y = self._wrap_y(self.y + dy)", "response": "Advance particles using a function to determine u and v."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nintegrating positions x y using velocity functions uv0fun uv1fun. Returns dx dy the displacements.", "response": "def _rk4_integrate(self, x, y, uv0fun, uv1fun, dt):\n \"\"\"Integrates positions x, y using velocity functions\n uv0fun, uv1fun. Returns dx and dy, the displacements.\"\"\"\n u0, v0 = uv0fun(x, y)\n k1u = dt*u0\n k1v = dt*v0\n x11 = self._wrap_x(x + 0.5*k1u)\n y11 = self._wrap_y(y + 0.5*k1v)\n u11, v11 = uv1fun(x11, y11)\n k2u = dt*u11\n k2v = dt*v11\n x12 = self._wrap_x(x + 0.5*k2u)\n y12 = self._wrap_y(y + 0.5*k2v)\n u12, v12 = uv1fun(x12, y12)\n k3u = dt*u12\n k3v = dt*v12\n x13 = self._wrap_x(x + k3u)\n y13 = self._wrap_y(y + k3v)\n u13, v13 = uv1fun(x13, y13)\n k4u = dt*u13\n k4v = dt*v13\n \n # update\n dx = 6**-1*(k1u + 2*k2u + 2*k3u + k4u)\n dy = 6**-1*(k1v + 2*k2v + 2*k3v + k4v)\n return dx, dy"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _distance(self, x0, y0, x1, y1):\n dx = x1-x0\n dy = y1-y0\n # roll displacements across the borders\n if self.pix:\n dx[ dx > self.Lx/2 ] -= self.Lx\n dx[ dx < -self.Lx/2 ] += self.Lx\n if self.piy:\n dy[ dy > self.Ly/2 ] -= self.Ly\n dy[ dy < -self.Ly/2 ] += self.Ly\n return dx, dy", "response": "Utitlity function to compute distance between points."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef interpolate_gridded_scalar(self, x, y, c, order=1, pad=1, offset=0):\n \n ## no longer necessary because we accept pre-padded arrays\n # assert c.shape == (self.Ny, self.Nx), 'Shape of c needs to be (Ny,Nx)'\n \n # first pad the array to deal with the boundaries\n # (map_coordinates can't seem to deal with this by itself)\n # pad twice so cubic interpolation can be used\n if pad > 0:\n cp = self._pad_field(c, pad=pad)\n else:\n cp = c\n # now the shape is Nx+2, Nx+2\n i = (x - self.xmin)/self.Lx*self.Nx + pad + offset - 0.5\n j = (y - self.ymin)/self.Ly*self.Ny + pad + offset - 0.5\n \n # for some reason this still does not work with high precision near the boundaries\n return scipy.ndimage.map_coordinates(cp, [j,i],\n mode='constant', order=order, cval=np.nan)", "response": "Interpolate gridded scalar C to points x y."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nadvances particles using a gridded velocity field.", "response": "def step_forward_with_gridded_uv(self, U0, V0, U1, V1, dt, order=1): \n \"\"\"Advance particles using a gridded velocity field. Because of the\n Runga-Kutta timestepping, we need two velocity fields at different\n times.\n \n Parameters\n ----------\n U0, V0 : array-like\n Gridded velocity fields at time t - dt.\n U1, V1 : array-like\n Gridded velocity fields at time t.\n dt : number\n Timestep.\n order : int\n Order of interpolation.\n \"\"\"\n # create interpolation functions which return u and v\n \n # pre-pad arrays so it only has to be done once\n # for linear interpolation (default order=1), only one pad is necessary\n pad = order\n [U0p, V0p, U1p, V1p] = [self._pad_field(c, pad=pad) for c in [U0, V0, U1, V1]]\n \n # pad u and v as necessary\n uv0fun = (lambda x, y : \n (self.interpolate_gridded_scalar(x, y, U0p,\n pad=0, order=order, offset=pad),\n self.interpolate_gridded_scalar(x, y, V0p,\n pad=0, order=order, offset=pad)))\n uv1fun = (lambda x, y : \n (self.interpolate_gridded_scalar(x, y, U1p, pad=0,\n order=order, offset=pad),\n self.interpolate_gridded_scalar(x, y, V1p, pad=0,\n order=order, offset=pad)))\n \n self.step_forward_with_function(uv0fun, uv1fun, dt)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncompute the variance of p from Fourier coefficients ph.", "response": "def spec_var(model, ph):\n \"\"\"Compute variance of ``p`` from Fourier coefficients ``ph``.\n\n Parameters\n ----------\n model : pyqg.Model instance\n The model object from which `ph` originates\n ph : complex array\n The field on which to compute the variance\n\n Returns\n -------\n var_dens : float\n The variance of `ph`\n \"\"\"\n\n var_dens = 2. * np.abs(ph)**2 / model.M**2\n # only half of coefs [0] and [nx/2+1] due to symmetry in real fft2\n var_dens[...,0] /= 2\n var_dens[...,-1] /= 2\n return var_dens.sum(axis=(-1,-2))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef spec_sum(ph2):\n\n ph2 = 2.*ph2\n ph2[...,0] = ph2[...,0]/2.\n ph2[...,-1] = ph2[...,-1]/2.\n\n return ph2.sum(axis=(-1,-2))", "response": "Compute total spectral sum of the real spectral quantity ph^2"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef calc_ispec(model, ph):\n\n if model.kk.max()>model.ll.max():\n kmax = model.ll.max()\n else:\n kmax = model.kk.max()\n\n # create radial wavenumber\n dkr = np.sqrt(model.dk**2 + model.dl**2)\n kr = np.arange(dkr/2.,kmax+dkr,dkr)\n phr = np.zeros(kr.size)\n\n for i in range(kr.size):\n fkr = (model.wv>=kr[i]-dkr/2) & (model.wv<=kr[i]+dkr/2)\n dth = pi / (fkr.sum()-1)\n phr[i] = ph[fkr].sum() * kr[i] * dth\n\n return kr, phr", "response": "Compute isotropic spectrum phr of ph of model."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _initialize_stretching_matrix(self):\n\n self.S = np.zeros((self.nz, self.nz))\n\n if (self.nz==2) and (self.rd) and (self.delta):\n\n self.del1 = self.delta/(self.delta+1.)\n self.del2 = (self.delta+1.)**-1\n self.Us = self.Ubg[0]-self.Ubg[1]\n\n self.F1 = self.rd**-2 / (1.+self.delta)\n self.F2 = self.delta*self.F1\n self.S[0,0], self.S[0,1] = -self.F1, self.F1\n self.S[1,0], self.S[1,1] = self.F2, -self.F2\n\n else:\n\n for i in range(self.nz):\n\n if i == 0:\n self.S[i,i] = -self.f2/self.Hi[i]/self.gpi[i]\n self.S[i,i+1] = self.f2/self.Hi[i]/self.gpi[i]\n\n elif i == self.nz-1:\n self.S[i,i] = -self.f2/self.Hi[i]/self.gpi[i-1]\n self.S[i,i-1] = self.f2/self.Hi[i]/self.gpi[i-1]\n\n else:\n self.S[i,i-1] = self.f2/self.Hi[i]/self.gpi[i-1]\n self.S[i,i] = -(self.f2/self.Hi[i]/self.gpi[i] +\n self.f2/self.Hi[i]/self.gpi[i-1])\n self.S[i,i+1] = self.f2/self.Hi[i]/self.gpi[i]", "response": "Initialize the stretching matrix."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _initialize_background(self):\n\n self.H = self.Hi.sum()\n if np.asarray(self.U).ndim == 2:\n self.Ubg = self.U * np.ones((self.ny))\n else:\n self.Ubg = np.expand_dims(self.U,axis=1) * np.ones((self.ny))\n\n\n if not (self.nz==2):\n self.gpi = self.g*(self.rhoi[1:]-self.rhoi[:-1])/self.rhoi[:-1]\n self.f2gpi = (self.f2/self.gpi)[:,np.newaxis,np.newaxis]\n\n assert self.gpi.size == self.nz-1, \"Invalid size of gpi\"\n\n assert np.all(self.gpi>0.), \"Buoyancy jump has negative sign!\"\n\n assert self.Hi.size == self.nz, self.logger.error('size of Hi does not' +\n 'match number of vertical levels nz')\n\n assert self.rhoi.size == self.nz, self.logger.error('size of rhoi does not' +\n 'match number of vertical levels nz')\n\n assert self.Ubg.size == self.nz * self.ny, self.logger.error('size of Ubg does not' +\n 'match number of vertical levels nz')\n\n assert self.Vbg.size == self.nz, self.logger.error('size of Vbg does not' +\n 'match number of vertical levels nz')\n\n else:\n self.f2gpi = np.array(self.rd**-2 *\n (self.Hi[0]*self.Hi[1])/self.H)[np.newaxis]\n\n\n self._initialize_stretching_matrix()\n\n # the meridional PV gradients in each layer\n self.Qy = self.beta - np.dot(self.S, self.Ubg) + np.gradient(np.gradient(self.Ubg, self.dy, axis=1), self.dy, axis=1)\n self.Qx = np.dot(self.S,self.Vbg)\n\n\n # complex versions, multiplied by k, speeds up computations to precompute\n self.ikQy = self.Qy[:,:,np.newaxis]*1j*self.k\n self.ilQx = self.Qx[:,np.newaxis,np.newaxis]*1j*self.l", "response": "Initialize the background state of the current state of the class."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nestimate the eddy turn - over time in days", "response": "def _calc_eddy_time(self):\n \"\"\" estimate the eddy turn-over time in days \"\"\"\n ens = 0.\n for j in range(self.nz):\n ens = .5*self.Hi[j] * self.spec_var(self.wv2*self.ph[j])\n\n return 2.*pi*np.sqrt( self.H / ens.sum() ) / 86400"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nadds diagnostics for the layered model.", "response": "def _initialize_model_diagnostics(self):\n \"\"\" Extra diagnostics for layered model \"\"\"\n\n self.add_diagnostic('entspec',\n description='barotropic enstrophy spectrum',\n function= (lambda self:\n np.abs((self.Hi[:,np.newaxis,np.newaxis]*self.qh).sum(axis=0))**2/self.H) )\n\n self.add_diagnostic('KEspec_modal',\n description='modal KE spectra',\n function= (lambda self:\n self.wv2*(np.abs(self.phn)**2)/self.M**2 ))\n\n self.add_diagnostic('PEspec_modal',\n description='modal PE spectra',\n function= (lambda self:\n self.kdi2[1:,np.newaxis,np.newaxis]*(np.abs(self.phn[1:,:,:])**2)/self.M**2 ))\n\n self.add_diagnostic('APEspec',\n description='available potential energy spectrum',\n function= (lambda self:\n (self.f2gpi*\n np.abs(self.ph[:-1]-self.ph[1:])**2).sum(axis=0)/self.H))\n\n self.add_diagnostic('KEflux',\n description='spectral divergence of flux of kinetic energy',\n function =(lambda self: (self.Hi[:,np.newaxis,np.newaxis]*\n (self.ph.conj()*self.Jpxi).real).sum(axis=0)/self.H))\n\n self.add_diagnostic('APEflux',\n description='spectral divergence of flux of available potential energy',\n function =(lambda self: (self.Hi[:,np.newaxis,np.newaxis]*\n (self.ph.conj()*self.JSp).real).sum(axis=0)/self.H))\n\n self.add_diagnostic('APEgenspec',\n description='the spectrum of the rate of generation of available potential energy',\n function =(lambda self: (self.Hi[:,np.newaxis,np.newaxis]*\n (self.Ubg[:,np.newaxis,np.newaxis]*self.k +\n self.Vbg[:,np.newaxis,np.newaxis]*self.l)*\n (1j*self.ph.conj()*self.Sph).real).sum(axis=0)/self.H))\n\n self.add_diagnostic('ENSflux',\n description='barotropic enstrophy flux',\n function = (lambda self: (-self.Hi[:,np.newaxis,np.newaxis]*\n (self.qh.conj()*self.Jq).real).sum(axis=0)/self.H))\n\n self.add_diagnostic('ENSgenspec',\n description='the spectrum of the rate of generation of barotropic enstrophy',\n function = (lambda self:\n -(self.Hi[:,np.newaxis,np.newaxis]*((self.ikQy -\n self.ilQx)*(self.Sph.conj()*self.ph)).real).sum(axis=0)/self.H))"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ninitializes the background state of the current log - level hierarchy.", "response": "def _initialize_background(self):\n \"\"\"Set up background state (zonal flow and PV gradients).\"\"\"\n\n # Background zonal flow (m/s):\n self.H = self.Hi.sum()\n self.set_U1U2(self.U1, self.U2)\n self.U = self.U1 - self.U2\n\n # the F parameters\n self.F1 = self.rd**-2 / (1.+self.delta)\n self.F2 = self.delta*self.F1\n\n # the meridional PV gradients in each layer\n # need to calculate actual PV gradient\n self.Qy1 = self.beta + self.F1*(self.U1 - self.U2) + np.gradient(np.gradient(self.U1,self.dy),self.dy)\n self.Qy2 = self.beta - self.F2*(self.U1 - self.U2) + np.gradient(np.gradient(self.U2,self.dy),self.dy)\n self.Qy = np.array([self.Qy1, self.Qy2])\n # complex versions, multiplied by k, speeds up computations to precompute\n self.ikQy1 = self.Qy1[:,np.newaxis] * 1j * self.k\n self.ikQy2 = self.Qy2[:,np.newaxis] * 1j * self.k\n\n # vector version\n self.ikQy = np.vstack([self.ikQy1[np.newaxis,...],\n self.ikQy2[np.newaxis,...]])\n self.ilQx = 0.\n\n # layer spacing\n self.del1 = self.delta/(self.delta+1.)\n self.del2 = (self.delta+1.)**-1"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nset the upper and lower layer PV anomalies.", "response": "def set_q1q2(self, q1, q2, check=False):\n \"\"\"Set upper and lower layer PV anomalies.\n\n Parameters\n ----------\n\n q1 : array-like\n Upper layer PV anomaly in spatial coordinates.\n q1 : array-like\n Lower layer PV anomaly in spatial coordinates.\n \"\"\"\n self.set_q(np.vstack([q1[np.newaxis,:,:], q2[np.newaxis,:,:]]))\n #self.q[0] = q1\n #self.q[1] = q2\n\n # initialize spectral PV\n #self.qh = self.fft2(self.q)\n\n # check that it works\n if check:\n np.testing.assert_allclose(self.q1, q1)\n np.testing.assert_allclose(self.q1, self.ifft2(self.qh1))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set_U1U2(self, U1, U2):\n if len(np.shape(U1)) == 0:\n U1 = U1 * np.ones((self.ny)) \n if len(np.shape(U2)) == 0:\n U2 = U2 * np.ones((self.ny)) \n #self.Ubg = np.array([U1,U2])[:,np.newaxis,np.newaxis]\n self.U1 = U1\n self.U2 = U2\n self.Ubg = np.array([U1,U2])", "response": "Set background zonal flow."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _calc_eddy_time(self):\n\n ens = .5*self.Hi[0] * self.spec_var(self.wv2*self.ph1) + \\\n .5*self.Hi[1] * self.spec_var(self.wv2*self.ph2)\n\n return 2.*pi*np.sqrt( self.H / ens ) / 86400", "response": "estimate the eddy turn - over time in days"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _initialize_model_diagnostics(self):\n\n self.add_diagnostic('entspec',\n description='barotropic enstrophy spectrum',\n function= (lambda self:\n np.abs(self.del1*self.qh[0] + self.del2*self.qh[1])**2.)\n )\n\n self.add_diagnostic('APEflux',\n description='spectral flux of available potential energy',\n function= (lambda self:\n self.rd**-2 * self.del1*self.del2 *\n np.real((self.ph[0]-self.ph[1])*np.conj(self.Jptpc)) )\n )\n\n self.add_diagnostic('KEflux',\n description='spectral flux of kinetic energy',\n function= (lambda self:\n np.real(self.del1*self.ph[0]*np.conj(self.Jpxi[0])) +\n np.real(self.del2*self.ph[1]*np.conj(self.Jpxi[1])) )\n )\n \n self.add_diagnostic('APEgenspec',\n description='spectrum of APE generation',\n function= (lambda self: self.U[:,np.newaxis] * self.rd**-2 * self.del1 * self.del2 *\n np.real(1j*self.k*(self.del1*self.ph[0] + self.del2*self.ph[1]) *\n np.conj(self.ph[0] - self.ph[1])) )\n )\n\n self.add_diagnostic('APEgen',\n description='total APE generation',\n function= (lambda self: self.U * self.rd**-2 * self.del1 * self.del2 *\n np.real((1j*self.k*\n (self.del1*self.ph[0] + self.del2*self.ph[1]) *\n np.conj(self.ph[0] - self.ph[1])).sum()\n +(1j*self.k[:,1:-2]*\n (self.del1*self.ph[0,:,1:-2] + self.del2*self.ph[1,:,1:-2]) *\n np.conj(self.ph[0,:,1:-2] - self.ph[1,:,1:-2])).sum()) /\n (self.M**2) )\n )", "response": "Initialize the diagnostics for the two - layer model."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _initialize_inversion_matrix(self):\n # The bt model is diagonal. The inversion is simply qh = -kappa**2 ph\n self.a = -(self.wv2i+self.kd2)[np.newaxis, np.newaxis, :, :]", "response": "initialize the inversion matrix"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nestimate the eddy turn - over time in days", "response": "def _calc_eddy_time(self):\n \"\"\" estimate the eddy turn-over time in days \"\"\"\n ens = .5*self.H * self.spec_var(self.wv2*self.ph)\n return 2.*pi*np.sqrt( self.H / ens ) / year"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncalculating velocity at x and y points due to vortex velocity field.", "response": "def calc_uv(self, x, y, prev=False):\n \"\"\"Calculate velocity at x and y points due to vortex velocity field.\n Assumes x and y are vortex positions and are ordered the same as\n x0 and y0. The ordering is used to neglect to vortex self interaction.\"\"\"\n assert len(x) == self.N\n assert len(y) == self.N\n u = np.zeros(self.N, self.x.dtype)\n v = np.zeros(self.N, self.y.dtype)\n for n in xrange(self.N):\n # don't include self interaction\n if prev:\n x0 = self.xprev[np.r_[:n,n+1:self.N]]\n y0 = self.yprev[np.r_[:n,n+1:self.N]]\n else:\n x0 = self.x[np.r_[:n,n+1:self.N]]\n y0 = self.y[np.r_[:n,n+1:self.N]]\n s0 = self.s[np.r_[:n,n+1:self.N]]\n u0, v0 = self.uv_at_xy(x[n], y[n], x0, y0, s0)\n u[n] = u0.sum()\n v[n] = v0.sum()\n return u, v"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef uv_at_xy(self, x, y, x0, y0, s0):\n dx, dy = self.distance(x0, y0, x, y)\n #print 'dx, dy:', dx, dy\n rr2 = (dx**2 + dy**2)**-1\n u = - s0 * dy * r_twopi * rr2\n v = s0 * dx * r_twopi * rr2\n #print 'u, v', u, v\n return u, v", "response": "Returns two arrays of u v at x y"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreducing the list of arguments into a single list of the same size.", "response": "def reduce(args):\n \"\"\"\n >>> reduce([(2, 4), (4, 9)])\n [(2, 4), (4, 9)]\n\n >>> reduce([(2, 6), (4, 10)])\n [(2, 10)]\n \"\"\"\n if len(args) < 2: return args\n args.sort()\n ret = [args[0]]\n for next_i, (s, e) in enumerate(args, start=1):\n if next_i == len(args):\n ret[-1] = ret[-1][0], max(ret[-1][1], e)\n break\n\n ns, ne = args[next_i]\n if e > ns or ret[-1][1] > ns:\n ret[-1] = ret[-1][0], max(e, ne, ret[-1][1])\n else:\n ret.append((ns, ne))\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef find(self, other):\n iset = self._iset\n l = binsearch_left_start(iset, other[0] - self._maxlen, 0, len(iset))\n r = binsearch_right_end(iset, other[1], 0, len(iset))\n iopts = iset[l:r]\n iiter = (s for s in iopts if s[0] <= other[1] and s[1] >= other[0])\n for o in iiter: yield o", "response": "Return an interable of elements that overlap other in the tree."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nloading the dictionary from a specific JSON file.", "response": "def loaddict(filename=DICTIONARY):\n \"\"\"\n Load the dictionary from a specific JSON file.\n \"\"\"\n global zhcdicts\n if zhcdicts:\n return\n if filename == _DEFAULT_DICT:\n zhcdicts = json.loads(get_module_res(filename).read().decode('utf-8'))\n else:\n with open(filename, 'rb') as f:\n zhcdicts = json.loads(f.read().decode('utf-8'))\n zhcdicts['SIMPONLY'] = frozenset(zhcdicts['SIMPONLY'])\n zhcdicts['TRADONLY'] = frozenset(zhcdicts['TRADONLY'])"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngenerates or get convertion dict cache for certain locale.", "response": "def getdict(locale):\n \"\"\"\n Generate or get convertion dict cache for certain locale.\n Dictionaries are loaded on demand.\n \"\"\"\n global zhcdicts, dict_zhcn, dict_zhsg, dict_zhtw, dict_zhhk, pfsdict\n if zhcdicts is None:\n loaddict(DICTIONARY)\n if locale == 'zh-cn':\n if dict_zhcn:\n got = dict_zhcn\n else:\n dict_zhcn = zhcdicts['zh2Hans'].copy()\n dict_zhcn.update(zhcdicts['zh2CN'])\n got = dict_zhcn\n elif locale == 'zh-tw':\n if dict_zhtw:\n got = dict_zhtw\n else:\n dict_zhtw = zhcdicts['zh2Hant'].copy()\n dict_zhtw.update(zhcdicts['zh2TW'])\n got = dict_zhtw\n elif locale == 'zh-hk' or locale == 'zh-mo':\n if dict_zhhk:\n got = dict_zhhk\n else:\n dict_zhhk = zhcdicts['zh2Hant'].copy()\n dict_zhhk.update(zhcdicts['zh2HK'])\n got = dict_zhhk\n elif locale == 'zh-sg' or locale == 'zh-my':\n if dict_zhsg:\n got = dict_zhsg\n else:\n dict_zhsg = zhcdicts['zh2Hans'].copy()\n dict_zhsg.update(zhcdicts['zh2SG'])\n got = dict_zhsg\n elif locale == 'zh-hans':\n got = zhcdicts['zh2Hans']\n elif locale == 'zh-hant':\n got = zhcdicts['zh2Hant']\n else:\n got = {}\n if locale not in pfsdict:\n pfsdict[locale] = getpfset(got)\n return got"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef issimp(s, full=False):\n if zhcdicts is None:\n loaddict(DICTIONARY)\n simp, trad = 0, 0\n if full:\n for ch in s:\n if ch in zhcdicts['SIMPONLY']:\n simp += 1\n elif ch in zhcdicts['TRADONLY']:\n trad += 1\n if simp > trad:\n return True\n elif simp < trad:\n return False\n else:\n return None\n else:\n for ch in s:\n if ch in zhcdicts['SIMPONLY']:\n return True\n elif ch in zhcdicts['TRADONLY']:\n return False\n return None", "response": "Detect text is whether Simplified Chinese or Traditional Chinese."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef convtable2dict(convtable, locale, update=None):\n rdict = update.copy() if update else {}\n for r in convtable:\n if ':uni' in r:\n if locale in r:\n rdict[r[':uni']] = r[locale]\n elif locale[:-1] == 'zh-han':\n if locale in r:\n for word in r.values():\n rdict[word] = r[locale]\n else:\n v = fallback(locale, r)\n for word in r.values():\n rdict[word] = v\n return rdict", "response": "Convert a list of conversion dict to a dict for a certain locale."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef tokenize(s, locale, update=None):\n zhdict = getdict(locale)\n pfset = pfsdict[locale]\n if update:\n zhdict = zhdict.copy()\n zhdict.update(update)\n newset = set()\n for word in update:\n for ch in range(len(word)):\n newset.add(word[:ch+1])\n pfset = pfset | newset\n ch = []\n N = len(s)\n pos = 0\n while pos < N:\n i = pos\n frag = s[pos]\n maxword = None\n maxpos = 0\n while i < N and frag in pfset:\n if frag in zhdict:\n maxword = frag\n maxpos = i\n i += 1\n frag = s[pos:i+1]\n if maxword is None:\n maxword = s[pos]\n pos += 1\n else:\n pos = maxpos + 1\n ch.append(maxword)\n return ch", "response": "Tokenize s according to corresponding locale dictionary."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert a MediaWiki string to a human - readable format.", "response": "def convert_for_mw(s, locale, update=None):\n \"\"\"\n Recognizes MediaWiki's human conversion format.\n Use locale='zh' for no conversion.\n\n Reference: (all tests passed)\n https://zh.wikipedia.org/wiki/Help:\u9ad8\u7ea7\u5b57\u8bcd\u8f6c\u6362\u8bed\u6cd5\n https://www.mediawiki.org/wiki/Writing_systems/Syntax\n\n >>> print(convert_for_mw('\u5728\u73b0\u4ee3\uff0c\u673a\u68b0\u8ba1\u7b97-{}-\u673a\u7684\u5e94\u7528\u5df2\u7ecf\u5b8c\u5168\u88ab\u7535\u5b50\u8ba1\u7b97-{}-\u673a\u6240\u53d6\u4ee3', 'zh-hk'))\n \u5728\u73fe\u4ee3\uff0c\u6a5f\u68b0\u8a08\u7b97\u6a5f\u7684\u61c9\u7528\u5df2\u7d93\u5b8c\u5168\u88ab\u96fb\u5b50\u8a08\u7b97\u6a5f\u6240\u53d6\u4ee3\n >>> print(convert_for_mw('-{zh-hant:\u8cc7\u8a0a\u5de5\u7a0b;zh-hans:\u8ba1\u7b97\u673a\u5de5\u7a0b\u5b66;}-\u662f\u7535\u5b50\u5de5\u7a0b\u7684\u4e00\u4e2a\u5206\u652f\uff0c\u4e3b\u8981\u7814\u7a76\u8ba1\u7b97\u673a\u8f6f\u786c\u4ef6\u548c\u4e8c\u8005\u95f4\u7684\u5f7c\u6b64\u8054\u7cfb\u3002', 'zh-tw'))\n \u8cc7\u8a0a\u5de5\u7a0b\u662f\u96fb\u5b50\u5de5\u7a0b\u7684\u4e00\u500b\u5206\u652f\uff0c\u4e3b\u8981\u7814\u7a76\u8a08\u7b97\u6a5f\u8edf\u786c\u9ad4\u548c\u4e8c\u8005\u9593\u7684\u5f7c\u6b64\u806f\u7e6b\u3002\n >>> print(convert_for_mw('\u5f35\u570b\u69ae\u66fe\u5728\u82f1\u570b-{zh:\u5229\u5179;zh-hans:\u5229\u5179;zh-hk:\u5217\u65af;zh-tw:\u91cc\u8332}-\u5927\u5b66\u5b78\u7fd2\u3002', 'zh-hant'))\n \u5f35\u570b\u69ae\u66fe\u5728\u82f1\u570b\u91cc\u8332\u5927\u5b78\u5b78\u7fd2\u3002\n >>> print(convert_for_mw('\u5f35\u570b\u69ae\u66fe\u5728\u82f1\u570b-{zh:\u5229\u5179;zh-hans:\u5229\u5179;zh-hk:\u5217\u65af;zh-tw:\u91cc\u8332}-\u5927\u5b66\u5b78\u7fd2\u3002', 'zh-sg'))\n \u5f20\u56fd\u8363\u66fe\u5728\u82f1\u56fd\u5229\u5179\u5927\u5b66\u5b66\u4e60\u3002\n >>> convert_for_mw('-{zh-hant:;\\\\nzh-cn:}-', 'zh-tw') == ''\n True\n >>> print(convert_for_mw('\u6beb\u7c73(\u6beb\u516c\u5206)\uff0c\u7b26\u865fmm\uff0c\u662f\u9577\u5ea6\u55ae\u4f4d\u548c\u964d\u96e8\u91cf\u55ae\u4f4d\uff0c-{zh-hans:\u53f0\u6e7e\u4f5c-{\u516c\u91d0}-\u6216-{\u516c\u5398}-;zh-hant:\u6e2f\u6fb3\u548c\u5927\u9678\u7a31\u70ba-{\u6beb\u7c73}-\uff08\u53f0\u7063\u4ea6\u6709\u4f7f\u7528\uff0c\u4f46\u8f03\u5e38\u4f7f\u7528\u540d\u7a31\u70ba\u6beb\u516c\u5206\uff09;zh-mo:\u53f0\u7063\u4f5c-{\u516c\u91d0}-\u6216-{\u516c\u5398}-;zh-hk:\u53f0\u7063\u4f5c-{\u516c\u91d0}-\u6216-{\u516c\u5398}-;}-\u3002', 'zh-tw'))\n \u6beb\u7c73(\u6beb\u516c\u5206)\uff0c\u7b26\u865fmm\uff0c\u662f\u9577\u5ea6\u55ae\u4f4d\u548c\u964d\u96e8\u91cf\u55ae\u4f4d\uff0c\u6e2f\u6fb3\u548c\u5927\u9678\u7a31\u70ba\u6beb\u7c73\uff08\u53f0\u7063\u4ea6\u6709\u4f7f\u7528\uff0c\u4f46\u8f03\u5e38\u4f7f\u7528\u540d\u7a31\u70ba\u6beb\u516c\u5206\uff09\u3002\n >>> print(convert_for_mw('\u6beb\u7c73(\u6beb\u516c\u5206)\uff0c\u7b26\u865fmm\uff0c\u662f\u9577\u5ea6\u55ae\u4f4d\u548c\u964d\u96e8\u91cf\u55ae\u4f4d\uff0c-{zh-hans:\u53f0\u6e7e\u4f5c-{\u516c\u91d0}-\u6216-{\u516c\u5398}-;zh-hant:\u6e2f\u6fb3\u548c\u5927\u9678\u7a31\u70ba-{\u6beb\u7c73}-\uff08\u53f0\u7063\u4ea6\u6709\u4f7f\u7528\uff0c\u4f46\u8f03\u5e38\u4f7f\u7528\u540d\u7a31\u70ba\u6beb\u516c\u5206\uff09;zh-mo:\u53f0\u7063\u4f5c-{\u516c\u91d0}-\u6216-{\u516c\u5398}-;zh-hk:\u53f0\u7063\u4f5c-{\u516c\u91d0}-\u6216-{\u516c\u5398}-;}-\u3002', 'zh-cn'))\n \u6beb\u7c73(\u6beb\u516c\u5206)\uff0c\u7b26\u53f7mm\uff0c\u662f\u957f\u5ea6\u5355\u4f4d\u548c\u964d\u96e8\u91cf\u5355\u4f4d\uff0c\u53f0\u6e7e\u4f5c\u516c\u91d0\u6216\u516c\u5398\u3002\n >>> print(convert_for_mw('\u6beb\u7c73(\u6beb\u516c\u5206)\uff0c\u7b26\u865fmm\uff0c\u662f\u9577\u5ea6\u55ae\u4f4d\u548c\u964d\u96e8\u91cf\u55ae\u4f4d\uff0c-{zh-hans:\u53f0\u6e7e\u4f5c-{\u516c\u91d0}-\u6216-{\u516c\u5398}-;zh-hant:\u6e2f\u6fb3\u548c\u5927\u9678\u7a31\u70ba-{\u6beb\u7c73}-\uff08\u53f0\u7063\u4ea6\u6709\u4f7f\u7528\uff0c\u4f46\u8f03\u5e38\u4f7f\u7528\u540d\u7a31\u70ba\u6beb\u516c\u5206\uff09;zh-mo:\u53f0\u7063\u4f5c-{\u516c\u91d0}-\u6216-{\u516c\u5398}-;zh-hk:\u53f0\u7063\u4f5c-{\u516c\u91d0}-\u6216-{\u516c\u5398', 'zh-hk')) # unbalanced test\n \u6beb\u7c73(\u6beb\u516c\u5206)\uff0c\u7b26\u865fmm\uff0c\u662f\u9577\u5ea6\u55ae\u4f4d\u548c\u964d\u96e8\u91cf\u55ae\u4f4d\uff0c\u53f0\u7063\u4f5c\u516c\u91d0\u6216\u516c\u5398\n >>> print(convert_for_mw('\u62a5\u5934\u7684\u201c-{\u53c3\u6537\u6d88\u606f}-\u201d\u56db\u5b57\u6458\u81ea\u9c81\u8fc5\u7b14\u8ff9-{zh-hans:\uff0c\u201c-{\u53c3}-\u201d\u662f\u201c-{\u53c2}-\u201d\u7684\u7e41\u4f53\u5b57\uff0c\u8bfb\u97f3c\u0101n\uff0c\u4e0e\u7b80\u4f53\u7684\u201c-{\u53c2}-\u201d\u5b57\u76f8\u540c\uff1b;zh-hant:\uff0c;}-\u201c-{\u6537}-\u201d\u662f\u201c\u8003\u201d\u7684\u5f02\u4f53\u5b57\uff0c\u8bfb\u97f3k\u01ceo\uff0c\u4e0e\u201c\u8003\u201d\u5b57\u76f8\u540c\u3002', 'zh-tw'))\n \u5831\u982d\u7684\u300c\u53c3\u6537\u6d88\u606f\u300d\u56db\u5b57\u6458\u81ea\u9b6f\u8fc5\u7b46\u8de1\uff0c\u300c\u6537\u300d\u662f\u300c\u8003\u300d\u7684\u7570\u9ad4\u5b57\uff0c\u8b80\u97f3k\u01ceo\uff0c\u8207\u300c\u8003\u300d\u5b57\u76f8\u540c\u3002\n >>> print(convert_for_mw('\u62a5\u5934\u7684\u201c-{\u53c3\u6537\u6d88\u606f}-\u201d\u56db\u5b57\u6458\u81ea\u9c81\u8fc5\u7b14\u8ff9-{zh-hans:\uff0c\u201c-{\u53c3}-\u201d\u662f\u201c-{\u53c2}-\u201d\u7684\u7e41\u4f53\u5b57\uff0c\u8bfb\u97f3c\u0101n\uff0c\u4e0e\u7b80\u4f53\u7684\u201c-{\u53c2}-\u201d\u5b57\u76f8\u540c\uff1b;zh-hant:\uff0c;}-\u201c-{\u6537}-\u201d\u662f\u201c\u8003\u201d\u7684\u5f02\u4f53\u5b57\uff0c\u8bfb\u97f3k\u01ceo\uff0c\u4e0e\u201c\u8003\u201d\u5b57\u76f8\u540c\u3002', 'zh-cn'))\n \u62a5\u5934\u7684\u201c\u53c3\u6537\u6d88\u606f\u201d\u56db\u5b57\u6458\u81ea\u9c81\u8fc5\u7b14\u8ff9\uff0c\u201c\u53c3\u201d\u662f\u201c\u53c2\u201d\u7684\u7e41\u4f53\u5b57\uff0c\u8bfb\u97f3c\u0101n\uff0c\u4e0e\u7b80\u4f53\u7684\u201c\u53c2\u201d\u5b57\u76f8\u540c\uff1b\u201c\u6537\u201d\u662f\u201c\u8003\u201d\u7684\u5f02\u4f53\u5b57\uff0c\u8bfb\u97f3k\u01ceo\uff0c\u4e0e\u201c\u8003\u201d\u5b57\u76f8\u540c\u3002\n >>> print(convert_for_mw('{{Col-break}}-->', 'zh-hant'))\n {{Col-break}}-->\n \"\"\"\n ch = []\n rules = []\n ruledict = update.copy() if update else {}\n nested = 0\n block = ''\n for frag in RE_langconv.split(s):\n if frag == '-{':\n nested += 1\n block += frag\n elif frag == '}-':\n if not nested:\n # bogus }-\n ch.append(frag)\n continue\n block += frag\n nested -= 1\n if nested:\n continue\n newrules = []\n delim = RE_splitflag.split(block[2:-2].strip(' \\t\\n\\r\\f\\v;'))\n if len(delim) == 1:\n flag = None\n mapping = RE_splitmap.split(delim[0])\n else:\n flag = RE_splitmap.split(delim[0].strip(' \\t\\n\\r\\f\\v;'))\n mapping = RE_splitmap.split(delim[1])\n rule = {}\n for m in mapping:\n uni = RE_splituni.split(m)\n if len(uni) == 1:\n pair = RE_splitpair.split(uni[0])\n else:\n if rule:\n newrules.append(rule)\n rule = {':uni': uni[0]}\n else:\n rule[':uni'] = uni[0]\n pair = RE_splitpair.split(uni[1])\n if len(pair) == 1:\n rule['zh'] = convert_for_mw(pair[0], 'zh', ruledict)\n else:\n rule[pair[0]] = convert_for_mw(pair[1], pair[0], ruledict)\n newrules.append(rule)\n if not flag:\n ch.append(fallback(locale, newrules[0]))\n elif any(ch in flag for ch in 'ATRD-HN'):\n for f in flag:\n # A: add rule for convert code (all text convert)\n # H: Insert a conversion rule without output\n if f in ('A', 'H'):\n for r in newrules:\n if not r in rules:\n rules.append(r)\n if f == 'A':\n if ':uni' in r:\n if locale in r:\n ch.append(r[locale])\n else:\n ch.append(convert(r[':uni'], locale))\n else:\n ch.append(fallback(locale, newrules[0]))\n # -: remove convert\n elif f == '-':\n for r in newrules:\n try:\n rules.remove(r)\n except ValueError:\n pass\n # D: convert description (useless)\n #elif f == 'D':\n #ch.append('; '.join(': '.join(x) for x in newrules[0].items()))\n # T: title convert (useless)\n # R: raw content (implied above)\n # N: current variant name (useless)\n #elif f == 'N':\n #ch.append(locale)\n ruledict = convtable2dict(rules, locale, update)\n else:\n fblimit = frozenset(flag) & frozenset(Locales[locale])\n limitedruledict = update.copy() if update else {}\n for r in rules:\n if ':uni' in r:\n if locale in r:\n limitedruledict[r[':uni']] = r[locale]\n else:\n v = None\n for l in Locales[locale]:\n if l in r and l in fblimit:\n v = r[l]\n break\n for word in r.values():\n limitedruledict[word] = v if v else convert(word, locale)\n ch.append(convert(delim[1], locale, limitedruledict))\n block = ''\n elif nested:\n block += frag\n else:\n ch.append(convert(frag, locale, ruledict))\n if nested:\n # unbalanced\n ch.append(convert_for_mw(block + '}-'*nested, locale, ruledict))\n return ''.join(ch)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef bucket_lister(manager, bucket_name, prefix=None, marker=None, limit=None):\n eof = False\n while not eof:\n ret, eof, info = manager.list(bucket_name, prefix=prefix, limit=limit,\n marker=marker)\n if ret is None:\n raise QiniuError(info)\n if not eof:\n marker = ret['marker']\n\n for item in ret['items']:\n yield item", "response": "A generator function for listing keys in a bucket."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_qiniu_config(name, default=None):\n config = os.environ.get(name, getattr(settings, name, default))\n if config is not None:\n if isinstance(config, six.string_types):\n return config.strip()\n else:\n return config\n else:\n raise ImproperlyConfigured(\n \"Can't find config for '%s' either in environment\"\n \"variable or in setting.py\" % name)", "response": "Get a Qiniu configuration variable from environment variable\n or django setting. py"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nperforming a register with type m. login. application_service AttributeNames", "response": "def register(self, username=\"\"):\n \"\"\"Performs /register with type: m.login.application_service\n\n Args:\n username(str): Username to register.\n \"\"\"\n if not username:\n username = utils.mxid2localpart(self.identity)\n content = {\n \"type\": \"m.login.application_service\",\n \"username\": username,\n }\n return self._send(\"POST\", \"/register\", content,\n api_path=MATRIX_V2_API_PATH)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef load_from_file(filename):\n if os.path.isdir(filename):\n logger.error(\"Err: File '%s' is a directory\", filename)\n return None\n if not os.path.isfile(filename):\n logger.error(\"Err: File '%s' does not exist\", filename)\n return None\n\n try:\n with open(filename, 'r') as sourcefile:\n songs = [line.strip() for line in sourcefile]\n except IOError as error:\n logger.exception(error)\n return None\n songs = set(Song.from_filename(song) for song in songs)\n return songs.difference({None})", "response": "Load a list of filenames from an external text file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nparsing command line arguments and return a dictionary of configuration variables.", "response": "def parse_argv():\n \"\"\"\n Parse command line arguments. Settings will be stored in the global\n variables declared above.\n \"\"\"\n parser = argparse.ArgumentParser(description='Find lyrics for a set of mp3'\n ' files and embed them as metadata')\n parser.add_argument('-j', '--jobs', help='Number of parallel processes',\n type=int, metavar='N', default=1)\n parser.add_argument('-o', '--overwrite', help='Overwrite lyrics of songs'\n ' that already have them', action='store_true')\n parser.add_argument('-s', '--stats', help='Print a series of statistics at'\n ' the end of the execution', action='store_true')\n parser.add_argument('-v', '--verbose', help='Set verbosity level (pass it'\n ' up to three times)', action='count')\n parser.add_argument('-d', '--debug', help='Enable debug output',\n action='store_true')\n group = parser.add_mutually_exclusive_group()\n group.add_argument('-r', '--recursive', help='Recursively search for'\n ' mp3 files', metavar='path', nargs='?', const='.')\n group.add_argument('--from-file', help='Read a list of files from a text'\n ' file', type=str)\n parser.add_argument('songs', help='The files/songs to search lyrics for',\n nargs='*')\n\n args = parser.parse_args()\n\n CONFIG['overwrite'] = args.overwrite\n CONFIG['print_stats'] = args.stats\n\n if args.verbose is None or args.verbose == 0:\n logger.setLevel(logging.CRITICAL)\n elif args.verbose == 1:\n logger.setLevel(logging.INFO)\n else:\n logger.setLevel(logging.DEBUG)\n\n if args.jobs <= 0:\n msg = 'Argument -j/--jobs should have a value greater than zero'\n parser.error(msg)\n else:\n CONFIG['jobcount'] = args.jobs\n\n songs = set()\n if args.from_file:\n songs = load_from_file(args.from_file)\n if not songs:\n raise ValueError('No file names found in file')\n elif args.recursive:\n mp3files = glob.iglob(args.recursive + '/**/*.mp3', recursive=True)\n songs = set(Song.from_filename(f) for f in mp3files)\n elif args.songs:\n if os.path.exists(args.songs[0]):\n parser = Song.from_filename\n else:\n parser = Song.from_string\n songs.update(map(parser, args.songs))\n else:\n songs.add(get_current_song())\n\n # Just in case some song constructors failed, remove all the Nones\n return songs.difference({None})"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndecoding a base64 - encoded UUID object.", "response": "def decode(slug):\n \"\"\"\n Returns the uuid.UUID object represented by the given v4 or \"nice\" slug\n \"\"\"\n if sys.version_info.major != 2 and isinstance(slug, bytes):\n slug = slug.decode('ascii')\n slug = slug + '==' # base64 padding\n return uuid.UUID(bytes=base64.urlsafe_b64decode(slug))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a randomly generated uuid v4 compliant slug that conforms to a set of nice properties.", "response": "def nice():\n \"\"\"\n Returns a randomly generated uuid v4 compliant slug which conforms to a set\n of \"nice\" properties, at the cost of some entropy. Currently this means one\n extra fixed bit (the first bit of the uuid is set to 0) which guarantees the\n slug will begin with [A-Za-f]. For example such slugs don't require special\n handling when used as command line parameters (whereas non-nice slugs may\n start with `-` which can confuse command line tools).\n\n Potentially other \"nice\" properties may be added in future to further\n restrict the range of potential uuids that may be generated.\n \"\"\"\n rawBytes = bytearray(uuid.uuid4().bytes)\n rawBytes[0] = rawBytes[0] & 0x7f # Ensure slug starts with [A-Za-f]\n return _convert_bytes_to_slug(rawBytes)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfiltering mutations that are in the given normal.", "response": "def filter_against_normal(self, normal_mutations, maf_min=0.2,\n maf_count_threshold=20, count_min=1):\n \"\"\"Filters mutations that are in the given normal\"\"\"\n assert(normal_mutations.chrom == self.chrom)\n assert(normal_mutations.pos == self.pos)\n assert(normal_mutations.ref == self.ref)\n\n def passes_normal_criteria(mut):\n return (mut.count >= maf_count_threshold and mut.maf > maf_min) or \\\n (mut.count < maf_count_threshold and mut.count > count_min)\n\n nms = normal_mutations\n muts = MutationsAtSinglePosition(self.chrom, self.pos, self.cov, self.ref)\n\n for snv in self.snvs:\n if not (snv in nms.snvs and passes_normal_criteria(nms.snvs[snv])):\n muts.add_snv(self.snvs[snv])\n\n for dlt in self.deletions:\n if not (dlt in nms.deletions and passes_normal_criteria(nms.deletions[dlt])):\n muts.add_deletion(self.deletions[dlt])\n\n for ins in self.insertions:\n if not (ins in nms.insertions and passes_normal_criteria(nms.insertions[ins])):\n muts.add_insertion(self.insertions[ins])\n\n return muts"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef to_oncotator(self):\n if self.type == \".\":\n ref = self.ref\n alt = self.change\n start = self.pos\n end = self.pos\n elif self.type == \"-\":\n ref = self.change\n alt = \"-\"\n start = self.pos + 1\n end = start + len(self.change)\n elif self.type == \"+\":\n ref = \"-\"\n alt = self.change\n start = self.pos\n end = start + len(self.change)\n else:\n raise(Exception(\"Unexpected mutation type: {}\".format(self.type)))\n return \"{chrom}\\t{start}\\t{end}\\t{ref}\\t{alt}\".format(chrom=self.chrom, start=start,\n end=end, ref=ref, alt=alt)", "response": "Returns the mutation in oncotator input format. Assumes mutations have\n vcf style positions."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd routes to Application that use specified handlers.", "response": "def add_handlers(self, room_handler=None, transaction_handler=None,\n user_handler=None):\n \"\"\"Adds routes to Application that use specified handlers.\"\"\"\n # Add all the normal matrix API routes\n if room_handler:\n room = resources.Room(room_handler,\n self.Api)\n self.add_route(\"/rooms/{room_alias}\", room)\n if transaction_handler:\n transaction = resources.Transaction(transaction_handler,\n self.Api)\n self.add_route(\"/transactions/{txn_id}\", transaction)\n if user_handler:\n user = resources.User(user_handler,\n self.Api)\n self.add_route(\"/users/{user_id}\", user)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmerges values in place from *right* into *left*. :param left: mapping to merge into :param right: mapping to merge from :param path: `list` of keys processed before (used for error reporting only, should only need to be provided by recursive calls) :param conflict: action to be taken on merge conflict, raising an error or overwriting an existing value :return: *left*, for convenience", "response": "def _merge(left, right, path=None, conflict=_Conflict.error):\n \"\"\"\n Merges values in place from *right* into *left*.\n\n :param left: mapping to merge into\n :param right: mapping to merge from\n :param path: `list` of keys processed before (used for error reporting\n only, should only need to be provided by recursive calls)\n :param conflict: action to be taken on merge conflict, raising an error\n or overwriting an existing value\n :return: *left*, for convenience\n \"\"\"\n path = path or []\n conflict = _Conflict(conflict)\n\n for key in right:\n if key in left:\n if isinstance(left[key], Mapping) and isinstance(right[key], Mapping):\n # recurse, merge left and right dict values, update path for current 'step'\n _merge(left[key], right[key], path + [key], conflict=conflict)\n elif left[key] != right[key]:\n if conflict is _Conflict.error:\n # not both dicts we could merge, but also not the same, this doesn't work\n conflict_path = '.'.join(path + [key])\n raise MergeConflictError('merge conflict at {}'.format(conflict_path), key=conflict_path)\n else:\n # overwrite left value with right value\n left[key] = right[key]\n # else: left[key] is already equal to right[key], no action needed\n else:\n # key not yet in left or not considering conflicts, simple addition of right's mapping to left\n left[key] = right[key]\n\n return left"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef log_mon_value(name, value=1, **kwargs):\n message = '{} => {}'.format(name, value)\n log_mon.info({'metric_name': name, 'value': value, 'message': message, **kwargs})", "response": "simplest monitoring function to be aggregated with sum\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef use_form(form_class, request=None, **top_kwargs):\n\n def validated_form(request, **kwargs):\n # import ipdb; ipdb.set_trace()\n data = request.query_params.dict() if request.method in ['GET'] else request.data\n if isinstance(data, QueryDict):\n form = form_class(data={**data.dict(), **kwargs})\n elif isinstance(data, dict):\n form = form_class(data={**data, **kwargs})\n else:\n form = form_class(data=data, **kwargs)\n form.is_valid(raise_exception=True)\n return form\n\n if request:\n kwargs = {}\n if request.resolver_match:\n kwargs = {**request.resolver_match.kwargs}\n if top_kwargs:\n kwargs = {**kwargs, **top_kwargs}\n\n return validated_form(request, **kwargs).validated_data\n\n def wrap(func):\n def method_wrap(view, request, *args, **kwargs):\n form = validated_form(request, **kwargs)\n if hasattr(view, 'log'):\n form.log = view.log\n return func(view, form.validated_data, *args, **kwargs)\n\n def function_wrap(request, *args, **kwargs):\n form = validated_form(request, **kwargs)\n return func(form.validated_data, *args, **kwargs)\n\n def inner(*args, **kwargs):\n is_method = isinstance(args[0], APIView)\n return (method_wrap if is_method else function_wrap)(*args, **kwargs)\n\n return inner\n\n return wrap", "response": "Use a form to create a new object."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef dx_orbit_sys(t, X):\n '''X = [\n m1x, m1y,\n m2x, m2y,\n m3x, m3y,\n m4x, m4y,\n m1vx, m1vy,\n m2vx, m2vy,\n m3vx, m3vy,\n m4vx, m4vy\n ]\n '''\n (m1x, m1y,\n m2x, m2y,\n m3x, m3y,\n m4x, m4y,\n m1vx, m1vy,\n m2vx, m2vy,\n m3vx, m3vy,\n m4vx, m4vy) = X\n m_moon1 = 7.342*(10**22) # kg\n m_moon2 = 7.342*(10**22) # kg\n m_moon3 = 7.342*(10**22) # kg\n m_moon4 = 7.342*(10**22) # kg\n G = 6.67408*(10**-11) # m**3 kg**\u22121 s**\u22122\n dm12 = sqrt((m1x - m2x)**2 + (m1y - m2y)**2)\n dm13 = sqrt((m1x - m3x)**2 + (m1y - m3y)**2)\n dm14 = sqrt((m1x - m4x)**2 + (m1y - m4y)**2)\n dm23 = sqrt((m2x - m3x)**2 + (m2y - m3y)**2)\n dm24 = sqrt((m2x - m4x)**2 + (m2y - m4y)**2)\n dm34 = sqrt((m3x - m4x)**2 + (m3y - m4y)**2)\n f12 = G * m_moon1 * m_moon2 / (dm12 * dm12)\n f13 = G * m_moon1 * m_moon3 / (dm13 * dm13)\n f14 = G * m_moon1 * m_moon4 / (dm14 * dm14)\n f23 = G * m_moon2 * m_moon3 / (dm23 * dm23)\n f24 = G * m_moon2 * m_moon4 / (dm24 * dm24)\n f34 = G * m_moon3 * m_moon4 / (dm34 * dm34)\n dr12 = atan2(m2y - m1y, m2x - m1x)\n dr13 = atan2(m3y - m1y, m3x - m1x)\n dr14 = atan2(m4y - m1y, m4x - m1x)\n dr23 = atan2(m3y - m2y, m3x - m2x)\n dr24 = atan2(m4y - m2y, m4x - m2x)\n dr34 = atan2(m4y - m3y, m4x - m3x)\n f1x = f12 * cos(dr12) + f13 * cos(dr13) + f14 * cos(dr14)\n f1y = f12 * sin(dr12) + f13 * sin(dr13) + f14 * sin(dr14)\n f2x = f12 * cos(dr12 + pi) + f23 * cos(dr23) + f24 * cos(dr24)\n f2y = f12 * sin(dr12 + pi) + f23 * sin(dr23) + f24 * sin(dr24)\n f3x = f13 * cos(dr13 + pi) + f23 * cos(dr23 + pi) + f34 * cos(dr34)\n f3y = f13 * sin(dr13 + pi) + f23 * sin(dr23 + pi) + f34 * sin(dr34)\n f4x = f14 * cos(dr14 + pi) + f24 * cos(dr24 + pi) + f34 * cos(dr34 + pi)\n f4y = f14 * sin(dr14 + pi) + f24 * sin(dr24 + pi) + f34 * sin(dr34 + pi)\n dX = [\n m1vx,\n m1vy,\n m2vx,\n m2vy,\n m3vx,\n m3vy,\n m4vx,\n m4vy,\n f1x / m_moon1,\n f1y / m_moon1,\n f2x / m_moon2,\n f2y / m_moon2,\n f3x / m_moon3,\n f3y / m_moon3,\n f4x / m_moon4,\n f4y / m_moon4,\n ]\n return dX", "response": "An orbit system of the given moon."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add(self, client):\n if client.pool_id in self._client_ids:\n log.info(\"%r is already in the penalty box. Ignoring.\", client)\n return\n release = time.time() + self._min_wait\n heapq.heappush(self._clients, (release, (client, self._min_wait)))\n self._client_ids.add(client.pool_id)", "response": "Adds a client to the penalty box."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get(self):\n now = time.time()\n while self._clients and self._clients[0][0] < now:\n _, (client, last_wait) = heapq.heappop(self._clients)\n connect_start = time.time()\n try:\n client.echo(\"test\") # reconnected if this succeeds.\n self._client_ids.remove(client.pool_id)\n yield client\n except (ConnectionError, TimeoutError):\n timer = time.time() - connect_start\n wait = min(int(last_wait * self._multiplier), self._max_wait)\n heapq.heappush(self._clients, (time.time() + wait, (client, wait)))\n log.info(\n \"%r is still down after a %s second attempt to connect. Retrying in %ss.\",\n client,\n timer,\n wait,\n )", "response": "Get any clients ready to be used."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef string(_object):\n if is_callable(_object):\n _validator = _object\n\n @wraps(_validator)\n def decorated(value):\n ensure(isinstance(value, basestring), \"not of type string\")\n return _validator(value)\n return decorated\n ensure(isinstance(_object, basestring), \"not of type string\")", "response": "Validates a given input is of type string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nvalidating a given input is of type boolean.", "response": "def boolean(_object):\n \"\"\"\n Validates a given input is of type boolean.\n\n Example usage::\n\n data = {'a' : True}\n schema = ('a', boolean)\n\n You can also use this as a decorator, as a way to check for the\n input before it even hits a validator you may be writing.\n\n .. note::\n If the argument is a callable, the decorating behavior will be\n triggered, otherwise it will act as a normal function.\n\n \"\"\"\n if is_callable(_object):\n _validator = _object\n\n @wraps(_validator)\n def decorated(value):\n ensure(isinstance(value, bool), \"not of type boolean\")\n return _validator(value)\n return decorated\n ensure(isinstance(_object, bool), \"not of type boolean\")"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef dictionary(_object, *args):\n error_msg = 'not of type dictionary'\n if is_callable(_object):\n _validator = _object\n\n @wraps(_validator)\n def decorated(value):\n ensure(isinstance(value, dict), error_msg)\n return _validator(value)\n return decorated\n try:\n ensure(isinstance(_object, dict), error_msg)\n except AssertionError:\n if args:\n msg = 'did not pass validation against callable: dictionary'\n raise Invalid('', msg=msg, reason=error_msg, *args)\n raise", "response": "Validates a given input is of type dictionary."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef array(_object):\n if is_callable(_object):\n _validator = _object\n\n @wraps(_validator)\n def decorated(value):\n ensure(isinstance(value, list), \"not of type array\")\n return _validator(value)\n return decorated\n ensure(isinstance(_object, list), \"not of type array\")", "response": "Validates a given input is of type list."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nvalidating a given input is of type int..", "response": "def integer(_object):\n \"\"\"\n Validates a given input is of type int..\n\n Example usage::\n\n data = {'a' : 21}\n schema = ('a', integer)\n\n You can also use this as a decorator, as a way to check for the\n input before it even hits a validator you may be writing.\n\n .. note::\n If the argument is a callable, the decorating behavior will be\n triggered, otherwise it will act as a normal function.\n \"\"\"\n if is_callable(_object):\n _validator = _object\n\n @wraps(_validator)\n def decorated(value):\n ensure(isinstance(value, int), \"not of type int\")\n return _validator(value)\n return decorated\n ensure(isinstance(_object, int), \"not of type int\")"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a constant value TensorFluent with given dtype.", "response": "def constant(cls,\n value: Value,\n dtype: tf.DType = tf.float32) -> 'TensorFluent':\n '''Returns a constant `value` TensorFluent with given `dtype`.\n\n Args:\n value: The constant value.\n dtype: The output's data type.\n\n Returns:\n A constant TensorFluent.\n '''\n t = tf.constant(value, dtype=dtype)\n scope = [] # type: List\n batch = False\n return TensorFluent(t, scope, batch=batch)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef Bernoulli(cls,\n mean: 'TensorFluent',\n batch_size: Optional[int] = None) -> Tuple[Distribution, 'TensorFluent']:\n '''Returns a TensorFluent for the Bernoulli sampling op with given mean parameter.\n\n Args:\n mean: The mean parameter of the Bernoulli distribution.\n batch_size: The size of the batch (optional).\n\n Returns:\n The Bernoulli distribution and a TensorFluent sample drawn from the distribution.\n '''\n probs = mean.tensor\n dist = tf.distributions.Bernoulli(probs=probs, dtype=tf.bool)\n batch = mean.batch\n if not batch and batch_size is not None:\n t = dist.sample(batch_size)\n batch = True\n else:\n t = dist.sample()\n scope = mean.scope.as_list()\n return (dist, TensorFluent(t, scope, batch=batch))", "response": "Returns a TensorFluent for the Bernoulli sampling op with given mean parameter."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a TensorFluent for the Uniform sampling op with given low and high parameters.", "response": "def Uniform(cls,\n low: 'TensorFluent', high: 'TensorFluent',\n batch_size: Optional[int] = None) -> Tuple[Distribution, 'TensorFluent']:\n '''Returns a TensorFluent for the Uniform sampling op with given low and high parameters.\n\n Args:\n low: The low parameter of the Uniform distribution.\n high: The high parameter of the Uniform distribution.\n batch_size: The size of the batch (optional).\n\n Returns:\n The Uniform distribution and a TensorFluent sample drawn from the distribution.\n\n Raises:\n ValueError: If parameters do not have the same scope.\n '''\n if low.scope != high.scope:\n raise ValueError('Uniform distribution: parameters must have same scope!')\n dist = tf.distributions.Uniform(low.tensor, high.tensor)\n batch = low.batch or high.batch\n if not batch and batch_size is not None:\n t = dist.sample(batch_size)\n batch = True\n else:\n t = dist.sample()\n scope = low.scope.as_list()\n return (dist, TensorFluent(t, scope, batch=batch))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef Normal(cls,\n mean: 'TensorFluent', variance: 'TensorFluent',\n batch_size: Optional[int] = None) -> Tuple[Distribution, 'TensorFluent']:\n '''Returns a TensorFluent for the Normal sampling op with given mean and variance.\n\n Args:\n mean: The mean parameter of the Normal distribution.\n variance: The variance parameter of the Normal distribution.\n batch_size: The size of the batch (optional).\n\n Returns:\n The Normal distribution and a TensorFluent sample drawn from the distribution.\n\n Raises:\n ValueError: If parameters do not have the same scope.\n '''\n if mean.scope != variance.scope:\n raise ValueError('Normal distribution: parameters must have same scope!')\n loc = mean.tensor\n scale = tf.sqrt(variance.tensor)\n dist = tf.distributions.Normal(loc, scale)\n batch = mean.batch or variance.batch\n if not batch and batch_size is not None:\n t = dist.sample(batch_size)\n batch = True\n else:\n t = dist.sample()\n scope = mean.scope.as_list()\n return (dist, TensorFluent(t, scope, batch=batch))", "response": "Returns a TensorFluent for the Normal sampling op with given mean and variance."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef Gamma(cls,\n shape: 'TensorFluent',\n scale: 'TensorFluent',\n batch_size: Optional[int] = None) -> Tuple[Distribution, 'TensorFluent']:\n '''Returns a TensorFluent for the Gamma sampling op with given shape and scale parameters.\n\n Args:\n shape: The shape parameter of the Gamma distribution.\n scale: The scale parameter of the Gamma distribution.\n batch_size: The size of the batch (optional).\n\n Returns:\n The Gamma distribution and a TensorFluent sample drawn from the distribution.\n\n Raises:\n ValueError: If parameters do not have the same scope.\n '''\n if shape.scope != scale.scope:\n raise ValueError('Gamma distribution: parameters must have same scope!')\n concentration = shape.tensor\n rate = 1 / scale.tensor\n dist = tf.distributions.Gamma(concentration, rate)\n batch = shape.batch or scale.batch\n if not batch and batch_size is not None:\n t = dist.sample(batch_size)\n batch = True\n else:\n t = dist.sample()\n scope = shape.scope.as_list()\n return (dist, TensorFluent(t, scope, batch=batch))", "response": "Returns a TensorFluent for the Gamma sampling op with given shape and scale parameters."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a TensorFluent for the Exponential sampling op with given mean parameter.", "response": "def Exponential(cls,\n mean: 'TensorFluent',\n batch_size: Optional[int] = None) -> Tuple[Distribution, 'TensorFluent']:\n '''Returns a TensorFluent for the Exponential sampling op with given mean parameter.\n\n Args:\n mean: The mean parameter of the Exponential distribution.\n batch_size: The size of the batch (optional).\n\n Returns:\n The Exponential distribution and a TensorFluent sample drawn from the distribution.\n '''\n rate = 1 / mean.tensor\n dist = tf.distributions.Exponential(rate)\n batch = mean.batch\n if not batch and batch_size is not None:\n t = dist.sample(batch_size)\n batch = True\n else:\n t = dist.sample()\n scope = mean.scope.as_list()\n return (dist, TensorFluent(t, scope, batch=batch))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef stop_gradient(cls, x: 'TensorFluent') -> 'TensorFluent':\n '''Returns a copy of the input fluent with stop_gradient at tensor level.\n\n Args:\n x: The input fluent.\n\n Returns:\n A TensorFluent that stops backpropagation of gradient computations.\n '''\n scope = x.scope.as_list()\n batch = x.batch\n return TensorFluent(tf.stop_gradient(x.tensor), scope, batch)", "response": "Returns a copy of the input fluent with stop_gradient at tensor level."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef stop_batch_gradient(cls, x: 'TensorFluent', stop_batch: tf.Tensor) -> 'TensorFluent':\n '''Returns a copy of the inputs fluent with stop_gradient applied at batch level.\n\n Args:\n x: The input fluent.\n stop_batch: A boolean tf.Tensor with shape=(batch_size, ...)\n\n Returns:\n A TensorFluent that conditionally stops backpropagation of gradient computations.\n '''\n scope = x.scope.as_list()\n batch = x.batch\n tensor = tf.where(stop_batch, tf.stop_gradient(x.tensor), x.tensor)\n return TensorFluent(tensor, scope, batch)", "response": "Returns a copy of the fluent with stop_gradient applied at batch level."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef abs(cls, x: 'TensorFluent') -> 'TensorFluent':\n '''Returns a TensorFluent for the abs function.\n\n Args:\n x: The input fluent.\n\n Returns:\n A TensorFluent wrapping the abs function.\n '''\n return cls._unary_op(x, tf.abs, tf.float32)", "response": "Returns a TensorFluent for the abs function."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef exp(cls, x: 'TensorFluent') -> 'TensorFluent':\n '''Returns a TensorFluent for the exp function.\n\n Args:\n x: The input fluent.\n\n Returns:\n A TensorFluent wrapping the exp function.\n '''\n return cls._unary_op(x, tf.exp, tf.float32)", "response": "Returns a TensorFluent for the exp function."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a TensorFluent for the log function.", "response": "def log(cls, x: 'TensorFluent') -> 'TensorFluent':\n '''Returns a TensorFluent for the log function.\n\n Args:\n x: The input fluent.\n\n Returns:\n A TensorFluent wrapping the log function.\n '''\n return cls._unary_op(x, tf.log, tf.float32)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a TensorFluent for the sqrt function.", "response": "def sqrt(cls, x: 'TensorFluent') -> 'TensorFluent':\n '''Returns a TensorFluent for the sqrt function.\n\n Args:\n x: The input fluent.\n\n Returns:\n A TensorFluent wrapping the sqrt function.\n '''\n return cls._unary_op(x, tf.sqrt, tf.float32)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a TensorFluent for the cos function.", "response": "def cos(cls, x: 'TensorFluent') -> 'TensorFluent':\n '''Returns a TensorFluent for the cos function.\n\n Args:\n x: The input fluent.\n\n Returns:\n A TensorFluent wrapping the cos function.\n '''\n return cls._unary_op(x, tf.cos, tf.float32)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef sin(cls, x: 'TensorFluent') -> 'TensorFluent':\n '''Returns a TensorFluent for the sin function.\n\n Args:\n x: The input fluent.\n\n Returns:\n A TensorFluent wrapping the sin function.\n '''\n return cls._unary_op(x, tf.sin, tf.float32)", "response": "Returns a TensorFluent for the sin function."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a TensorFluent for the tan function.", "response": "def tan(cls, x: 'TensorFluent') -> 'TensorFluent':\n '''Returns a TensorFluent for the tan function.\n\n Args:\n x: The input fluent.\n\n Returns:\n A TensorFluent wrapping the tan function.\n '''\n return cls._unary_op(x, tf.tan, tf.float32)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a TensorFluent for the acos function.", "response": "def acos(cls, x: 'TensorFluent') -> 'TensorFluent':\n '''Returns a TensorFluent for the arccos function.\n\n Args:\n x: The input fluent.\n\n Returns:\n A TensorFluent wrapping the arccos function.\n '''\n return cls._unary_op(x, tf.acos, tf.float32)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a TensorFluent for the arcsin function.", "response": "def asin(cls, x: 'TensorFluent') -> 'TensorFluent':\n '''Returns a TensorFluent for the arcsin function.\n\n Args:\n x: The input fluent.\n\n Returns:\n A TensorFluent wrapping the arcsin function.\n '''\n return cls._unary_op(x, tf.asin, tf.float32)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef atan(cls, x: 'TensorFluent') -> 'TensorFluent':\n '''Returns a TensorFluent for the arctan function.\n\n Args:\n x: The input fluent.\n\n Returns:\n A TensorFluent wrapping the arctan function.\n '''\n return cls._unary_op(x, tf.atan2, tf.float32)", "response": "Returns a TensorFluent for the arctan function."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef round(cls, x: 'TensorFluent') -> 'TensorFluent':\n '''Returns a TensorFluent for the round function.\n\n Args:\n x: The input fluent.\n\n Returns:\n A TensorFluent wrapping the round function.\n '''\n return cls._unary_op(x, tf.round, tf.float32)", "response": "Returns a TensorFluent for the round function."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef ceil(cls, x: 'TensorFluent') -> 'TensorFluent':\n '''Returns a TensorFluent for the ceil function.\n\n Args:\n x: The input fluent.\n\n Returns:\n A TensorFluent wrapping the ceil function.\n '''\n return cls._unary_op(x, tf.ceil, tf.float32)", "response": "Returns a TensorFluent for the ceil function."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef floor(cls, x: 'TensorFluent') -> 'TensorFluent':\n '''Returns a TensorFluent for the floor function.\n\n Args:\n x: The input fluent.\n\n Returns:\n A TensorFluent wrapping the floor function.\n '''\n return cls._unary_op(x, tf.floor, tf.float32)", "response": "Returns a TensorFluent for the floor function."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a TensorFluent for the pow function.", "response": "def pow(cls, x: 'TensorFluent', y: 'TensorFluent') -> 'TensorFluent':\n '''Returns a TensorFluent for the pow function.TensorFluent\n\n Args:\n x: The first operand.\n y: The second operand.\n\n Returns:\n A TensorFluent wrapping the pow function.\n '''\n return cls._binary_op(x, y, tf.pow, tf.float32)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a TensorFluent for the maximum function.", "response": "def max(cls, x: 'TensorFluent', y: 'TensorFluent') -> 'TensorFluent':\n '''Returns a TensorFluent for the maximum function.TensorFluent\n\n Args:\n x: The first operand.\n y: The second operand.\n\n Returns:\n A TensorFluent wrapping the maximum function.\n '''\n return cls._binary_op(x, y, tf.maximum, tf.float32)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a TensorFluent for the minimum function.", "response": "def min(cls, x: 'TensorFluent', y: 'TensorFluent') -> 'TensorFluent':\n '''Returns a TensorFluent for the minimum function.\n\n Args:\n x: The first operand.\n y: The second operand.\n\n Returns:\n A TensorFluent wrapping the minimum function.\n '''\n return cls._binary_op(x, y, tf.minimum, tf.float32)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a TensorFluent for the if - then - else control op.", "response": "def if_then_else(cls,\n condition: 'TensorFluent',\n true_case: 'TensorFluent',\n false_case: 'TensorFluent') -> 'TensorFluent':\n '''Returns a TensorFluent for the control op if-then-else.\n\n Args:\n condition: Boolean fluent for the if condition.\n true_case: Fluent returned in the true clause.\n false_case: Fluent returned in the false clause.\n\n Returns:\n A TensorFluent wrapping the if-then-else control statement.\n\n Raises:\n ValueError: If cases don't have same shape.\n '''\n true = TensorFluent.constant(True, tf.bool)\n false = TensorFluent.constant(False, tf.bool)\n ite = (condition == true) * true_case + (condition == false) * false_case\n if true_case.dtype == tf.bool and false_case.dtype == tf.bool:\n ite = ite.cast(tf.bool)\n return ite"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a TensorFluent for the binary op applied to fluents x and y.", "response": "def _binary_op(cls,\n x: 'TensorFluent',\n y: 'TensorFluent',\n op: Callable[[tf.Tensor, tf.Tensor], tf.Tensor],\n dtype: tf.DType) -> 'TensorFluent':\n '''Returns a TensorFluent for the binary `op` applied to fluents `x` and `y`.\n\n Args:\n x: The first operand.\n y: The second operand.\n op: The binary operator.\n dtype: The output's data type.\n\n Returns:\n A TensorFluent wrapping the binary operator's output.\n '''\n # scope\n s1 = x.scope.as_list()\n s2 = y.scope.as_list()\n scope, perm1, perm2 = TensorFluentScope.broadcast(s1, s2)\n if x.batch and perm1 != []:\n perm1 = [0] + [p+1 for p in perm1]\n if y.batch and perm2 != []:\n perm2 = [0] + [p+1 for p in perm2]\n x = x.transpose(perm1)\n y = y.transpose(perm2)\n\n # shape\n reshape1, reshape2 = TensorFluentShape.broadcast(x.shape, y.shape)\n if reshape1 is not None:\n x = x.reshape(reshape1)\n if reshape2 is not None:\n y = y.reshape(reshape2)\n\n # dtype\n x = x.cast(dtype)\n y = y.cast(dtype)\n\n # operation\n t = op(x.tensor, y.tensor)\n\n # batch\n batch = x.batch or y.batch\n\n return TensorFluent(t, scope, batch=batch)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _unary_op(cls,\n x: 'TensorFluent',\n op: Callable[[tf.Tensor], tf.Tensor],\n dtype: tf.DType) -> 'TensorFluent':\n '''Returns a TensorFluent for the unary `op` applied to fluent `x`.\n\n Args:\n x: The input fluent.\n op: The unary operation.\n dtype: The output's data type.\n\n Returns:\n A TensorFluent wrapping the unary operator's output.\n '''\n x = x.cast(dtype)\n t = op(x.tensor)\n scope = x.scope.as_list()\n batch = x.batch\n return TensorFluent(t, scope, batch=batch)", "response": "Returns a TensorFluent for the unary op applied to fluent x."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _aggregation_op(cls,\n op: Callable[[tf.Tensor, Optional[Sequence[int]]], tf.Tensor],\n x: 'TensorFluent',\n vars_list: List[str]) -> 'TensorFluent':\n '''Returns a TensorFluent for the aggregation `op` applied to fluent `x`.\n\n Args:\n op: The aggregation operation.\n x: The input fluent.\n vars_list: The list of variables to be aggregated over.\n\n Returns:\n A TensorFluent wrapping the aggregation operator's output.\n '''\n axis = cls._varslist2axis(x, vars_list)\n t = op(x.tensor, axis)\n\n scope = []\n for var in x.scope.as_list():\n if var not in vars_list:\n scope.append(var)\n\n batch = x.batch\n\n return TensorFluent(t, scope, batch=batch)", "response": "Returns a TensorFluent for the aggregation op applied to fluent x."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _varslist2axis(cls, fluent: 'TensorFluent', vars_list: List[str]) -> List[int]:\n '''Maps the `vars_list` into a list of axis indices\n corresponding to the `fluent` scope.\n\n Args:\n x: The fluent.\n vars_list: The list of variables to be aggregated over.\n\n Returns:\n List[int]: a list of axis.\n '''\n axis = []\n for var in vars_list:\n if var in fluent.scope.as_list():\n ax = fluent.scope.index(var)\n if fluent.batch:\n ax += 1\n axis.append(ax)\n return axis", "response": "Maps the vars_list into a list of axis indices\n corresponding to the fluent scope."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a TensorFluent for the cast operation with given dtype.", "response": "def cast(self, dtype: tf.DType) -> 'TensorFluent':\n '''Returns a TensorFluent for the cast operation with given `dtype`.\n\n Args:\n dtype: The output's data type.\n\n Returns:\n A TensorFluent wrapping the cast operation.\n '''\n if self.dtype == dtype:\n return self\n t = tf.cast(self.tensor, dtype)\n scope = self.scope.as_list()\n batch = self.batch\n return TensorFluent(t, scope, batch=batch)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a TensorFluent for the reshape operation with given shape.", "response": "def reshape(self, shape: tf.TensorShape) -> 'TensorFluent':\n '''Returns a TensorFluent for the reshape operation with given `shape`.\n\n Args:\n shape: The output's shape.\n\n Returns:\n A TensorFluent wrapping the reshape operation.\n '''\n t = tf.reshape(self.tensor, shape)\n scope = self.scope.as_list()\n batch = self.batch\n return TensorFluent(t, scope, batch=batch)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a TensorFluent for the transpose operation with given permutation.", "response": "def transpose(self, permutation: Optional[List[int]] = None) -> 'TensorFluent':\n '''Returns a TensorFluent for the transpose operation with given `permutation`.\n\n Args:\n permutation: The output's shape permutation.\n\n Returns:\n A TensorFluent wrapping the transpose operation.\n '''\n if permutation == []:\n return self\n t = tf.transpose(self.tensor, permutation) if permutation != [] else self.tensor\n scope = self.scope.as_list()\n batch = self.batch\n return TensorFluent(t, scope, batch=batch)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sum(self, vars_list: List[str]) -> 'TensorFluent':\n '''Returns the TensorFluent for the sum aggregation function.\n\n Args:\n vars_list: The list of variables to be aggregated over.\n\n Returns:\n A TensorFluent wrapping the sum aggregation function.\n '''\n operand = self\n if operand.dtype == tf.bool:\n operand = operand.cast(tf.float32)\n return self._aggregation_op(tf.reduce_sum, operand, vars_list)", "response": "Returns the TensorFluent for the sum aggregation function."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef avg(self, vars_list: List[str]) -> 'TensorFluent':\n '''Returns the TensorFluent for the avg aggregation function.\n\n Args:\n vars_list: The list of variables to be aggregated over.\n\n Returns:\n A TensorFluent wrapping the avg aggregation function.\n '''\n operand = self\n if operand.dtype == tf.bool:\n operand = operand.cast(tf.float32)\n return self._aggregation_op(tf.reduce_mean, operand, vars_list)", "response": "Returns the TensorFluent for the avg aggregation function."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef prod(self, vars_list: List[str]) -> 'TensorFluent':\n '''Returns the TensorFluent for the prod aggregation function.\n\n Args:\n vars_list: The list of variables to be aggregated over.\n\n Returns:\n A TensorFluent wrapping the prod aggregation function.\n '''\n operand = self\n if operand.dtype == tf.bool:\n operand = operand.cast(tf.float32)\n return self._aggregation_op(tf.reduce_prod, operand, vars_list)", "response": "Returns the TensorFluent for the prod aggregation function."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the TensorFluent for the maximum aggregation function.", "response": "def maximum(self, vars_list: List[str]) -> 'TensorFluent':\n '''Returns the TensorFluent for the maximum aggregation function.\n\n Args:\n vars_list: The list of variables to be aggregated over.\n\n Returns:\n A TensorFluent wrapping the maximum aggregation function.\n '''\n return self._aggregation_op(tf.reduce_max, self, vars_list)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the TensorFluent for the minimum aggregation function.", "response": "def minimum(self, vars_list: List[str]) -> 'TensorFluent':\n '''Returns the TensorFluent for the minimum aggregation function.\n\n Args:\n vars_list: The list of variables to be aggregated over.\n\n Returns:\n A TensorFluent wrapping the minimum aggregation function.\n '''\n return self._aggregation_op(tf.reduce_min, self, vars_list)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the TensorFluent for the forall aggregation function.", "response": "def forall(self, vars_list: List[str]) -> 'TensorFluent':\n '''Returns the TensorFluent for the forall aggregation function.\n\n Args:\n vars_list: The list of variables to be aggregated over.\n\n Returns:\n A TensorFluent wrapping the forall aggregation function.\n '''\n return self._aggregation_op(tf.reduce_all, self, vars_list)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the TensorFluent for the exists aggregation function.", "response": "def exists(self, vars_list: List[str]) -> 'TensorFluent':\n '''Returns the TensorFluent for the exists aggregation function.\n\n Args:\n vars_list: The list of variables to be aggregated over.\n\n Returns:\n A TensorFluent wrapping the exists aggregation function.\n '''\n return self._aggregation_op(tf.reduce_any, self, vars_list)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef exclude_sources(exclude, section=False):\n newlist = sources.copy()\n if not isinstance(exclude, list):\n exclude = [exclude]\n\n for source in exclude:\n if not section:\n newlist.remove(source)\n else:\n pos = newlist.index(source) + 1\n if pos == len(sources):\n return []\n newlist = sources[pos:]\n return newlist", "response": "Returns a narrower list of sources."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsearches for lyrics of a single song and returns a Result object with the lyrics and various stats collected in the process.", "response": "def get_lyrics(song, l_sources=None):\n \"\"\"\n Searches for lyrics of a single song and returns a Result object with the\n various stats collected in the process.\n\n The optional parameter 'sources' specifies an alternative list of sources.\n If not present, the main list will be used.\n \"\"\"\n if l_sources is None:\n l_sources = sources\n\n if song.lyrics and not CONFIG['overwrite']:\n logger.debug('%s already has embedded lyrics', song)\n return None\n\n runtimes = {}\n source = None\n for l_source in l_sources:\n start = time.time()\n try:\n lyrics = l_source(song)\n except (HTTPError, HTTPException, URLError, ConnectionError):\n lyrics = ''\n\n runtimes[l_source] = time.time() - start\n if lyrics != '':\n source = l_source\n break\n\n if lyrics != '':\n logger.info('++ %s: Found lyrics for %s\\n', source.__name__, song)\n song.lyrics = lyrics\n else:\n logger.info(\"Couldn't find lyrics for %s\\n\", song)\n source = None\n\n return Result(song, source, runtimes)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_lyrics_threaded(song, l_sources=None):\n if l_sources is None:\n l_sources = sources\n\n if song.lyrics and not CONFIG['overwrite']:\n logger.debug('%s already has embedded lyrics', song)\n return None\n\n runtimes = {}\n queue = Queue()\n pool = [LyrThread(source, song, queue) for source in l_sources]\n for thread in pool:\n thread.start()\n\n for _ in range(len(pool)):\n result = queue.get()\n runtimes[result['source']] = result['runtime']\n if result['lyrics']:\n break\n\n if result['lyrics']:\n song.lyrics = result['lyrics']\n source = result['source']\n else:\n source = None\n\n return Result(song, source, runtimes)", "response": "Returns a list of all the lyrics of a single song in a single thread."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef process_result(result):\n found = result.source is not None\n if found:\n if hasattr(result.song, 'filename'):\n audiofile = eyed3.load(result.song.filename)\n audiofile.tag.lyrics.set(result.song.lyrics)\n audiofile.tag.save()\n print(f'{id_source(result.source)} Lyrics added for {result.song}')\n else:\n print(f\"\"\"FROM {id_source(result.source, full=True)}\n\n{result.song.lyrics}\n-----------------------------------------------------------------------------\\\n\"\"\")\n else:\n print(f'Lyrics for {result.song} not found')\n\n return found", "response": "Process a result object by finding the lyrics and saving them to the corresponding file."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncall get_lyrics_threaded for a song or list of songs.", "response": "def run(songs):\n \"\"\"\n Calls get_lyrics_threaded for a song or list of songs.\n \"\"\"\n if not hasattr(songs, '__iter__'):\n result = get_lyrics_threaded(songs)\n process_result(result)\n else:\n start = time.time()\n stats = run_mp(songs)\n end = time.time()\n if CONFIG['print_stats']:\n stats.print_stats()\n total_time = end - start\n total_time = '%d:%02d:%02d' % (total_time / 3600,\n (total_time / 3600) / 60,\n (total_time % 3600) % 60)\n print(f'Total time: {total_time}')"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef parse_setup(options: Union[List, str]) -> str:\n if isinstance(options, str):\n return options\n return \"\\n\".join(options)", "response": "Convert potentially a list of commands into a single string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a scheduler file from a job.", "response": "def create_scheduler_file(scheduler: str, job: Job) -> str:\n \"\"\"Substitute values into a template scheduler file.\"\"\"\n logger.debug(\"Create Scheduler File Function\")\n\n if job.scheduler_options is None:\n scheduler_options: Dict[str, Any] = {}\n else:\n scheduler_options = deepcopy(job.scheduler_options)\n try:\n setup_string = parse_setup(scheduler_options[\"setup\"])\n del scheduler_options[\"setup\"]\n except KeyError:\n setup_string = \"\"\n # Create header\n header_string = create_header_string(scheduler, **scheduler_options)\n header_string += get_array_string(scheduler, len(job))\n\n if scheduler.upper() == \"SLURM\":\n workdir = r\"$SLURM_SUBMIT_DIR\"\n array_index = r\"$SLURM_ARRAY_TASK_ID\"\n elif scheduler.upper() == \"PBS\":\n workdir = r\"$PBS_O_WORKDIR\"\n array_index = r\"$PBS_ARRAY_INDEX\"\n\n return header_string + SCHEDULER_TEMPLATE.format(\n workdir=workdir,\n command_list=job.as_bash_array(),\n setup=setup_string,\n array_index=array_index,\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ninitialize the state of the UI.", "response": "def initialize_state(self):\n \"\"\" Call this to initialize the state of the UI after everything has been connected. \"\"\"\n if self.__hardware_source:\n self.__data_item_states_changed_event_listener = self.__hardware_source.data_item_states_changed_event.listen(self.__data_item_states_changed)\n self.__acquisition_state_changed_event_listener = self.__hardware_source.acquisition_state_changed_event.listen(self.__acquisition_state_changed)\n if self.on_display_name_changed:\n self.on_display_name_changed(self.display_name)\n self.__update_buttons()\n if self.on_data_item_states_changed:\n self.on_data_item_states_changed(list())"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalling this when the user clicks the play button.", "response": "def handle_play_clicked(self):\n \"\"\" Call this when the user clicks the play/pause button. \"\"\"\n if self.__hardware_source:\n if self.is_playing:\n self.__hardware_source.stop_playing()\n else:\n self.__hardware_source.start_playing()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef range_type_to_dtype(range_type: str) -> Optional[tf.DType]:\n '''Maps RDDL range types to TensorFlow dtypes.'''\n range2dtype = {\n 'real': tf.float32,\n 'int': tf.int32,\n 'bool': tf.bool\n }\n return range2dtype[range_type]", "response": "Maps RDDL range types to TensorFlow dtypes."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef python_type_to_dtype(python_type: type) -> Optional[tf.DType]:\n '''Maps python types to TensorFlow dtypes.'''\n dtype = None\n if python_type == float:\n dtype = tf.float32\n elif python_type == int:\n dtype = tf.int32\n elif python_type == bool:\n dtype = tf.bool\n return dtype", "response": "Maps python types to TensorFlow dtypes."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_dependency_type(_type):\n if _type == DependencyTypes.AFTER:\n return 'after'\n elif _type == DependencyTypes.AFTER_ANY:\n return 'afterany'\n elif _type == DependencyTypes.AFTER_CORR:\n return 'aftercorr'\n elif _type == DependencyTypes.AFTER_NOT_OK:\n return 'afternotok'\n elif _type == DependencyTypes.AFTER_OK:\n return 'afterok'\n else:\n return None", "response": "Returns the string for the dependency type for SlurmPrinter."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_header():\n username, userid, uname = get_user_information()\n\n header = '''\\\n# This Slurm batch script was generated\n# By user: %s (%s)\n# On host: %s\n# At date: %s\n# Using: Pyqueue v%s\n\n''' % (username, userid, uname, datetime.now().strftime('%a. %B %w %X %Y'), __version__)\n\n return header", "response": "Makes the header section for the scripts\n "} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ngenerate a job submission script from a job object.", "response": "def generate(self, job):\n \"\"\"\n Generates a job submission script from a job object\n\n :param job: An instance of JobInterface\n :type job: pyqueue.job.JobInterface\n \"\"\"\n\n options = job.get_options().copy()\n job_name = options.pop('name', None)\n job_account = options.pop('account', None)\n job_walltime = options.pop('walltime', None)\n job_mem_per_cpu = options.pop('mem_per_cpu', None)\n job_memory = options.pop('memory', None)\n job_working_directory = options.pop('working_directory', None)\n job_error_path = options.pop('error_path', None)\n job_output_path = options.pop('output_path', None)\n job_dependency = options.pop('depending', None)\n job_shell = options.pop('shell', '/bin/bash')\n job_custom_options = options.pop('__custom__', [])\n\n directives_lines = []\n\n if job_name is not None:\n directives_lines.append('--job-name=%s' % job_name)\n\n if job_account is not None:\n directives_lines.append('--account=%s' % job_account)\n\n if job_working_directory is not None:\n directives_lines.append('--workdir=%s' % job_working_directory)\n\n if job_error_path is not None:\n directives_lines.append('--error=%s' % job_error_path)\n\n if job_output_path is not None:\n directives_lines.append('--output=%s' % job_output_path)\n\n if job_walltime is not None:\n directives_lines.append('--time=%s' %\n strfdelta(job_walltime, '%H:%M:%S'))\n\n if job_mem_per_cpu is not None:\n directives_lines.append('--mem-per-cpu=%d' % job_mem_per_cpu)\n\n if job_memory is not None:\n directives_lines.append('--mem=%d' % job_memory)\n\n if job_dependency is not None:\n master = job_dependency['job']\n dependency_type = SlurmPrinter.get_dependency_type(\n job_dependency['dependency_type']\n )\n job_id = master.get_id() if isinstance(master, JobInterface) else master\n\n directives_lines.append(\n '--dependency=%s:%s' %\n (dependency_type, job_id)\n )\n\n for custom_option in job_custom_options:\n directives_lines.append(custom_option)\n\n directives = '\\n'.join([\n '#SBATCH %s' % directive for directive in directives_lines\n ])\n\n commands = '\\n'.join([\n '\\n'.join(command_container.get_commands()) for command_container in job.get_commands()\n ])\n\n script = '#!%s\\n' % job_shell\n script += SlurmPrinter.get_header()\n script += directives\n script += '\\n\\n'\n script += commands\n\n return script"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsubmit a given job to the cluster.", "response": "def submit(self, job):\n \"\"\"\n Submits a given job\n\n :param job: The job to submit\n :type job: pyqueue.job.JobInterface\n \"\"\"\n from subprocess import Popen, PIPE\n script = self._printer.generate(job)\n process = Popen('sbatch', stdout=PIPE, stdin=PIPE, stderr=PIPE)\n stdout, sterr = process.communicate(input=script)\n process.stdin.close()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsubmit a given job to the remote host.", "response": "def submit(self, job):\n \"\"\"\n Submits a given job\n\n :param job: The job to submit\n :type job: pyqueue.job.JobInterface\n \"\"\"\n script = self._printer.generate(job)\n stdin, stdout, stderr = self._ssh.exec_command('sbatch')\n stdin.write(script)\n stdin.flush()\n stdin.channel.shutdown_write()\n return stdout.read()"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_info_mpris2(name):\n # qdbus org.mpris.MediaPlayer2.<name> /org/mpris/MediaPlayer2\\\n # org.freedesktop.DBus.Properties.Get org.mpris.MediaPlayer2.Player Metadat\n bus_name = 'org.mpris.MediaPlayer2.' + name\n path = '/org/mpris/MediaPlayer2'\n interface = 'org.mpris.MediaPlayer2.Player'\n address = DBusAddress(path, bus_name=bus_name, interface=interface)\n msg = Properties(address).get('Metadata')\n connection = connect_and_authenticate()\n response = connection.send_and_get_reply(msg)\n metadata = dict(response[0][1])\n keys = ['album', 'title', 'artist', 'albumartist']\n\n info = {}\n metadata = {k: v for k, v in metadata.items() if 'xesam:' in k}\n for key, value in metadata.items():\n name = key.split(':')[1].lower()\n value = value[1]\n if name not in keys or name in info:\n continue\n if isinstance(value, list):\n value = value[0]\n info[name] = value\n\n if 'albumartist' in info:\n info['artist'] = info['albumartist']\n del info['albumartist']\n\n return Song(**info)", "response": "Get the current playing song from an mpris2 compliant player."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_current_clementine():\n # mpris_version 2\n try:\n return get_info_mpris2('clementine')\n except DBusErrorResponse:\n bus_name = 'org.mpris.clementine'\n path = '/Player'\n interface = 'org.freedesktop.MediaPlayer'\n return dbus_get_metadata(path, bus_name, interface)", "response": "Get the current song from clementine."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_current_cmus():\n result = subprocess.run('cmus-remote -Q'.split(' '), check=True,\n stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)\n info = {}\n for line in result.stdout.decode().split('\\n'):\n line = line.split(' ')\n if line[0] != 'tag':\n continue\n key = line[1]\n if key in ['album', 'title', 'artist', 'albumartist'] and\\\n key not in info:\n info[key] = ' '.join(line[2:])\n\n if 'albumartist' in info:\n info['artist'] = info['albumartist']\n del info['albumartist']\n\n return Song(**info)", "response": "Get the current song from cmus."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a new Song object from a file.", "response": "def from_filename(cls, filename):\n \"\"\"\n Class constructor using the path to the corresponding mp3 file. The\n metadata will be read from this file to create the song object, so it\n must at least contain valid ID3 tags for artist and title.\n \"\"\"\n if not filename:\n logger.error('No filename specified')\n return None\n\n if not os.path.exists(filename):\n logger.error(\"Err: File '%s' does not exist\", filename)\n return None\n\n if os.path.isdir(filename):\n logger.error(\"Err: File '%s' is a directory\", filename)\n return None\n\n try:\n audiofile = eyed3.load(filename)\n except Exception as error:\n print(type(error), error)\n return None\n\n # Sometimes eyed3 may return a null object and not raise any exceptions\n if audiofile is None:\n return None\n\n tags = audiofile.tag\n album = tags.album\n title = tags.title\n lyrics = ''.join([l.text for l in tags.lyrics])\n artist = tags.album_artist\n if not artist:\n artist = tags.artist\n\n song = cls(artist, title, album, lyrics)\n song.filename = filename\n return song"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef from_string(cls, name, separator='-', reverse=False):\n recv = [t.strip() for t in name.split(separator)]\n if len(recv) < 2:\n logger.error('Wrong format!')\n return None\n\n if reverse:\n title = recv[0]\n artist = ''.join(recv[1:])\n else:\n artist = recv[0]\n title = ''.join(recv[1:])\n\n if not artist or not title:\n logger.error('Wrong format!')\n return None\n\n song = cls(artist, title)\n return song", "response": "This method is used to construct a new Song object from a string with the artist and title."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fetch_album_name(self):\n response = get_lastfm('track.getInfo', artist=self.artist,\n track=self.title)\n if response:\n try:\n self.album = response['track']['album']['title']\n logger.debug('Found album %s from lastfm', self.album)\n except Exception:\n logger.warning('Could not fetch album name for %s', self)\n else:\n logger.warning('Could not fetch album name for %s', self)", "response": "Fetch the name of the album from lastfm."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the closest monday to the provided date.", "response": "def _get_closest_week(self, metric_date):\n \"\"\"\n Gets the closest monday to the date provided.\n \"\"\"\n #find the offset to the closest monday\n days_after_monday = metric_date.isoweekday() - 1\n\n return metric_date - datetime.timedelta(days=days_after_monday)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_daily_date_range(self, metric_date, delta):\n dates = [metric_date]\n start_date = metric_date\n end_date = metric_date + delta\n\n while start_date.month < end_date.month or start_date.year < end_date.year:\n days_in_month = calendar.monthrange(start_date.year, start_date.month)[1]\n #shift along to the next month as one of the months we will have to see. We don't care that the exact date\n #is the 1st in each subsequent date range as we only care about the year and the month\n start_date = start_date + datetime.timedelta(days=days_in_month - start_date.day + 1)\n dates.append(start_date)\n\n return dates", "response": "Get the range of days that we need to scan redis."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting the range of years that we need to get metrics from redis.", "response": "def _get_weekly_date_range(self, metric_date, delta):\n \"\"\"\n Gets the range of years that we need to use as keys to get metrics from redis.\n \"\"\"\n dates = [metric_date]\n end_date = metric_date + delta\n #Figure out how many years our metric range spans\n spanning_years = end_date.year - metric_date.year\n for i in range(spanning_years):\n #for the weekly keys, we only care about the year\n dates.append(\n datetime.date(\n year=metric_date.year + (i + 1), month=1, day=1))\n return dates"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndelete all sandsnake related data from redis.", "response": "def clear_all(self):\n \"\"\"\n Deletes all ``sandsnake`` related data from redis.\n\n .. warning::\n\n Very expensive and destructive operation. Use with causion\n \"\"\"\n keys = self._analytics_backend.keys()\n\n for key in itertools.chain(*keys):\n with self._analytics_backend.map() as conn:\n if key.startswith(self._prefix):\n conn.delete(key)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ntrack a count of the object with the given unique identifier.", "response": "def track_count(self, unique_identifier, metric, inc_amt=1, **kwargs):\n \"\"\"\n Tracks a metric just by count. If you track a metric this way, you won't be able\n to query the metric by day, week or month.\n\n :param unique_identifier: Unique string indetifying the object this metric is for\n :param metric: A unique name for the metric you want to track\n :param inc_amt: The amount you want to increment the ``metric`` for the ``unique_identifier``\n :return: ``True`` if successful ``False`` otherwise\n \"\"\"\n return self._analytics_backend.incr(self._prefix + \":\" + \"analy:%s:count:%s\" % (unique_identifier, metric), inc_amt)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef track_metric(self, unique_identifier, metric, date=None, inc_amt=1, **kwargs):\n metric = [metric] if isinstance(metric, basestring) else metric\n unique_identifier = [unique_identifier] if not isinstance(unique_identifier, (types.ListType, types.TupleType, types.GeneratorType,)) else unique_identifier\n results = []\n if date is None:\n date = datetime.date.today()\n with self._analytics_backend.map() as conn:\n for uid in unique_identifier:\n\n hash_key_daily = self._get_daily_metric_key(uid, date)\n closest_monday = self._get_closest_week(date)\n hash_key_weekly = self._get_weekly_metric_key(uid, date)\n\n for single_metric in metric:\n daily_metric_name = self._get_daily_metric_name(single_metric, date)\n weekly_metric_name = self._get_weekly_metric_name(single_metric, closest_monday)\n monthly_metric_name = self._get_monthly_metric_name(single_metric, date)\n\n results.append(\n [\n conn.hincrby(hash_key_daily, daily_metric_name, inc_amt),\n conn.hincrby(hash_key_weekly, weekly_metric_name, inc_amt),\n conn.hincrby(hash_key_weekly, monthly_metric_name, inc_amt),\n conn.incr(self._prefix + \":\" + \"analy:%s:count:%s\" % (uid, single_metric), inc_amt)\n ]\n )\n\n return results", "response": "Track a metric for a specific unique identifier for a specific date."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_metric_by_day(self, unique_identifier, metric, from_date, limit=30, **kwargs):\n conn = kwargs.get(\"connection\", None)\n date_generator = (from_date + datetime.timedelta(days=i) for i in itertools.count())\n metric_key_date_range = self._get_daily_date_range(from_date, datetime.timedelta(days=limit))\n #generate a list of mondays in between the start date and the end date\n series = list(itertools.islice(date_generator, limit))\n\n metric_keys = [self._get_daily_metric_name(metric, daily_date) for daily_date in series]\n\n metric_func = lambda conn: [conn.hmget(self._get_daily_metric_key(unique_identifier, \\\n metric_key_date), metric_keys) for metric_key_date in metric_key_date_range]\n\n if conn is not None:\n results = metric_func(conn)\n else:\n with self._analytics_backend.map() as conn:\n results = metric_func(conn)\n series, results = self._parse_and_process_metrics(series, results)\n\n return series, results", "response": "Get the metric for the unique_identifier segmented by day from from_date."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the metric for the unique_identifier segmented by week starting from from_date.", "response": "def get_metric_by_week(self, unique_identifier, metric, from_date, limit=10, **kwargs):\n \"\"\"\n Returns the ``metric`` for ``unique_identifier`` segmented by week\n starting from``from_date``\n\n :param unique_identifier: Unique string indetifying the object this metric is for\n :param metric: A unique name for the metric you want to track\n :param from_date: A python date object\n :param limit: The total number of weeks to retrive starting from ``from_date``\n \"\"\"\n conn = kwargs.get(\"connection\", None)\n closest_monday_from_date = self._get_closest_week(from_date)\n metric_key_date_range = self._get_weekly_date_range(closest_monday_from_date, datetime.timedelta(weeks=limit))\n\n date_generator = (closest_monday_from_date + datetime.timedelta(days=i) for i in itertools.count(step=7))\n #generate a list of mondays in between the start date and the end date\n series = list(itertools.islice(date_generator, limit))\n\n metric_keys = [self._get_weekly_metric_name(metric, monday_date) for monday_date in series]\n\n metric_func = lambda conn: [conn.hmget(self._get_weekly_metric_key(unique_identifier, \\\n metric_key_date), metric_keys) for metric_key_date in metric_key_date_range]\n\n if conn is not None:\n results = metric_func(conn)\n else:\n with self._analytics_backend.map() as conn:\n results = metric_func(conn)\n series, results = self._parse_and_process_metrics(series, results)\n\n return series, results"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_metric_by_month(self, unique_identifier, metric, from_date, limit=10, **kwargs):\n conn = kwargs.get(\"connection\", None)\n first_of_month = datetime.date(year=from_date.year, month=from_date.month, day=1)\n metric_key_date_range = self._get_weekly_date_range(\n first_of_month, relativedelta(months=limit))\n\n date_generator = (first_of_month + relativedelta(months=i) for i in itertools.count())\n #generate a list of first_of_month's in between the start date and the end date\n series = list(itertools.islice(date_generator, limit))\n\n metric_keys = [self._get_monthly_metric_name(metric, month_date) for month_date in series]\n\n metric_func = lambda conn: [conn.hmget(\n self._get_weekly_metric_key(\n unique_identifier, metric_key_date), metric_keys) for metric_key_date in metric_key_date_range]\n\n if conn is not None:\n results = metric_func(conn)\n else:\n with self._analytics_backend.map() as conn:\n results = metric_func(conn)\n series, results = self._parse_and_process_metrics(series, results)\n\n return series, results", "response": "This method returns the metric for the unique_identifier segmented by month from_date. It will retrieve the metrics for the unique_identifier starting from from_date."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_metrics(self, metric_identifiers, from_date, limit=10, group_by=\"week\", **kwargs):\n results = []\n #validation of types:\n allowed_types = {\n \"day\": self.get_metric_by_day,\n \"week\": self.get_metric_by_week,\n \"month\": self.get_metric_by_month,\n }\n if group_by.lower() not in allowed_types:\n raise Exception(\"Allowed values for group_by are day, week or month.\")\n\n group_by_func = allowed_types[group_by.lower()]\n #pass a connection object so we can pipeline as much as possible\n with self._analytics_backend.map() as conn:\n for unique_identifier, metric in metric_identifiers:\n results.append(group_by_func(unique_identifier, metric, from_date, limit=limit, connection=conn))\n\n #we have to merge all the metric results afterwards because we are using a custom context processor\n return [\n self._parse_and_process_metrics(series, list_of_metrics) for\n series, list_of_metrics in results]", "response": "Retrieves multiple metrics as efficiently as possible."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget the count for the metric for the unique_identifier.", "response": "def get_count(self, unique_identifier, metric, start_date=None, end_date=None, **kwargs):\n \"\"\"\n Gets the count for the ``metric`` for ``unique_identifier``. You can specify a ``start_date``\n and an ``end_date``, to only get metrics within that time range.\n\n :param unique_identifier: Unique string indetifying the object this metric is for\n :param metric: A unique name for the metric you want to track\n :param start_date: Get the specified metrics after this date\n :param end_date: Get the sepcified metrics before this date\n :return: The count for the metric, 0 otherwise\n \"\"\"\n result = None\n if start_date and end_date:\n start_date, end_date = (start_date, end_date,) if start_date < end_date else (end_date, start_date,)\n\n start_date = start_date if hasattr(start_date, 'date') else datetime.datetime.combine(start_date, datetime.time())\n end_date = end_date if hasattr(end_date, 'date') else datetime.datetime.combine(end_date, datetime.time())\n\n monthly_metrics_dates = list(rrule.rrule(rrule.MONTHLY, dtstart=start_date, bymonthday=1, until=end_date))\n\n #We can sorta optimize this by getting most of the data by month\n if len(monthly_metrics_dates) >= 3:\n\n with self._analytics_backend.map() as conn:\n monthly_metric_series, monthly_metric_results, starting_metric_series, starting_metric_results, ending_metric_series, ending_metric_results = self._get_counts(\n conn, metric, unique_identifier, monthly_metrics_dates, start_date, end_date)\n\n monthly_metric_series, monthly_metric_results = self._parse_and_process_metrics(monthly_metric_series, monthly_metric_results)\n starting_metric_series, starting_metric_results = self._parse_and_process_metrics(starting_metric_series, starting_metric_results)\n ending_metric_series, ending_metric_results = self._parse_and_process_metrics(ending_metric_series, ending_metric_results)\n\n result = sum(monthly_metric_results.values()) + sum(starting_metric_results.values()) + sum(ending_metric_results.values())\n else:\n diff = end_date - start_date\n metric_results = self.get_metric_by_day(unique_identifier, metric, start_date, limit=diff.days + 1)\n result = sum(metric_results[1].values())\n\n else:\n try:\n result = int(self._analytics_backend.get(self._prefix + \":\" + \"analy:%s:count:%s\" % (unique_identifier, metric,)))\n except TypeError:\n result = 0\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nretrieves a multiple metrics as efficiently as possible.", "response": "def get_counts(self, metric_identifiers, **kwargs):\n \"\"\"\n Retrieves a multiple metrics as efficiently as possible.\n\n :param metric_identifiers: a list of tuples of the form `(unique_identifier, metric_name`) identifying which metrics to retrieve.\n For example [('user:1', 'people_invited',), ('user:2', 'people_invited',), ('user:1', 'comments_posted',), ('user:2', 'comments_posted',)]\n \"\"\"\n parsed_results = []\n results = [\n self.get_count(unique_identifier, metric, **kwargs) for\n unique_identifier, metric in metric_identifiers]\n\n for result in results:\n try:\n parsed_result = int(result)\n except TypeError:\n parsed_result = 0\n\n parsed_results.append(parsed_result)\n\n return parsed_results"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsets the count for the metric for the unique_identifier for the given date.", "response": "def set_metric_by_day(self, unique_identifier, metric, date, count, sync_agg=True, update_counter=True):\n \"\"\"\n Sets the count for the ``metric`` for ``unique_identifier``.\n You must specify a ``date`` for the ``count`` to be set on. Useful for resetting a metric count to 0 or decrementing a metric.\n\n The redis backend supports lists for both ``unique_identifier`` and ``metric`` allowing for the setting of \n multiple metrics for multiple unique_identifiers efficiently. Not all backends may support this.\n\n :param unique_identifier: Unique string indetifying the object this metric is for\n :param metric: A unique name for the metric you want to track\n :param date: Sets the specified metrics for this date\n :param count: Sets the sepcified metrics to value of count\n :param sync_agg: Boolean used to determine if week and month metrics should be updated\n :param update_counter: Boolean used to determine if overall counter should be updated\n \"\"\"\n metric = [metric] if isinstance(metric, basestring) else metric\n unique_identifier = [unique_identifier] if not isinstance(unique_identifier, (types.ListType, types.TupleType, types.GeneratorType,)) else unique_identifier\n results = []\n with self._analytics_backend.map() as conn:\n for uid in unique_identifier:\n hash_key_daily = self._get_daily_metric_key(uid, date)\n\n for single_metric in metric:\n daily_metric_name = self._get_daily_metric_name(single_metric, date)\n\n if update_counter: # updates overall counter for metric\n overall_count = self.get_count(uid, single_metric)\n day, daily_count = self.get_metric_by_day(uid, single_metric, date, 1)[1].popitem()\n self._analytics_backend.set(self._prefix + \":\" + \"analy:%s:count:%s\" % (uid, single_metric), overall_count + (count - daily_count))\n\n results.append([conn.hset(hash_key_daily, daily_metric_name, count)])\n\n if sync_agg:\n self.sync_agg_metric(unique_identifier, metric, date, date)\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sync_agg_metric(self, unique_identifier, metric, start_date, end_date):\n self.sync_week_metric(unique_identifier, metric, start_date, end_date)\n self.sync_month_metric(unique_identifier, metric, start_date, end_date)", "response": "Synchronizes the metric for the given unique_identifier with the given start and end dates."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sync_week_metric(self, unique_identifier, metric, start_date, end_date):\n metric = [metric] if isinstance(metric, basestring) else metric\n unique_identifier = [unique_identifier] if not isinstance(unique_identifier, (types.ListType, types.TupleType, types.GeneratorType,)) else unique_identifier\n closest_monday_from_date = self._get_closest_week(start_date)\n num_weeks = self._num_weeks(start_date, end_date)\n metric_key_date_range = self._get_weekly_date_range(closest_monday_from_date, datetime.timedelta(weeks=num_weeks))\n\n week_date_generator = (closest_monday_from_date + datetime.timedelta(days=i) for i in itertools.count(step=7))\n #generate a list of mondays in between the start date and the end date\n weeks_to_update = list(itertools.islice(week_date_generator, num_weeks))\n for uid in unique_identifier:\n for single_metric in metric:\n for week in weeks_to_update:\n _, series_results = self.get_metric_by_day(uid, single_metric, from_date=week, limit=7)\n week_counter = sum([value for key, value in series_results.items()])\n\n hash_key_weekly = self._get_weekly_metric_key(uid, week)\n weekly_metric_name = self._get_weekly_metric_name(single_metric, week)\n with self._analytics_backend.map() as conn:\n conn.hset(hash_key_weekly, weekly_metric_name, week_counter)", "response": "Synchronizes the metric for the given unique_identifier for the given start and end date."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef sync_month_metric(self, unique_identifier, metric, start_date, end_date):\n metric = [metric] if isinstance(metric, basestring) else metric\n unique_identifier = [unique_identifier] if not isinstance(unique_identifier, (types.ListType, types.TupleType, types.GeneratorType,)) else unique_identifier\n num_months = self._num_months(start_date, end_date)\n first_of_month = datetime.date(year=start_date.year, month=start_date.month, day=1)\n metric_key_date_range = self._get_weekly_date_range(\n first_of_month, relativedelta(months=num_months))\n\n month_date_generator = (first_of_month + relativedelta(months=i) for i in itertools.count())\n #generate a list of first_of_month's in between the start date and the end date\n months_to_update = list(itertools.islice(month_date_generator, num_months))\n for uid in unique_identifier:\n for single_metric in metric:\n for month in months_to_update:\n _, series_results = self.get_metric_by_day(uid, single_metric, from_date=month, limit=monthrange(month.year, month.month)[1])\n month_counter = sum([value for key, value in series_results.items()])\n\n hash_key_monthly = self._get_weekly_metric_key(uid, month)\n monthly_metric_name = self._get_monthly_metric_name(single_metric, month)\n with self._analytics_backend.map() as conn:\n conn.hset(hash_key_monthly, monthly_metric_name, month_counter)", "response": "Synchronizes the given metric for the given unique identifier for the given start and end date."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef is_full_mxid(user_string):\n if not user_string[0] == \"@\":\n return False\n parts = user_string[1:].split(\":\")\n localpart_chars = ascii_lowercase + digits + \"._-=\"\n if not (len(parts) == 2 and all([i in localpart_chars for i in parts[0]])):\n return False\n return True", "response": "Returns True if a string is a valid mxid."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nhelps object methods handle MatrixRequestError. Args: method(function): Object method to be wrapped Method's object must have _handle_request_exception method that deals with specific status codes and errcodes.", "response": "def intent(method):\n \"\"\"Helps object methods handle MatrixRequestError.\n\n Args:\n method(function): Object method to be wrapped\n\n Method's object must have _handle_request_exception method that deals with\n specific status codes and errcodes.\n \"\"\"\n\n def wrapper(self, *args, **kwargs):\n try:\n return method(self, *args, **kwargs)\n except exceptions.MatrixError as e:\n if isinstance(e.original_exception,\n matrix_client.errors.MatrixRequestError):\n self._handle_request_exception(e)\n # May still throw exception for other reasons; not handled\n return method(self, *args, **kwargs)\n else:\n raise e\n\n return wrapper"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfind all the variables specified in a format string. This returns a list of all the variables that are not in the format string.", "response": "def get_variables(self) -> Set[str]:\n \"\"\"Find all the variables specified in a format string.\n\n This returns a list of all the different variables specified in a format string,\n that is the variables inside the braces.\n\n \"\"\"\n variables = set()\n for cmd in self._cmd:\n for var in self.__formatter.parse(cmd):\n logger.debug(\"Checking variable: %s\", var)\n # creates and requires are special class values\n if var[1] is not None and var[1] not in [\"creates\", \"requires\"]:\n variables.add(var[1])\n return variables"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef as_bash_array(self) -> str:\n return_string = \"( \\\\\\n\"\n for command in self:\n return_string += '\"' + str(command) + '\" \\\\\\n'\n return_string += \")\"\n return return_string", "response": "Return a representation of the job as a bash array."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses a docstring into a dict", "response": "def parse_doc(doc):\n \"\"\"\n Parse docstrings to dict, it should look like:\n key: value\n \"\"\"\n if not doc:\n return {}\n out = {}\n for s in doc.split('\\n'):\n s = s.strip().split(':', maxsplit=1)\n if len(s) == 2:\n out[s[0]] = s[1]\n return out"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef combine_dictionaries(dicts: List[Dict[str, Any]]) -> Dict[str, Any]:\n return dict(ChainMap(*dicts))", "response": "Combine a list of dictionaries into a single dictionary."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef iterator_zip(variables: VarType, parent: str = None) -> Iterable[VarMatrix]:\n\n logger.debug(\"Yielding from zip iterator\")\n if isinstance(variables, list):\n for item in variables:\n yield list(variable_matrix(item, parent, \"zip\"))\n else:\n yield list(variable_matrix(variables, parent, \"zip\"))", "response": "Returns an iterator over the set of variables."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns an iterator over the product of the given set of variables.", "response": "def iterator_product(variables: VarType, parent: str = None) -> Iterable[VarMatrix]:\n \"\"\"Apply the product operator to a set of variables.\n\n This uses the python itertools.product iterator to combine multiple variables\n such that all possible combinations are generated. This is the default iterator\n however this is a method of manually specifying the option.\n\n Args:\n variables: The variables object\n parent: Unused\n\n \"\"\"\n logger.debug(\"Yielding from product iterator\")\n if isinstance(variables, list):\n raise ValueError(\n f\"Product only takes mappings of values, got {variables} of type {type(variables)}\"\n )\n\n yield list(variable_matrix(variables, parent, \"product\"))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef iterator_chain(variables: VarType, parent: str = None) -> Iterable[VarMatrix]:\n logger.debug(\"Yielding from append iterator\")\n if not isinstance(variables, list):\n raise ValueError(\n f\"Append keyword only takes a list of arguments, got {variables} of type {type(variables)}\"\n )\n\n # Create a single list containing all the values\n yield list(\n chain.from_iterable(\n variable_matrix(item, parent, \"product\") for item in variables\n )\n )", "response": "This function returns an iterator over the values generated for each element of an array and puts all the values generated for each element in the list into a single list of values."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef iterator_arange(variables: VarType, parent: str) -> Iterable[VarMatrix]:\n assert parent is not None\n if isinstance(variables, (int, float)):\n yield [{parent: i} for i in np.arange(variables)]\n\n elif isinstance(variables, dict):\n if variables.get(\"stop\"):\n yield [{parent: i} for i in arange(**variables)]\n else:\n raise ValueError(f\"Stop is a required keyword for the arange iterator.\")\n\n else:\n raise ValueError(\n f\"The arange keyword only takes a dict as arguments, got {variables} of type {type(variables)}\"\n )", "response": "Create a list of values using the numpy. arange function."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef iterator_cycle(variables: VarType, parent: str) -> Iterable[VarMatrix]:\n if isinstance(variables, dict):\n if variables.get(\"times\"):\n times = int(variables[\"times\"])\n del variables[\"times\"]\n\n yield list(variable_matrix(variables, parent, \"product\")) * times\n\n else:\n raise ValueError(f\"times is a required keyword for the repeat iterator.\")\n else:\n raise ValueError(\n f\"The repeat operator only takes a dict as arguments, got {variables} of type {type(variables)}\"\n )", "response": "Cycle through a list of values a specified number of times."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngenerates a list of the appropriate combinations of variables in the base tree tree.", "response": "def variable_matrix(\n variables: VarType, parent: str = None, iterator: str = \"product\"\n) -> Iterable[Dict[str, YamlValue]]:\n \"\"\"Process the variables into a list of the appropriate combinations.\n\n This function performs recursive processing of the input variables, creating an\n iterator which has all the combinations of variables specified in the input.\n\n \"\"\"\n _iters: Dict[str, Callable] = {\"product\": product, \"zip\": zip}\n _special_keys: Dict[str, Callable[[VarType, Any], Iterable[VarMatrix]]] = {\n \"zip\": iterator_zip,\n \"product\": iterator_product,\n \"arange\": iterator_arange,\n \"chain\": iterator_chain,\n \"append\": iterator_chain,\n \"cycle\": iterator_cycle,\n \"repeat\": iterator_cycle,\n }\n\n if isinstance(variables, dict):\n key_vars: List[List[Dict[str, YamlValue]]] = []\n\n # Handling of specialised iterators\n for key, function in _special_keys.items():\n if variables.get(key):\n item = variables[key]\n assert item is not None\n for val in function(item, parent):\n key_vars.append(val)\n\n del variables[key]\n\n for key, value in variables.items():\n key_vars.append(list(variable_matrix(value, key, iterator)))\n\n logger.debug(\"key vars: %s\", key_vars)\n\n # Iterate through all possible products generating a dictionary\n for i in _iters[iterator](*key_vars):\n logger.debug(\"dicts: %s\", i)\n yield combine_dictionaries(i)\n\n # Iterate through a list of values\n elif isinstance(variables, list):\n for item in variables:\n yield from variable_matrix(item, parent, iterator)\n\n # Stopping condition -> we have either a single value from a list\n # or a value had only one item\n else:\n assert parent is not None\n yield {parent: variables}"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nremove duplicate entries in a list retaining order.", "response": "def uniqueify(my_list: Any) -> List[Any]:\n \"\"\"Remove duplicate entries in a list retaining order.\"\"\"\n if sys.version_info >= (3, 6):\n # An implementation specific detail of py3.6 is the retention of order\n # within a dictionary. In py3.7 this becomes the documented behaviour.\n return list(dict.fromkeys(my_list))\n\n # Slower method of order preserving unique list in older python versions\n seen = set()\n return [x for x in my_list if x not in seen and not seen.add(x)]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef process_command(command: CommandInput, matrix: VarMatrix) -> List[Command]:\n assert command is not None\n if isinstance(command, str):\n command_list = [Command(command, variables=variables) for variables in matrix]\n elif isinstance(command, list):\n command_list = [Command(command, variables=variables) for variables in matrix]\n else:\n if command.get(\"command\") is not None:\n cmd = command.get(\"command\")\n else:\n cmd = command.get(\"cmd\")\n creates = str(command.get(\"creates\", \"\"))\n requires = str(command.get(\"requires\", \"\"))\n\n assert isinstance(cmd, (list, str))\n command_list = [\n Command(cmd, variables, creates, requires) for variables in matrix\n ]\n return uniqueify(command_list)", "response": "Generates all combinations of commands given a variable matrix."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef read_file(filename: PathLike = \"experiment.yml\") -> Dict[str, Any]:\n logger.debug(\"Input file: %s\", filename)\n\n with open(filename, \"r\") as stream:\n structure = yaml.safe_load(stream)\n return structure", "response": "Read and parse yaml file."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef run_bash_jobs(\n jobs: Iterator[Job], directory: PathLike = Path.cwd(), dry_run: bool = False\n) -> None:\n \"\"\"Submit commands to the bash shell.\n\n This function runs the commands iteratively but handles errors in the\n same way as with the pbs_commands function. A command will run for all\n combinations of variables in the variable matrix, however if any one of\n those commands fails then the next command will not run.\n\n \"\"\"\n logger.debug(\"Running commands in bash shell\")\n # iterate through command groups\n for job in jobs:\n # Check shell exists\n if shutil.which(job.shell) is None:\n raise ProcessLookupError(f\"The shell '{job.shell}' was not found.\")\n\n failed = False\n for command in job:\n for cmd in command:\n logger.info(cmd)\n if dry_run:\n print(f\"{job.shell} -c '{cmd}'\")\n else:\n result = subprocess.run(\n [job.shell, \"-c\", f\"{cmd}\"], cwd=str(directory)\n )\n if result.returncode != 0:\n failed = True\n logger.error(\"Command failed: %s\", command)\n break\n if failed:\n logger.error(\"A command failed, not continuing further.\")\n return", "response": "Submit commands to the bash shell."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef run_scheduler_jobs(\n scheduler: str,\n jobs: Iterator[Job],\n directory: PathLike = Path.cwd(),\n basename: str = \"experi\",\n dry_run: bool = False,\n) -> None:\n \"\"\"Submit a series of commands to a batch scheduler.\n\n This takes a list of strings which are the contents of the pbs files, writes the\n files to disk and submits the job to the scheduler. Files which match the pattern of\n the resulting files <basename>_<index>.pbs are deleted before writing the new files.\n\n To ensure that commands run consecutively the aditional requirement to the run\n script `-W depend=afterok:<prev_jobid>` is added. This allows for all the components\n of the experiment to be conducted in a single script.\n\n Note: Having this function submit jobs requires that the command `qsub` exists,\n implying that a job scheduler is installed.\n\n \"\"\"\n submit_job = True\n logger.debug(\"Creating commands in %s files.\", scheduler)\n\n # Check scheduler submit command exists\n if scheduler == \"pbs\":\n submit_executable = \"qsub\"\n elif scheduler == \"slurm\":\n submit_executable = \"sbatch\"\n else:\n raise ValueError(\"scheduler can only take values ['pbs', 'slurm']\")\n\n if shutil.which(submit_executable) is None:\n logger.warning(\n \"The `%s` command is not found.\"\n \"Skipping job submission and just generating files\",\n submit_executable,\n )\n submit_job = False\n\n # Ensure directory is a Path\n directory = Path(directory)\n\n # remove existing files\n for fname in directory.glob(basename + f\"*.{scheduler}\"):\n print(\"Removing {}\".format(fname))\n os.remove(str(fname))\n\n # Write new files and generate commands\n prev_jobids: List[str] = []\n for index, job in enumerate(jobs):\n # Generate scheduler file\n content = create_scheduler_file(scheduler, job)\n logger.debug(\"File contents:\\n%s\", content)\n # Write file to disk\n fname = Path(directory / \"{}_{:02d}.{}\".format(basename, index, scheduler))\n with fname.open(\"w\") as dst:\n dst.write(content)\n\n if submit_job or dry_run:\n # Construct command\n submit_cmd = [submit_executable]\n\n if prev_jobids:\n # Continue to append all previous jobs to submit_cmd so subsequent jobs die along\n # with the first.\n afterok = f\"afterok:{':'.join(prev_jobids)}\"\n if scheduler == \"pbs\":\n submit_cmd += [\"-W\", f\"depend={afterok}\"]\n elif scheduler == \"slurm\":\n submit_cmd += [\"--dependency\", afterok]\n\n # actually run the command\n logger.info(str(submit_cmd))\n try:\n if dry_run:\n print(f\"{submit_cmd} {fname.name}\")\n prev_jobids.append(\"dry_run\")\n else:\n cmd_res = subprocess.check_output(\n submit_cmd + [fname.name], cwd=str(directory)\n )\n prev_jobids.append(cmd_res.decode().strip())\n except subprocess.CalledProcessError:\n logger.error(\"Submitting job to the queue failed.\")\n break", "response": "Submit a series of jobs to a batch scheduler."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndetermining the scheduler to use to run the jobs.", "response": "def determine_scheduler(\n scheduler: Optional[str], experiment_definition: Dict[str, YamlValue]\n) -> str:\n \"\"\"Determine the scheduler to use to run the jobs.\"\"\"\n\n # Scheduler value from command line has first priority\n if scheduler is not None:\n if scheduler in [\"shell\", \"pbs\", \"slurm\"]:\n return scheduler\n raise ValueError(\n \"Argument scheduler only supports input values of ['shell', 'pbs', 'slurm']\"\n )\n\n # Next priority goes to the experiment.yml file\n if experiment_definition.get(\"pbs\"):\n return \"pbs\"\n if experiment_definition.get(\"slurm\"):\n return \"slurm\"\n if experiment_definition.get(\"shell\"):\n return \"shell\"\n\n # Final priority goes to the auto-discovery\n if shutil.which(\"pbs\") is not None:\n return \"pbs\"\n if shutil.which(\"slurm\") is not None:\n return \"slurm\"\n\n # Default if nothing else is found goes to shell\n return \"shell\""} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nmaking sure that the incoming data complies with the class type we expect.", "response": "def safe_type(self, data, tree):\n \"\"\"\n Make sure that the incoming data complies with the class type we\n are expecting it to be. In this case, classes that inherit from this\n base class expect data to be of type ``list``.\n \"\"\"\n if not isinstance(data, list):\n name = self.__class__.__name__\n msg = \"did not pass validation against callable: %s\" % name\n reason = 'expected a list but got %s' % safe_repr(data)\n raise Invalid(self.schema, tree, reason=reason, pair='value', msg=msg)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_context(context):\n new_context = {\n 'model': context['model'],\n 'session': context['session'],\n 'user': context.get('user'),\n 'ignore_auth': context.get('ignore_auth', False),\n 'use_cache': context.get('use_cache', False),\n }\n\n if 'validate' in context:\n new_context['validate'] = context['validate']\n\n return new_context", "response": "An internal context generator."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns unique dict for each uwsgi request. note: won't work on non-uwsgi cases", "response": "def request_uniq(func):\n \"\"\"\n return unique dict for each uwsgi request.\n note: won't work on non-uwsgi cases\n \"\"\"\n\n def _wrapped(*args, **kwargs):\n data = _get_request_unique_cache()\n return func(data, *args, **kwargs)\n\n return _wrapped"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a string representation of the object.", "response": "def safe_repr(obj):\n \"\"\"\n Try to get ``__name__`` first, ``__class__.__name__`` second\n and finally, if we can't get anything acceptable, fallback\n to user a ``repr()`` call.\n \"\"\"\n name = getattr(obj, '__name__', getattr(obj.__class__, '__name__'))\n if name == 'ndict':\n name = 'dict'\n return name or repr(obj)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreceives a data object that will be in the form and filter out keys that match the required_items.", "response": "def sift(data, required_items=None):\n \"\"\"\n Receive a ``data`` object that will be in the form\n of a normalized structure (e.g. ``{0: {'a': 0}}``) and\n filter out keys that match the ``required_items``.\n \"\"\"\n required_items = required_items or []\n new_data = {}\n for k, v in data.items():\n if v[0] in required_items:\n new_data[k] = v\n continue\n for required_item in required_items:\n key = getattr(required_item, '_object', False)\n if key:\n if v[0] == key:\n new_data[k] = v\n\n return re_sort(new_data)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef data_item(data):\n if isinstance(data, ndict):\n # OK, we have something that looks like {0: ('a', 'b')}\n # or something that is a regular dictionary\n # so try to return 'a' regardless of the length\n for item in data:\n return repr(data[item][0])\n elif isinstance(data, dict):\n for item in data:\n return repr(data[item])\n elif isinstance(data, list):\n return repr(data[0])\n return repr(data)", "response": "Return a string representation of an unexpected data item."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef ensure(assertion, message=None):\n message = message or assertion\n\n if not assertion:\n raise AssertionError(message)\n\n return True", "response": "Checks an assertion argument for truth -ness. Will return True or\n explicitly raise AssertionError."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a copy of the fluent shape ignoring batch size if in batch mode.", "response": "def fluent_shape(self) -> Sequence[int]:\n '''Returns a copy of the fluent shape, ignoring batch size if in batch mode.'''\n return tuple(self._shape.as_list()[1:] if self._batch else self._shape.as_list()[:])"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef broadcast(cls,\n shape1: 'TensorFluentShape',\n shape2: 'TensorFluentShape') -> Tuple[Reshaping, Reshaping]:\n '''It broadcasts the fluent shapes if any input is in batch mode.\n\n It handles input shapes in different modes, expanding its\n dimensions if necessary. It outputs a tuple with new shapes.\n If no input shape is in batch mode, return (None, None).\n If an input shape does not need to be changed, return None.\n\n Args:\n shape1: A fluent's shape.\n shape2: A fluent's shape.\n\n Returns:\n A pair of new shapes.\n '''\n reshape_1, reshape_2 = None, None\n\n if not (shape1._batch or shape2._batch):\n return reshape_1, reshape_2\n\n size_1, size_2 = shape1.fluent_size, shape2.fluent_size\n size_diff = abs(size_1 - size_2)\n if size_diff == 0:\n return reshape_1, reshape_2\n\n if size_2 > size_1 and not (size_1 == 0 and not shape1._batch):\n reshape_1 = [1] * size_diff + list(shape1.fluent_shape)\n if shape1._batch:\n reshape_1 = [shape1.batch_size] + reshape_1\n elif size_1 > size_2 and not (size_2 == 0 and not shape2._batch):\n reshape_2 = [1] * size_diff + list(shape2.fluent_shape)\n if shape2._batch:\n reshape_2 = [shape2.batch_size] + reshape_2\n return reshape_1, reshape_2", "response": "This function broadcasts the fluent shapes if any input is in batch mode. It broadcasts the fluent shapes if any input is in batch mode and expanding its\n dimensions if necessary. It outputs a tuple with new shapes."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef run(bam, chrom, pos1, pos2, reffa, chr_reffa, parameters):\n\n # check for chr ref\n is_chr_query = chrom.startswith('chr')\n if is_chr_query and chr_reffa is None:\n chr_reffa = reffa\n\n # check bam ref type\n bam_header = subprocess.check_output(\"samtools view -H {}\".format(bam), shell=True)\n is_chr_bam = bam_header.find('SN:chr') != -1\n if is_chr_bam:\n reffa = chr_reffa\n if not is_chr_query and is_chr_bam:\n chrom = 'chr' + chrom\n if is_chr_query and not is_chr_bam:\n chrom = re.sub(r'^chr', '', chrom)\n\n posmin = min(pos1, pos2)\n posmax = max(pos1, pos2)\n cmd = \"samtools view -bh {bam} {chrom}:{pos1}-{pos2} \" \\\n \"| samtools mpileup {parameters} -f {reffa} -\".format(bam=bam, chrom=chrom,\n pos1=posmin, pos2=posmax,\n reffa=reffa, parameters=parameters)\n if pos1 == pos2:\n cmd += \" | awk '$2 == {pos}'\".format(pos=pos1)\n else:\n cmd += \" | tail -n +2 | awk '$2 >= {posmin} && $2 <= {posmax}'\".format(posmin=posmin, posmax=posmax)\n sys.stderr.write(\"Running:\\n{}\\n\".format(cmd))\n child = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)\n stdout, stderr = child.communicate()\n if child.returncode != 0:\n if len(stdout) == 0 and stderr is None:\n warnings.warn(\"Command:\\n{cmd}\\n did not exit with zero exit code. \"\n \"Possibly no coverage for sample.\".format(cmd=cmd))\n else:\n raise(Exception(\"Command:\\n{cmd}\\n did not exit with zero exit code. \"\n \"Check command.\".format(cmd=cmd)))\n else:\n return stdout", "response": "Run mpileup on given chrom and pos"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nexecuting a file and return True if it exists False otherwise", "response": "def execfile(fname, _globals, _locals):\n \"\"\"\n Usage: execfile('path/to/file.py', globals(), locals())\n \"\"\"\n if os.path.exists(fname):\n with open(fname) as f:\n code = compile(f.read(), os.path.basename(fname), 'exec')\n exec(code, _globals, _locals)\n return True\n else:\n return False"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef auto_directory(rel_name):\n dir_name = rel_path(rel_name, check=False)\n if not os.path.exists(dir_name):\n os.makedirs(dir_name, exist_ok=True)\n return dir_name", "response": "Create a directory with the given name."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nparse an ISO 8601 date string into a date object.", "response": "def parse_iso8601_date(string):\n \"\"\"\n Parse an ISO 8601 date string\n \"\"\"\n\n # Match ISO 8601?\n match = _RE_ISO8601_DATE.search(string)\n if not match:\n raise ValueError('Expected ISO 8601 date')\n\n # Extract ISO 8601 components\n year = int(match.group('year'))\n month = int(match.group('month'))\n day = int(match.group('day'))\n\n return date(year, month, day)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nparses an ISO 8601 date or time string into a datetime object.", "response": "def parse_iso8601_datetime(string):\n \"\"\"\n Parse an ISO 8601 date/time string\n \"\"\"\n\n # Match ISO 8601?\n match = _RE_ISO8601_DATETIME.search(string)\n if not match:\n raise ValueError('Expected ISO 8601 date/time')\n\n # Extract ISO 8601 components\n year = int(match.group('year'))\n month = int(match.group('month'))\n day = int(match.group('day'))\n hour = int(match.group('hour')) if match.group('hour') else 0\n minute = int(match.group('min')) if match.group('min') else 0\n sec = int(match.group('sec')) if match.group('sec') else 0\n microsec = int(float('.' + match.group('fracsec')) * 1000000) if match.group('fracsec') else 0\n offhour = int(match.group('offsign') + match.group('offhour')) if match.group('offhour') else 0\n offmin = int(match.group('offsign') + match.group('offmin')) if match.group('offmin') else 0\n\n return (datetime(year, month, day, hour, minute, sec, microsec, TZUTC) -\n timedelta(hours=offhour, minutes=offmin))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a tuple of tensors representing the initial state fluents.", "response": "def compile_initial_state(self, batch_size: Optional[int] = None) -> Sequence[tf.Tensor]:\n '''Returns a tuple of tensors representing the initial state fluents.\n\n Args:\n batch_size (Optional[int]): The batch size.\n\n Returns:\n Sequence[tf.Tensor]: A tuple of tensors.\n '''\n with self.graph.as_default():\n with tf.name_scope('initial_state'):\n self._initialize_initial_state_fluents()\n if batch_size is None:\n return self.initial_state_fluents\n return self._compile_batch_fluents(self.initial_state_fluents, batch_size)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef compile_default_action(self, batch_size: Optional[int] = None) -> Sequence[tf.Tensor]:\n '''Returns a tuple of tensors representing the default action fluents.\n\n Args:\n batch_size (int): The batch size.\n\n Returns:\n Sequence[tf.Tensor]: A tuple of tensors.\n '''\n with self.graph.as_default():\n with tf.name_scope('default_action'):\n self._initialize_default_action_fluents()\n if batch_size is None:\n return self.default_action_fluents\n return self._compile_batch_fluents(self.default_action_fluents, batch_size)", "response": "Returns a tuple of tensors representing the default action fluents."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef cpfs(self,\n state: Sequence[tf.Tensor],\n action: Sequence[tf.Tensor],\n noise: Optional[Noise] = None) -> Tuple[List[TensorFluent], List[TensorFluent]]:\n '''Compiles the intermediate and next state fluent CPFs given\n the current `state` and `action`.\n\n Args:\n state (Sequence[tf.Tensor]): A tuple of state tensors.\n action (Sequence[tf.Tensor]): A tuple of action tensors.\n\n Returns:\n Tuple[List[TensorFluent], List[TensorFluent]]: A pair of lists of TensorFluent\n representing the intermediate and state CPFs.\n '''\n scope = self.transition_scope(state, action)\n batch_size = int(state[0].shape[0])\n interm_fluents, next_state_fluents = self.compile_cpfs(scope, batch_size, noise)\n interms = [fluent for _, fluent in interm_fluents]\n next_state = [fluent for _, fluent in next_state_fluents]\n return interms, next_state", "response": "Compiles the intermediate and next state fluent CPFs given\n the current state and action."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef reward(self,\n state: Sequence[tf.Tensor],\n action: Sequence[tf.Tensor],\n next_state: Sequence[tf.Tensor]) -> tf.Tensor:\n '''Compiles the reward function given the current `state`, `action` and\n `next_state`.\n\n Args:\n state (Sequence[tf.Tensor]): A tuple of current state tensors.\n action (Sequence[tf.Tensor]): A tuple of action tensors.\n next_state (Sequence[tf.Tensor]): A tuple of next state tensors.\n\n Returns:\n (:obj:`tf.Tensor`): A tensor representing the reward function.\n '''\n scope = self.reward_scope(state, action, next_state)\n r = self.compile_reward(scope).tensor\n with self.graph.as_default():\n with tf.name_scope('reward'):\n return tf.expand_dims(r, -1)", "response": "Compiles the reward function given the current state action and next_state."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncompiles the intermediate and next state fluent CPFs given the current state and action scope.", "response": "def compile_cpfs(self,\n scope: Dict[str, TensorFluent],\n batch_size: Optional[int] = None,\n noise: Optional[Noise] = None) -> Tuple[List[CPFPair], List[CPFPair]]:\n '''Compiles the intermediate and next state fluent CPFs given the current `state` and `action` scope.\n\n Args:\n scope (Dict[str, :obj:`rddl2tf.fluent.TensorFluent`]): The fluent scope for CPF evaluation.\n batch_size (Optional[int]): The batch size.\n\n Returns:\n Tuple[List[CPFPair], List[CPFPair]]: A pair of lists of TensorFluent\n representing the intermediate and state CPFs.\n '''\n interm_fluents = self.compile_intermediate_cpfs(scope, batch_size, noise)\n scope.update(dict(interm_fluents))\n next_state_fluents = self.compile_state_cpfs(scope, batch_size, noise)\n return interm_fluents, next_state_fluents"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncompiling the intermediate fluent CPFs given the current state and action scope.", "response": "def compile_intermediate_cpfs(self,\n scope: Dict[str, TensorFluent],\n batch_size: Optional[int] = None,\n noise: Optional[Noise] = None) -> List[CPFPair]:\n '''Compiles the intermediate fluent CPFs given the current `state` and `action` scope.\n\n Args:\n scope (Dict[str, :obj:`rddl2tf.fluent.TensorFluent`]): The fluent scope for CPF evaluation.\n batch_size (Optional[int]): The batch size.\n\n Returns:\n A list of intermediate fluent CPFs compiled to :obj:`rddl2tf.fluent.TensorFluent`.\n '''\n interm_fluents = []\n\n with self.graph.as_default():\n with tf.name_scope('intermediate_cpfs'):\n\n for cpf in self.rddl.domain.intermediate_cpfs:\n cpf_noise = noise.get(cpf.name, None) if noise is not None else None\n\n name_scope = utils.identifier(cpf.name)\n with tf.name_scope(name_scope):\n t = self._compile_expression(cpf.expr, scope, batch_size, cpf_noise)\n\n interm_fluents.append((cpf.name, t))\n scope[cpf.name] = t\n\n return interm_fluents"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef compile_state_cpfs(self,\n scope: Dict[str, TensorFluent],\n batch_size: Optional[int] = None,\n noise: Optional[Noise] = None) -> List[CPFPair]:\n '''Compiles the next state fluent CPFs given the current `state` and `action` scope.\n\n Args:\n scope (Dict[str, :obj:`rddl2tf.fluent.TensorFluent`]): The fluent scope for CPF evaluation.\n batch_size (Optional[int]): The batch size.\n\n Returns:\n A list of state fluent CPFs compiled to :obj:`rddl2tf.fluent.TensorFluent`.\n '''\n next_state_fluents = []\n\n with self.graph.as_default():\n with tf.name_scope('state_cpfs'):\n\n for cpf in self.rddl.domain.state_cpfs:\n cpf_noise = noise.get(cpf.name, None) if noise is not None else None\n\n name_scope = utils.identifier(cpf.name)\n with tf.name_scope(name_scope):\n t = self._compile_expression(cpf.expr, scope, batch_size, cpf_noise)\n\n next_state_fluents.append((cpf.name, t))\n\n key = lambda f: self.rddl.domain.next_state_fluent_ordering.index(f[0])\n next_state_fluents = sorted(next_state_fluents, key=key)\n\n return next_state_fluents", "response": "Compiles the next state fluent CPFs given the current state and action scope."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncompiling the reward function given the fluent scope.", "response": "def compile_reward(self, scope: Dict[str, TensorFluent]) -> TensorFluent:\n '''Compiles the reward function given the fluent `scope`.\n\n Args:\n scope (Dict[str, :obj:`rddl2tf.fluent.TensorFluent`]): The fluent scope for reward evaluation.\n\n Returns:\n A :obj:`rddl2tf.fluent.TensorFluent` representing the reward function.\n '''\n reward_expr = self.rddl.domain.reward\n with self.graph.as_default():\n with tf.name_scope('reward'):\n return self._compile_expression(reward_expr, scope)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef compile_state_action_constraints(self,\n state: Sequence[tf.Tensor],\n action: Sequence[tf.Tensor]) -> List[TensorFluent]:\n '''Compiles the state-action constraints given current `state` and `action` fluents.\n\n Args:\n state (Sequence[tf.Tensor]): The current state fluents.\n action (Sequence[tf.Tensor]): The action fluents.\n\n Returns:\n A list of :obj:`rddl2tf.fluent.TensorFluent`.\n '''\n scope = self.transition_scope(state, action)\n constraints = []\n with self.graph.as_default():\n with tf.name_scope('state_action_constraints'):\n for p in self.rddl.domain.constraints:\n fluent = self._compile_expression(p, scope)\n constraints.append(fluent)\n return constraints", "response": "Compiles the state - action constraints given current state and action fluents."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef compile_action_preconditions(self,\n state: Sequence[tf.Tensor],\n action: Sequence[tf.Tensor]) -> List[TensorFluent]:\n '''Compiles the action preconditions given current `state` and `action` fluents.\n\n Args:\n state (Sequence[tf.Tensor]): The current state fluents.\n action (Sequence[tf.Tensor]): The action fluents.\n\n Returns:\n A list of :obj:`rddl2tf.fluent.TensorFluent`.\n '''\n scope = self.action_precondition_scope(state, action)\n preconds = []\n with self.graph.as_default():\n with tf.name_scope('action_preconditions'):\n for p in self.rddl.domain.preconds:\n fluent = self._compile_expression(p, scope)\n preconds.append(fluent)\n return preconds", "response": "Compiles the action preconditions given current state and action fluents."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncompile the state invarints given current state fluents.", "response": "def compile_state_invariants(self,\n state: Sequence[tf.Tensor]) -> List[TensorFluent]:\n '''Compiles the state invarints given current `state` fluents.\n\n Args:\n state (Sequence[tf.Tensor]): The current state fluents.\n\n Returns:\n A list of :obj:`rddl2tf.fluent.TensorFluent`.\n '''\n scope = self.state_invariant_scope(state)\n invariants = []\n with self.graph.as_default():\n with tf.name_scope('state_invariants'):\n for p in self.rddl.domain.invariants:\n fluent = self._compile_expression(p, scope)\n invariants.append(fluent)\n return invariants"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef compile_action_preconditions_checking(self,\n state: Sequence[tf.Tensor],\n action: Sequence[tf.Tensor]) -> tf.Tensor:\n '''Combines the action preconditions into an applicability checking op.\n\n Args:\n state (Sequence[tf.Tensor]): The current state fluents.\n action (Sequence[tf.Tensor]): The action fluents.\n\n Returns:\n A boolean tensor for checking if `action` is application in `state`.\n '''\n with self.graph.as_default():\n with tf.name_scope('action_preconditions_checking'):\n preconds = self.compile_action_preconditions(state, action)\n all_preconds = tf.stack([p.tensor for p in preconds], axis=1)\n checking = tf.reduce_all(all_preconds, axis=1)\n return checking", "response": "Combines the action preconditions into an applicability checking op.\n "} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncompiles all actions bounds for the given state.", "response": "def compile_action_bound_constraints(self,\n state: Sequence[tf.Tensor]) -> Dict[str, Bounds]:\n '''Compiles all actions bounds for the given `state`.\n\n Args:\n state (Sequence[tf.Tensor]): The current state fluents.\n\n Returns:\n A mapping from action names to a pair of\n :obj:`rddl2tf.fluent.TensorFluent` representing\n its lower and upper bounds.\n '''\n scope = self.action_precondition_scope(state)\n\n lower_bounds = self.rddl.domain.action_lower_bound_constraints\n upper_bounds = self.rddl.domain.action_upper_bound_constraints\n\n with self.graph.as_default():\n with tf.name_scope('action_bound_constraints'):\n\n bounds = {}\n for name in self.rddl.domain.action_fluent_ordering:\n\n lower_expr = lower_bounds.get(name)\n lower = None\n if lower_expr is not None:\n with tf.name_scope('lower_bound'):\n lower = self._compile_expression(lower_expr, scope)\n\n upper_expr = upper_bounds.get(name)\n upper = None\n if upper_expr is not None:\n with tf.name_scope('upper_bound'):\n upper = self._compile_expression(upper_expr, scope)\n\n bounds[name] = (lower, upper)\n\n return bounds"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef non_fluents_scope(self) -> Dict[str, TensorFluent]:\n '''Returns a partial scope with non-fluents.\n\n Returns:\n A mapping from non-fluent names to :obj:`rddl2tf.fluent.TensorFluent`.\n '''\n if self.__dict__.get('non_fluents') is None:\n self._initialize_non_fluents()\n return dict(self.non_fluents)", "response": "Returns a partial scope with non - fluents."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a partial scope with current state - fluents.", "response": "def state_scope(self, state_fluents: Sequence[tf.Tensor]) -> Dict[str, TensorFluent]:\n '''Returns a partial scope with current state-fluents.\n\n Args:\n state_fluents (Sequence[tf.Tensor]): The current state fluents.\n\n Returns:\n A mapping from state fluent names to :obj:`rddl2tf.fluent.TensorFluent`.\n '''\n return dict(zip(self.rddl.domain.state_fluent_ordering, state_fluents))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a partial scope with current action - fluents.", "response": "def action_scope(self, action_fluents: Sequence[tf.Tensor]) -> Dict[str, TensorFluent]:\n '''Returns a partial scope with current action-fluents.\n\n Args:\n action_fluents (Sequence[tf.Tensor]): The action fluents.\n\n Returns:\n A mapping from action fluent names to :obj:`rddl2tf.fluent.TensorFluent`.\n '''\n return dict(zip(self.rddl.domain.action_fluent_ordering, action_fluents))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef next_state_scope(self, next_state_fluents: Sequence[tf.Tensor]) -> Dict[str, TensorFluent]:\n '''Returns a partial scope with current next state-fluents.\n\n Args:\n next_state_fluents (Sequence[tf.Tensor]): The next state fluents.\n\n Returns:\n A mapping from next state fluent names to :obj:`rddl2tf.fluent.TensorFluent`.\n '''\n return dict(zip(self.rddl.domain.next_state_fluent_ordering, next_state_fluents))", "response": "Returns a partial scope with current next state - fluents."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the complete transition fluent scope for the current state and action fluents.", "response": "def transition_scope(self,\n state: Sequence[tf.Tensor],\n action: Sequence[tf.Tensor]) -> Dict[str, TensorFluent]:\n '''Returns the complete transition fluent scope\n for the current `state` and `action` fluents.\n\n Args:\n state (Sequence[tf.Tensor]): The current state fluents.\n action (Sequence[tf.Tensor]): The action fluents.\n\n Returns:\n A mapping from fluent names to :obj:`rddl2tf.fluent.TensorFluent`.\n '''\n scope = {}\n scope.update(self.non_fluents_scope())\n scope.update(self.state_scope(state))\n scope.update(self.action_scope(action))\n return scope"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the complete reward fluent scope for the current state action and next_state fluents.", "response": "def reward_scope(self,\n state: Sequence[tf.Tensor],\n action: Sequence[tf.Tensor],\n next_state: Sequence[tf.Tensor]) -> Dict[str, TensorFluent]:\n '''Returns the complete reward fluent scope for the\n current `state`, `action` fluents, and `next_state` fluents.\n\n Args:\n state (Sequence[tf.Tensor]): The current state fluents.\n action (Sequence[tf.Tensor]): The action fluents.\n next_state (Sequence[tf.Tensor]): The next state fluents.\n\n Returns:\n A mapping from fluent names to :obj:`rddl2tf.fluent.TensorFluent`.\n '''\n scope = {}\n scope.update(self.non_fluents_scope())\n scope.update(self.state_scope(state))\n scope.update(self.action_scope(action))\n scope.update(self.next_state_scope(next_state))\n return scope"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the state invariant fluent scope for the current state.", "response": "def state_invariant_scope(self, state: Sequence[tf.Tensor]):\n '''Returns the state invariant fluent scope for the current `state`.\n\n Args:\n state (Sequence[tf.Tensor]): The current state fluents.\n\n Returns:\n A mapping from fluent names to :obj:`rddl2tf.fluent.TensorFluent`.\n '''\n scope = {}\n scope.update(self.non_fluents_scope())\n scope.update(self.state_scope(state))\n return scope"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the non - fluents instantiated.", "response": "def _initialize_non_fluents(self):\n '''Returns the non-fluents instantiated.'''\n non_fluents = self.rddl.domain.non_fluents\n initializer = self.rddl.non_fluents.init_non_fluent\n self.non_fluents = self._initialize_pvariables(\n non_fluents,\n self.rddl.domain.non_fluent_ordering,\n initializer)\n return self.non_fluents"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _initialize_initial_state_fluents(self):\n '''Returns the initial state-fluents instantiated.'''\n state_fluents = self.rddl.domain.state_fluents\n initializer = self.rddl.instance.init_state\n self.initial_state_fluents = self._initialize_pvariables(\n state_fluents,\n self.rddl.domain.state_fluent_ordering,\n initializer)\n return self.initial_state_fluents", "response": "Returns the initial state - fluents instantiated."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _initialize_default_action_fluents(self):\n '''Returns the default action-fluents instantiated.'''\n action_fluents = self.rddl.domain.action_fluents\n self.default_action_fluents = self._initialize_pvariables(\n action_fluents,\n self.rddl.domain.action_fluent_ordering)\n return self.default_action_fluents", "response": "Returns the default action - fluents instantiated."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncompiling fluents into tensors with given batch_size.", "response": "def _compile_batch_fluents(self,\n fluents: List[Tuple[str, TensorFluent]],\n batch_size: int) -> Sequence[tf.Tensor]:\n '''Compiles `fluents` into tensors with given `batch_size`.\n\n Returns:\n Sequence[tf.Tensor]: A tuple of tensors with first dimension\n corresponding to the batch size.\n '''\n batch_fluents = []\n with self.graph.as_default():\n for name, fluent in fluents:\n name_scope = utils.identifier(name)\n with tf.name_scope(name_scope):\n t = tf.stack([fluent.tensor] * batch_size)\n batch_fluents.append(t)\n return tuple(batch_fluents)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncompiles the expression expr into a TensorFluent in the given scope with optional batch size.", "response": "def _compile_expression(self,\n expr: Expression,\n scope: Dict[str, TensorFluent],\n batch_size: Optional[int] = None,\n noise: Optional[List[tf.Tensor]] = None) -> TensorFluent:\n '''Compile the expression `expr` into a TensorFluent\n in the given `scope` with optional batch size.\n\n Args:\n expr (:obj:`rddl2tf.expr.Expression`): A RDDL expression.\n scope (Dict[str, :obj:`rddl2tf.fluent.TensorFluent`]): A fluent scope.\n batch_size (Optional[size]): The batch size.\n\n Returns:\n :obj:`rddl2tf.fluent.TensorFluent`: The compiled TensorFluent.\n '''\n etype2compiler = {\n 'constant': self._compile_constant_expression,\n 'pvar': self._compile_pvariable_expression,\n 'randomvar': self._compile_random_variable_expression,\n 'arithmetic': self._compile_arithmetic_expression,\n 'boolean': self._compile_boolean_expression,\n 'relational': self._compile_relational_expression,\n 'func': self._compile_function_expression,\n 'control': self._compile_control_flow_expression,\n 'aggregation': self._compile_aggregation_expression\n }\n\n etype = expr.etype\n if etype[0] not in etype2compiler:\n raise ValueError('Expression type unknown: {}'.format(etype))\n\n with self.graph.as_default():\n compiler_fn = etype2compiler[etype[0]]\n return compiler_fn(expr, scope, batch_size, noise)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncompiling a constant expression expr into a TensorFluent in the given scope with optional batch size.", "response": "def _compile_constant_expression(self,\n expr: Expression,\n scope: Dict[str, TensorFluent],\n batch_size: Optional[int] = None,\n noise: Optional[List[tf.Tensor]] = None) -> TensorFluent:\n '''Compile a constant expression `expr` into a TensorFluent\n in the given `scope` with optional batch size.\n\n Args:\n expr (:obj:`rddl2tf.expr.Expression`): A RDDL constant expression.\n scope (Dict[str, :obj:`rddl2tf.fluent.TensorFluent`]): A fluent scope.\n batch_size (Optional[size]): The batch size.\n\n Returns:\n :obj:`rddl2tf.fluent.TensorFluent`: The compiled expression as a TensorFluent.\n '''\n etype = expr.etype\n args = expr.args\n dtype = utils.python_type_to_dtype(etype[1])\n fluent = TensorFluent.constant(args, dtype=dtype)\n return fluent"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncompiles a pvariable expression expr into a TensorFluent in the given scope with optional batch size.", "response": "def _compile_pvariable_expression(self,\n expr: Expression,\n scope: Dict[str, TensorFluent],\n batch_size: Optional[int] = None,\n noise: Optional[List[tf.Tensor]] = None) -> TensorFluent:\n '''Compile a pvariable expression `expr` into a TensorFluent\n in the given `scope` with optional batch size.\n\n Args:\n expr (:obj:`rddl2tf.expr.Expression`): A RDDL pvariable expression.\n scope (Dict[str, :obj:`rddl2tf.fluent.TensorFluent`]): A fluent scope.\n batch_size (Optional[size]): The batch size.\n\n Returns:\n :obj:`rddl2tf.fluent.TensorFluent`: The compiled expression as a TensorFluent.\n '''\n etype = expr.etype\n args = expr.args\n name = expr._pvar_to_name(args)\n if name not in scope:\n raise ValueError('Variable {} not in scope.'.format(name))\n fluent = scope[name]\n scope = args[1] if args[1] is not None else []\n if isinstance(fluent, TensorFluent):\n fluent = TensorFluent(fluent.tensor, scope, batch=fluent.batch)\n elif isinstance(fluent, tf.Tensor):\n fluent = TensorFluent(fluent, scope, batch=self.batch_mode)\n else:\n raise ValueError('Variable in scope must be TensorFluent-like: {}'.format(fluent))\n return fluent"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncompile a random variable expression expr into a TensorFluent in the given scope with optional batch size and optional noise.", "response": "def _compile_random_variable_expression(self,\n expr: Expression,\n scope: Dict[str, TensorFluent],\n batch_size: Optional[int] = None,\n noise: Optional[List[tf.Tensor]] = None) -> TensorFluent:\n '''Compile a random variable expression `expr` into a TensorFluent\n in the given `scope` with optional batch size.\n\n If `reparam` tensor is given, then it conditionally stops gradient\n backpropagation at the batch level where `reparam` is False.\n\n Args:\n expr (:obj:`rddl2tf.expr.Expression`): A RDDL random variable expression.\n scope (Dict[str, :obj:`rddl2tf.fluent.TensorFluent`]): A fluent scope.\n batch_size (Optional[size]): The batch size.\n\n Returns:\n :obj:`rddl2tf.fluent.TensorFluent`: The compiled expression as a TensorFluent.\n '''\n etype = expr.etype\n args = expr.args\n\n if etype[1] == 'KronDelta':\n sample = self._compile_expression(args[0], scope, batch_size, noise)\n elif etype[1] == 'Bernoulli':\n mean = self._compile_expression(args[0], scope, batch_size, noise)\n dist, sample = TensorFluent.Bernoulli(mean, batch_size)\n elif etype[1] == 'Uniform':\n low = self._compile_expression(args[0], scope, batch_size, noise)\n high = self._compile_expression(args[1], scope, batch_size, noise)\n dist, sample = TensorFluent.Uniform(low, high, batch_size)\n elif etype[1] == 'Normal':\n if noise is None:\n mean = self._compile_expression(args[0], scope, batch_size, noise)\n variance = self._compile_expression(args[1], scope, batch_size, noise)\n dist, sample = TensorFluent.Normal(mean, variance, batch_size)\n else:\n xi = noise.pop()\n xi = TensorFluent(xi, scope=[], batch=True)\n mean = self._compile_expression(args[0], scope, batch_size, noise)\n variance = self._compile_expression(args[1], scope, batch_size, noise)\n sample = mean + TensorFluent.sqrt(variance) * xi\n elif etype[1] == 'Laplace':\n mean = self._compile_expression(args[0], scope, batch_size, noise)\n variance = self._compile_expression(args[1], scope, batch_size, noise)\n dist, sample = TensorFluent.Laplace(mean, variance, batch_size)\n elif etype[1] == 'Gamma':\n shape = self._compile_expression(args[0], scope, batch_size, noise)\n scale = self._compile_expression(args[1], scope, batch_size, noise)\n dist, sample = TensorFluent.Gamma(shape, scale, batch_size)\n elif etype[1] == 'Exponential':\n mean = self._compile_expression(args[0], scope, batch_size, noise)\n dist, sample = TensorFluent.Exponential(mean, batch_size)\n else:\n raise ValueError('Invalid random variable expression:\\n{}.'.format(expr))\n\n return sample"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncompile an arithmetic expression expr into a TensorFluent in the given scope with optional batch size.", "response": "def _compile_arithmetic_expression(self,\n expr: Expression,\n scope: Dict[str, TensorFluent],\n batch_size: Optional[int] = None,\n noise: Optional[List[tf.Tensor]] = None) -> TensorFluent:\n '''Compile an arithmetic expression `expr` into a TensorFluent\n in the given `scope` with optional batch size.\n\n Args:\n expr (:obj:`rddl2tf.expr.Expression`): A RDDL arithmetic expression.\n scope (Dict[str, :obj:`rddl2tf.fluent.TensorFluent`]): A fluent scope.\n batch_size (Optional[size]): The batch size.\n\n Returns:\n :obj:`rddl2tf.fluent.TensorFluent`: The compiled expression as a TensorFluent.\n '''\n etype = expr.etype\n args = expr.args\n\n if len(args) == 1:\n etype2op = {\n '+': lambda x: x,\n '-': lambda x: -x\n }\n\n if etype[1] not in etype2op:\n raise ValueError('Invalid binary arithmetic expression:\\n{}'.format(expr))\n\n op = etype2op[etype[1]]\n x = self._compile_expression(args[0], scope, batch_size, noise)\n fluent = op(x)\n\n else:\n etype2op = {\n '+': lambda x, y: x + y,\n '-': lambda x, y: x - y,\n '*': lambda x, y: x * y,\n '/': lambda x, y: x / y,\n }\n\n if etype[1] not in etype2op:\n raise ValueError('Invalid binary arithmetic expression:\\n{}'.format(expr))\n\n op = etype2op[etype[1]]\n x = self._compile_expression(args[0], scope, batch_size, noise)\n y = self._compile_expression(args[1], scope, batch_size, noise)\n fluent = op(x, y)\n\n return fluent"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncompiles a RDDL unary function expression expr into a TensorFluent in the given scope with optional batch size.", "response": "def _compile_function_expression(self,\n expr: Expression,\n scope: Dict[str, TensorFluent],\n batch_size: Optional[int] = None,\n noise: Optional[List[tf.Tensor]] = None) -> TensorFluent:\n '''Compile a function expression `expr` into a TensorFluent\n in the given `scope` with optional batch size.\n\n Args:\n expr (:obj:`rddl2tf.expr.Expression`): A RDDL function expression.\n scope (Dict[str, :obj:`rddl2tf.fluent.TensorFluent`]): A fluent scope.\n batch_size (Optional[size]): The batch size.\n\n Returns:\n :obj:`rddl2tf.fluent.TensorFluent`: The compiled expression as a TensorFluent.\n '''\n etype = expr.etype\n args = expr.args\n\n if len(args) == 1:\n\n etype2func = {\n 'abs': TensorFluent.abs,\n 'exp': TensorFluent.exp,\n 'log': TensorFluent.log,\n 'sqrt': TensorFluent.sqrt,\n 'cos': TensorFluent.cos,\n 'sin': TensorFluent.sin,\n 'tan': TensorFluent.tan,\n 'acos': TensorFluent.acos,\n 'arccos': TensorFluent.acos,\n 'asin': TensorFluent.asin,\n 'arcsin': TensorFluent.asin,\n 'atan': TensorFluent.atan,\n 'arctan': TensorFluent.atan,\n 'round': TensorFluent.round,\n 'ceil': TensorFluent.ceil,\n 'floor': TensorFluent.floor\n }\n\n if etype[1] not in etype2func:\n raise ValueError('Invalid unary function expression:\\n{}'.format(expr))\n\n op = etype2func[etype[1]]\n x = self._compile_expression(args[0], scope, batch_size, noise)\n fluent = op(x)\n\n else:\n etype2func = {\n 'pow': TensorFluent.pow,\n 'max': TensorFluent.max,\n 'min': TensorFluent.min\n }\n\n if etype[1] not in etype2func:\n raise ValueError('Invalid binary function expression:\\n{}'.format(expr))\n\n op = etype2func[etype[1]]\n x = self._compile_expression(args[0], scope, batch_size, noise)\n y = self._compile_expression(args[1], scope, batch_size, noise)\n fluent = op(x, y)\n\n return fluent"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _compile_control_flow_expression(self,\n expr: Expression,\n scope: Dict[str, TensorFluent],\n batch_size: Optional[int] = None,\n noise: Optional[List[tf.Tensor]] = None) -> TensorFluent:\n '''Compile a control flow expression `expr` into a TensorFluent\n in the given `scope` with optional batch size.\n\n Args:\n expr (:obj:`rddl2tf.expr.Expression`): A RDDL control flow expression.\n scope (Dict[str, :obj:`rddl2tf.fluent.TensorFluent`]): A fluent scope.\n batch_size (Optional[size]): The batch size.\n\n Returns:\n :obj:`rddl2tf.fluent.TensorFluent`: The compiled expression as a TensorFluent.\n '''\n etype = expr.etype\n args = expr.args\n if etype[1] == 'if':\n condition = self._compile_expression(args[0], scope, batch_size, noise)\n true_case = self._compile_expression(args[1], scope, batch_size, noise)\n false_case = self._compile_expression(args[2], scope, batch_size, noise)\n fluent = TensorFluent.if_then_else(condition, true_case, false_case)\n else:\n raise ValueError('Invalid control flow expression:\\n{}'.format(expr))\n return fluent", "response": "Compile a control flow expression expr into a TensorFluent in the given scope with optional batch size."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncompile an aggregation expression expr into a TensorFluent in the given scope with optional batch size.", "response": "def _compile_aggregation_expression(self,\n expr: Expression,\n scope: Dict[str, TensorFluent],\n batch_size: Optional[int] = None,\n noise: Optional[List[tf.Tensor]] = None) -> TensorFluent:\n '''Compile an aggregation expression `expr` into a TensorFluent\n in the given `scope` with optional batch size.\n\n Args:\n expr (:obj:`rddl2tf.expr.Expression`): A RDDL aggregation expression.\n scope (Dict[str, :obj:`rddl2tf.fluent.TensorFluent`]): A fluent scope.\n batch_size (Optional[size]): The batch size.\n\n Returns:\n :obj:`rddl2tf.fluent.TensorFluent`: The compiled expression as a TensorFluent.\n '''\n etype = expr.etype\n args = expr.args\n\n typed_var_list = args[:-1]\n vars_list = [var for _, (var, _) in typed_var_list]\n expr = args[-1]\n\n x = self._compile_expression(expr, scope)\n\n etype2aggr = {\n 'sum': x.sum,\n 'prod': x.prod,\n 'avg': x.avg,\n 'maximum': x.maximum,\n 'minimum': x.minimum,\n 'exists': x.exists,\n 'forall': x.forall\n }\n\n if etype[1] not in etype2aggr:\n raise ValueError('Invalid aggregation expression {}.'.format(expr))\n\n aggr = etype2aggr[etype[1]]\n fluent = aggr(vars_list=vars_list)\n\n return fluent"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a numpy array that contains the number of entries in table that satisfy the queries.", "response": "def mask(self, table):\n \"\"\"\n Use the current Query object to count the number of entries in `table`\n that satisfy `queries`.\n\n Parameters\n ----------\n table : NumPy structured array, astropy Table, etc.\n\n Returns\n -------\n mask : numpy bool array\n \"\"\"\n if self._operator is None:\n if self._operands is None:\n return np.ones(self._get_table_len(table), dtype=np.bool)\n else:\n return self._create_mask(table, self._operands)\n\n if self._operator == 'NOT':\n return ~self._operands.mask(table)\n\n if self._operator == 'AND':\n op_func = np.logical_and\n elif self._operator == 'OR':\n op_func = np.logical_or\n elif self._operator == 'XOR':\n op_func = np.logical_xor\n\n mask_this = self._operands[0].mask(table)\n for op in self._operands[1:]:\n mask_this = op_func(mask_this, op.mask(table), out=mask_this)\n\n return mask_this"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef filter(self, table, column_slice=None):\n if self._operator is None and self._operands is None:\n return table if column_slice is None else self._get_table_column(table, column_slice)\n\n if self._operator == 'AND' and column_slice is None:\n for op in self._operands:\n table = op.filter(table)\n return table\n\n return self._mask_table(\n table if column_slice is None else self._get_table_column(table, column_slice),\n self.mask(table)\n )", "response": "Returns a filtered version of the table."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the number of entries in table that satisfy the queries.", "response": "def count(self, table):\n \"\"\"\n Use the current Query object to count the number of entries in `table`\n that satisfy `queries`.\n\n Parameters\n ----------\n table : NumPy structured array, astropy Table, etc.\n\n Returns\n -------\n count : int\n \"\"\"\n if self._operator is None and self._operands is None:\n return self._get_table_len(table)\n\n return np.count_nonzero(self.mask(table))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef variable_names(self):\n if self._variable_names is None:\n\n if self._operator is None:\n if self._operands is None:\n self._variable_names = tuple()\n else:\n self._variable_names = self._get_variable_names(self._operands)\n\n elif self._operator == 'NOT':\n self._variable_names = self._operands.variable_names\n\n else:\n v = list()\n for op in self._operands:\n v.extend(op.variable_names)\n self._variable_names = tuple(set(v))\n\n return self._variable_names", "response": "Get all variable names required for this query."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef user(self):\n try:\n return self._user\n except AttributeError:\n self._user = MatrixUser(self.mxid, self.Api(identity=self.mxid))\n return self._user", "response": "Creates a User object when requested."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates a Room object when requested.", "response": "def room(self):\n \"\"\"Creates a Room object when requested.\"\"\"\n try:\n return self._room\n except AttributeError:\n room_id = self.json[\"room_id\"]\n self._room = MatrixRoom(room_id, self.Api)\n return self._room"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef join(self, room_str):\n response = self.user_api.join_room(room_str)\n return self._mkroom(response[\"room_id\"])", "response": "Joins a room id or alias even if it must first be created."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef refresh_rooms(self):\n for room_id in self.user_api.get_joined_rooms()[\"joined_rooms\"]:\n self._rooms[room_id] = MatrixRoom(room_id, self.user_api)", "response": "Calls GET / joined_rooms to refresh rooms list."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef not_empty(_object):\n if is_callable(_object):\n _validator = _object\n\n @wraps(_validator)\n @instance_of()\n def decorated(value):\n ensure(value, \"%s is empty\" % safe_repr(value))\n return _validator(value)\n return decorated\n try:\n ensure(len(_object), \"%s is empty\" % safe_repr(_object))\n except TypeError:\n raise AssertionError(\"not of any valid types: [list, dict, str]\")", "response": "Validates the given input is not empty."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef set_walltime(self, walltime):\n if not isinstance(walltime, timedelta):\n raise TypeError(\n 'walltime must be an instance of datetime.timedelta. %s given' %\n type(walltime)\n )\n\n self._options['walltime'] = walltime\n return self", "response": "Set a walltime for the job"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_url(url, parser='html'):\n url = request.quote(url, safe=':/?=&')\n logger.debug('URL: %s', url)\n req = request.Request(url, headers={'User-Agent': 'foobar'})\n try:\n response = request.urlopen(req)\n except HTTPError:\n raise\n except (ssl.SSLError, URLError):\n # Some websites (like metal-archives) use older TLS versions and can\n # make the ssl module trow a VERSION_TOO_LOW error. Here we try to use\n # the older TLSv1 to see if we can fix that\n context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)\n response = request.urlopen(req, context=context)\n\n response = response.read()\n if parser == 'html':\n return BeautifulSoup(response, 'html.parser', from_encoding='utf-8')\n elif parser == 'json':\n return json.loads(response)\n elif parser == 'raw':\n return response.decode()\n raise ValueError('Unrecognized parser')", "response": "Requests the specified url and returns a BeautifulSoup object with its\n contents."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_lastfm(method, lastfm_key='', **kwargs):\n if not lastfm_key:\n if 'lastfm_key' not in CONFIG or not CONFIG['lastfm_key']:\n logger.warning('No lastfm key configured')\n return ''\n else:\n lastfm_key = CONFIG['lastfm_key']\n\n url = 'http://ws.audioscrobbler.com/2.0/?method={}&api_key={}&format=json'\n url = url.format(method, lastfm_key)\n for key in kwargs:\n url += '&{}={}'.format(key, kwargs[key])\n\n response = get_url(url, parser='json')\n if 'error' in response:\n logger.error('Error number %d in lastfm query: %s',\n response['error'], response['message'])\n return ''\n\n return response", "response": "Request the specified method from the lastfm api."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef normalize(string, chars_to_remove=None, replacement=''):\n ret = string.translate(str.maketrans({\n '\u00e1': 'a',\n '\u00e4': 'a',\n '\u00e6': 'ae',\n '\u00e9': 'e',\n '\u00ed': 'i',\n '\u00f3': 'o',\n '\u00f6': 'o',\n '\u00fa': 'u',\n '\u00fc': 'u',\n '\u00f1': 'n',\n }))\n\n if isinstance(chars_to_remove, dict):\n for chars, replace in chars_to_remove.items():\n reg = '[' + re.escape(chars) + ']'\n ret = re.sub(reg, replace, ret)\n\n elif isinstance(chars_to_remove, str):\n reg = '[' + re.escape(chars_to_remove) + ']'\n ret = re.sub(reg, replacement, ret)\n\n return ret", "response": "Normalizes a string according to the given characters."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the lyrics found in metrolyrics for the specified mp3 file or an empty string if not found.", "response": "def metrolyrics(song):\n \"\"\"\n Returns the lyrics found in metrolyrics for the specified mp3 file or an\n empty string if not found.\n \"\"\"\n translate = {URLESCAPE: '', ' ': '-'}\n title = song.title.lower()\n title = normalize(title, translate)\n title = re.sub(r'\\-{2,}', '-', title)\n artist = song.artist.lower()\n artist = normalize(artist, translate)\n artist = re.sub(r'\\-{2,}', '-', artist)\n\n url = 'http://www.metrolyrics.com/{}-lyrics-{}.html'.format(title, artist)\n soup = get_url(url)\n body = soup.find(id='lyrics-body-text')\n if body is None:\n return ''\n\n text = ''\n verses = body.find_all('p')\n for verse in verses:\n text += verse.get_text().strip()\n text += '\\n\\n'\n\n return text.strip()"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the lyrics found in darklyrics for the specified mp3 file.", "response": "def darklyrics(song):\n \"\"\"\n Returns the lyrics found in darklyrics for the specified mp3 file or an\n empty string if not found.\n \"\"\"\n\n # Darklyrics relies on the album name\n if not hasattr(song, 'album') or not song.album:\n song.fetch_album_name()\n if not hasattr(song, 'album') or not song.album:\n # If we don't have the name of the album, there's nothing we can do\n # on darklyrics\n return ''\n\n artist = song.artist.lower()\n artist = normalize(artist, URLESCAPES, '')\n album = song.album.lower()\n album = normalize(album, URLESCAPES, '')\n title = song.title\n\n url = 'http://www.darklyrics.com/lyrics/{}/{}.html'.format(artist, album)\n soup = get_url(url)\n text = ''\n for header in soup.find_all('h3'):\n song = str(header.get_text())\n next_sibling = header.next_sibling\n if song.lower().find(title.lower()) != -1:\n while next_sibling is not None and\\\n (next_sibling.name is None or next_sibling.name != 'h3'):\n if next_sibling.name is None:\n text += str(next_sibling)\n next_sibling = next_sibling.next_sibling\n\n return text.strip()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef azlyrics(song):\n artist = song.artist.lower()\n if artist[0:2] == 'a ':\n artist = artist[2:]\n artist = normalize(artist, URLESCAPES, '')\n title = song.title.lower()\n title = normalize(title, URLESCAPES, '')\n\n url = 'https://www.azlyrics.com/lyrics/{}/{}.html'.format(artist, title)\n soup = get_url(url)\n body = soup.find_all('div', class_='')[-1]\n return body.get_text().strip()", "response": "Returns the lyrics found in azlyrics for the specified mp3 file."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the lyrics found in genius. com for the specified mp3 file or an empty string if not found.", "response": "def genius(song):\n \"\"\"\n Returns the lyrics found in genius.com for the specified mp3 file or an\n empty string if not found.\n \"\"\"\n translate = {\n '@': 'at',\n '&': 'and',\n URLESCAPE: '',\n ' ': '-'\n }\n artist = song.artist.capitalize()\n artist = normalize(artist, translate)\n title = song.title.capitalize()\n title = normalize(title, translate)\n\n url = 'https://www.genius.com/{}-{}-lyrics'.format(artist, title)\n soup = get_url(url)\n for content in soup.find_all('p'):\n if content:\n text = content.get_text().strip()\n if text:\n return text\n\n return ''"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef metalarchives(song):\n artist = normalize(song.artist)\n title = normalize(song.title)\n\n url = 'https://www.metal-archives.com/search/ajax-advanced/searching/songs'\n url += f'/?songTitle={title}&bandName={artist}&ExactBandMatch=1'\n soup = get_url(url, parser='json')\n if not soup:\n return ''\n\n song_id_re = re.compile(r'lyricsLink_([0-9]*)')\n ids = set(re.search(song_id_re, a) for sub in soup['aaData'] for a in sub)\n if not ids:\n return ''\n\n if None in ids:\n ids.remove(None)\n ids = map(lambda a: a.group(1), ids)\n for song_id in ids:\n url = 'https://www.metal-archives.com/release/ajax-view-lyrics/id/{}'\n lyrics = get_url(url.format(song_id), parser='html')\n lyrics = lyrics.get_text().strip()\n if not re.search('lyrics not available', lyrics):\n return lyrics\n\n return ''", "response": "Returns the lyrics found in MetalArchives for the specified mp3 file or an empty string if not found."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef lyricswikia(song):\n artist = song.artist.title()\n artist = normalize(artist, ' ', '_')\n title = song.title\n title = normalize(title, ' ', '_')\n\n url = 'https://lyrics.wikia.com/wiki/{}:{}'.format(artist, title)\n soup = get_url(url)\n text = ''\n content = soup.find('div', class_='lyricbox')\n if not content:\n return ''\n\n for unformat in content.findChildren(['i', 'b']):\n unformat.unwrap()\n for remove in content.findChildren(['div', 'span']):\n remove.decompose()\n\n nlcount = 0\n for line in content.children:\n if line is None or line == '<br/>' or line == '\\n':\n if nlcount == 2:\n text += '\\n\\n'\n nlcount = 0\n else:\n nlcount += 1\n else:\n nlcount = 0\n text += str(line).replace('<br/>', '\\n')\n return text.strip()", "response": "Returns the lyrics found in lyrics. wikia. com for the specified mp3 file or an empty string if not found."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the lyrics found in musixmatch for the specified mp3 file or an empty string if not found.", "response": "def musixmatch(song):\n \"\"\"\n Returns the lyrics found in musixmatch for the specified mp3 file or an\n empty string if not found.\n \"\"\"\n escape = re.sub(\"'-\u00a1\u00bf\", '', URLESCAPE)\n translate = {\n escape: '',\n ' ': '-'\n }\n artist = song.artist.title()\n artist = re.sub(r\"( '|' )\", '', artist)\n artist = re.sub(r\"'\", '-', artist)\n title = song.title\n title = re.sub(r\"( '|' )\", '', title)\n title = re.sub(r\"'\", '-', title)\n\n artist = normalize(artist, translate)\n artist = re.sub(r'\\-{2,}', '-', artist)\n title = normalize(title, translate)\n title = re.sub(r'\\-{2,}', '-', title)\n\n url = 'https://www.musixmatch.com/lyrics/{}/{}'.format(artist, title)\n soup = get_url(url)\n text = ''\n contents = soup.find_all('p', class_='mxm-lyrics__content')\n for p in contents:\n text += p.get_text().strip()\n if p != contents[-1]:\n text += '\\n\\n'\n\n return text.strip()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef songlyrics(song):\n translate = {\n URLESCAPE: '',\n ' ': '-'\n }\n artist = song.artist.lower()\n artist = normalize(artist, translate)\n title = song.title.lower()\n title = normalize(title, translate)\n\n artist = re.sub(r'\\-{2,}', '-', artist)\n title = re.sub(r'\\-{2,}', '-', title)\n\n url = 'http://www.songlyrics.com/{}/{}-lyrics'.format(artist, title)\n soup = get_url(url)\n text = soup.find(id='songLyricsDiv')\n if not text:\n return ''\n\n text = text.getText().strip()\n if not text or text.lower().startswith('we do not have the lyrics for'):\n return ''\n\n return text", "response": "Returns the lyrics found in songlyrics. com for the specified mp3 file or an empty string if not found."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the lyrics found in lyrics. com for the specified mp3 file or an empty string if not found.", "response": "def lyricscom(song):\n \"\"\"\n Returns the lyrics found in lyrics.com for the specified mp3 file or an\n empty string if not found.\n \"\"\"\n artist = song.artist.lower()\n artist = normalize(artist, ' ', '+')\n title = song.title\n\n url = 'https://www.lyrics.com/artist/{}'.format(artist)\n soup = get_url(url)\n location = ''\n for a in soup.select('tr a'):\n if a.string.lower() == title.lower():\n location = a['href']\n break\n if location == '':\n return ''\n\n url = 'https://www.lyrics.com' + location\n soup = get_url(url)\n body = soup.find(id='lyric-body-text')\n if not body:\n return ''\n\n return body.get_text().strip()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef vagalume(song):\n translate = {\n '@': 'a',\n URLESCAPE: '',\n ' ': '-'\n }\n artist = song.artist.lower()\n artist = normalize(artist, translate)\n artist = re.sub(r'\\-{2,}', '-', artist)\n title = song.title.lower()\n title = normalize(title, translate)\n title = re.sub(r'\\-{2,}', '-', title)\n\n url = 'https://www.vagalume.com.br/{}/{}.html'.format(artist, title)\n soup = get_url(url)\n body = soup.select('div#lyrics')\n if body == []:\n return ''\n\n content = body[0]\n for br in content.find_all('br'):\n br.replace_with('\\n')\n\n return content.get_text().strip()", "response": "Returns the lyrics found in vagalume. com. br for the specified mp3 file or an empty string if not found."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef lyricsmode(song):\n translate = {\n URLESCAPE: '',\n ' ': '_'\n }\n artist = song.artist.lower()\n artist = normalize(artist, translate)\n title = song.title.lower()\n title = normalize(title, translate)\n\n artist = re.sub(r'\\_{2,}', '_', artist)\n title = re.sub(r'\\_{2,}', '_', title)\n\n if artist[0:4].lower() == 'the ':\n artist = artist[4:]\n\n if artist[0:2].lower() == 'a ':\n prefix = artist[2]\n else:\n prefix = artist[0]\n\n url = 'http://www.lyricsmode.com/lyrics/{}/{}/{}.html'\n url = url.format(prefix, artist, title)\n soup = get_url(url)\n content = soup.find(id='lyrics_text')\n\n return content.get_text().strip()", "response": "Returns the lyrics found in lyricsmode. com for the specified mp3 file."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the lyrics found in letras. com for the specified mp3 file or an empty string if not found.", "response": "def letras(song):\n \"\"\"\n Returns the lyrics found in letras.com for the specified mp3 file or an\n empty string if not found.\n \"\"\"\n translate = {\n '&': 'a',\n URLESCAPE: '',\n ' ': '-'\n }\n artist = song.artist.lower()\n artist = normalize(artist, translate)\n title = song.title.lower()\n title = normalize(title, translate)\n\n url = 'https://www.letras.com/{}/{}/'.format(artist, title)\n soup = get_url(url)\n if not soup:\n return ''\n\n found_title = soup.select_one('div.cnt-head_title h1')\n if not found_title:\n # The site didn't find lyrics and took us to the homepage\n return ''\n\n found_title = found_title.get_text()\n found_title = re.sub(r'[\\W_]+', '', found_title.lower())\n if found_title != re.sub(r'[\\W_]+', '', song.title.lower()):\n # The site took us to the wrong song page\n return ''\n\n content = soup.find('article')\n if not content:\n return ''\n\n text = ''\n for br in content.find_all('br'):\n br.replace_with('\\n')\n\n for p in content.find_all('p'):\n text += p.get_text() + '\\n\\n'\n\n return text.strip()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the name of a website - scrapping function.", "response": "def id_source(source, full=False):\n \"\"\"\n Returns the name of a website-scrapping function.\n \"\"\"\n if source not in source_ids:\n return ''\n\n if full:\n return source_ids[source][1]\n else:\n return source_ids[source][0]"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nupdating metrics for a single gitlab instance.", "response": "async def update_gitlab_loop(update_metrics, params):\n \"\"\"\n app = Sanic()\n mserver = MetricsServer(app)\n mserver.add_task(update_gitlab_loop, params={'url': GITLAB_URL, 'token': token})\n \"\"\"\n gitlab_api = gitlab.Gitlab(url=params['url'], private_token=params['token'], api_version=4)\n while True:\n try:\n metrics = update_gitlab_runners(gitlab_api)\n update_metrics(metrics)\n except Exception:\n update_metrics({})\n log.exception('During loop')\n await asyncio.sleep(LOOP)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_probe_position(self, new_probe_position):\n if new_probe_position is not None:\n # convert the probe position to a FloatPoint and limit it to the 0.0 to 1.0 range in both axes.\n new_probe_position = Geometry.FloatPoint.make(new_probe_position)\n new_probe_position = Geometry.FloatPoint(y=max(min(new_probe_position.y, 1.0), 0.0),\n x=max(min(new_probe_position.x, 1.0), 0.0))\n old_probe_position = self.__probe_position_value.value\n if ((old_probe_position is None) != (new_probe_position is None)) or (old_probe_position != new_probe_position):\n # this path is only taken if set_probe_position is not called as a result of the probe_position model\n # value changing.\n self.__probe_position_value.value = new_probe_position\n # update the probe position for listeners and also explicitly update for probe_graphic_connections.\n self.probe_state_changed_event.fire(self.probe_state, self.probe_position)", "response": "Sets the probe position in normalized coordinates with origin at top left."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef apply_metadata_groups(self, properties: typing.Dict, metatdata_groups: typing.Tuple[typing.List[str], str]) -> None:\n pass", "response": "Applies metadata groups to properties."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_analytic_backend(settings):\n backend = settings.get('backend')\n if isinstance(backend, basestring):\n backend = import_string(backend)\n elif backend:\n backend = backend\n else:\n raise KeyError('backend')\n\n return backend(settings.get(\"settings\", {}))", "response": "Creates a new Analytics backend from the settings"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef initialize_state(self):\n if self.__scan_hardware_source:\n self.__profile_changed_event_listener = self.__scan_hardware_source.profile_changed_event.listen(self.__update_profile_index)\n self.__frame_parameters_changed_event_listener = self.__scan_hardware_source.frame_parameters_changed_event.listen(self.__update_frame_parameters)\n self.__data_item_states_changed_event_listener = self.__scan_hardware_source.data_item_states_changed_event.listen(self.__data_item_states_changed)\n self.__acquisition_state_changed_event_listener = self.__scan_hardware_source.acquisition_state_changed_event.listen(self.__acquisition_state_changed)\n self.__probe_state_changed_event_listener = self.__scan_hardware_source.probe_state_changed_event.listen(self.__probe_state_changed)\n self.__channel_state_changed_event_listener = self.__scan_hardware_source.channel_state_changed_event.listen(self.__channel_state_changed)\n subscan_state_model = self.__scan_hardware_source.subscan_state_model\n\n def subscan_state_changed(name):\n if callable(self.on_subscan_state_changed):\n self.on_subscan_state_changed(subscan_state_model.value)\n\n self.__subscan_state_changed_listener = subscan_state_model.property_changed_event.listen(subscan_state_changed)\n subscan_state_changed(\"value\")\n if self.on_display_name_changed:\n self.on_display_name_changed(self.display_name)\n if self.on_subscan_state_changed:\n self.on_subscan_state_changed(self.__scan_hardware_source.subscan_state)\n channel_count = self.__scan_hardware_source.channel_count\n if self.on_channel_count_changed:\n self.on_channel_count_changed(channel_count)\n self.__channel_enabled = [False] * channel_count\n for channel_index in range(channel_count):\n channel_id, name, enabled = self.__scan_hardware_source.get_channel_state(channel_index)\n self.__channel_state_changed(channel_index, channel_id, name, enabled)\n self.__channel_enabled[channel_index] = enabled\n self.__update_buttons()\n if self.on_profiles_changed:\n profile_items = list(ScanControlStateController.profiles.items())\n profile_items.sort(key=lambda k_v: k_v[1])\n profiles = map(lambda k_v: k_v[0], profile_items)\n self.on_profiles_changed(profiles)\n self.__update_profile_index(self.__scan_hardware_source.selected_profile_index)\n if self.on_linked_changed:\n self.on_linked_changed(self.__linked)\n if self.on_simulate_button_state_changed:\n use_simulator = self.__scan_hardware_source.use_hardware_simulator\n self.on_simulate_button_state_changed(use_simulator, use_simulator)\n if self.on_data_item_states_changed:\n self.on_data_item_states_changed(list())\n probe_state = self.__scan_hardware_source.probe_state\n probe_position = self.__scan_hardware_source.probe_position\n self.__probe_state_changed(probe_state, probe_position)", "response": "Initialize the state of the UI."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncalling this when the user clicks the play or pause button.", "response": "def handle_play_pause_clicked(self):\n \"\"\" Call this when the user clicks the play/pause button. \"\"\"\n if self.__scan_hardware_source:\n if self.is_playing:\n self.__scan_hardware_source.stop_playing()\n else:\n self.__scan_hardware_source.start_playing()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef handle_record_clicked(self, callback_fn):\n assert callable(callback_fn)\n if self.__scan_hardware_source:\n\n def finish_record(data_and_metadata_list):\n record_index = self.__scan_hardware_source.record_index\n for data_and_metadata in data_and_metadata_list:\n data_item = DataItem.DataItem()\n data_item.ensure_data_source()\n\n display_name = data_and_metadata.metadata.get(\"hardware_source\", dict()).get(\"hardware_source_name\")\n display_name = display_name if display_name else _(\"Record\")\n channel_name = data_and_metadata.metadata.get(\"hardware_source\", dict()).get(\"channel_name\")\n title_base = \"{} ({})\".format(display_name, channel_name) if channel_name else display_name\n data_item.title = \"{} {}\".format(title_base, record_index)\n\n data_item.set_xdata(data_and_metadata)\n callback_fn(data_item)\n self.__scan_hardware_source.record_index += 1\n\n self.__scan_hardware_source.record_async(finish_record)", "response": "Handle the record button clicked."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef size_to_content(self, horizontal_padding=None, vertical_padding=None):\n\n if horizontal_padding is None:\n horizontal_padding = 0\n\n if vertical_padding is None:\n vertical_padding = 0\n\n self.sizing.set_fixed_size(Geometry.IntSize(18 + 2 * horizontal_padding, 18 + 2 * vertical_padding))", "response": "Size the canvas item to the text content."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef round_controlled(cycled_iterable, rounds=1):\n round_start = None\n rounds_completed = 0\n\n for item in cycled_iterable:\n if round_start is None:\n round_start = item\n elif item == round_start:\n rounds_completed += 1\n\n if rounds_completed == rounds:\n return\n\n yield item", "response": "Return after <rounds > passes through a cycled iterable."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef success_rate(self):\n if self.successes + self.fails == 0:\n success_rate = 0\n else:\n total_attempts = self.successes + self.fails\n success_rate = (self.successes * 100 / total_attempts)\n\n return success_rate", "response": "Returns a float with the rate of success from all the logged results."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add_result(self, source, found, runtime):\n self.source_stats[source.__name__].add_runtime(runtime)\n if found:\n self.source_stats[source.__name__].successes += 1\n else:\n self.source_stats[source.__name__].fails += 1", "response": "Adds a new record to the statistics database."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the average time taken to scrape the lyrics from a source or the total average time taken to scrape the lyrics from that source.", "response": "def avg_time(self, source=None):\n \"\"\"\n Returns the average time taken to scrape lyrics. If a string or a\n function is passed as source, return the average time taken to scrape\n lyrics from that source, otherwise return the total average.\n \"\"\"\n if source is None:\n runtimes = []\n for rec in self.source_stats.values():\n runtimes.extend([r for r in rec.runtimes if r != 0])\n return avg(runtimes)\n else:\n if callable(source):\n return avg(self.source_stats[source.__name__].runtimes)\n else:\n return avg(self.source_stats[source].runtimes)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncalculates the overall counts of best worst fastest slowest total not found and total runtime.", "response": "def calculate(self):\n \"\"\"\n Calculate the overall counts of best, worst, fastest, slowest, total\n found, total not found and total runtime\n\n Results are returned in a dictionary with the above parameters as keys.\n \"\"\"\n best, worst, fastest, slowest = (), (), (), ()\n found = notfound = total_time = 0\n for source, rec in self.source_stats.items():\n if not best or rec.successes > best[1]:\n best = (source, rec.successes, rec.success_rate())\n if not worst or rec.successes < worst[1]:\n worst = (source, rec.successes, rec.success_rate())\n\n avg_time = self.avg_time(source)\n if not fastest or (avg_time != 0 and avg_time < fastest[1]):\n fastest = (source, avg_time)\n if not slowest or (avg_time != 0 and avg_time > slowest[1]):\n slowest = (source, avg_time)\n\n found += rec.successes\n notfound += rec.fails\n total_time += sum(rec.runtimes)\n\n return {\n 'best': best,\n 'worst': worst,\n 'fastest': fastest,\n 'slowest': slowest,\n 'found': found,\n 'notfound': notfound,\n 'total_time': total_time\n }"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nprint a series of relevant stats about a full execution. This function is meant to be called at the end of the program.", "response": "def print_stats(self):\n \"\"\"\n Print a series of relevant stats about a full execution. This function\n is meant to be called at the end of the program.\n \"\"\"\n stats = self.calculate()\n total_time = '%d:%02d:%02d' % (stats['total_time'] / 3600,\n (stats['total_time'] / 3600) / 60,\n (stats['total_time'] % 3600) % 60)\n output = \"\"\"\\\nTotal runtime: {total_time}\n Lyrics found: {found}\n Lyrics not found:{notfound}\n Most useful source:\\\n{best} ({best_count} lyrics found) ({best_rate:.2f}% success rate)\n Least useful source:\\\n{worst} ({worst_count} lyrics found) ({worst_rate:.2f}% success rate)\n Fastest website to scrape: {fastest} (Avg: {fastest_time:.2f}s per search)\n Slowest website to scrape: {slowest} (Avg: {slowest_time:.2f}s per search)\n Average time per website: {avg_time:.2f}s\n\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxx PER WEBSITE STATS: xxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\"\"\"\n output = output.format(total_time=total_time,\n found=stats['found'],\n notfound=stats['notfound'],\n best=stats['best'][0].capitalize(),\n best_count=stats['best'][1],\n best_rate=stats['best'][2],\n worst=stats['worst'][0].capitalize(),\n worst_count=stats['worst'][1],\n worst_rate=stats['worst'][2],\n fastest=stats['fastest'][0].capitalize(),\n fastest_time=stats['fastest'][1],\n slowest=stats['slowest'][0].capitalize(),\n slowest_time=stats['slowest'][1],\n avg_time=self.avg_time())\n for source in sources:\n stat = str(self.source_stats[source.__name__])\n output += f'\\n{source.__name__.upper()}\\n{stat}\\n'\n\n print(output)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef broadcast(cls, s1: ParamsList, s2: ParamsList) -> BroadcastTuple:\n '''It broadcasts the smaller scope over the larger scope.\n\n It handles scope intersection as well as differences in scopes\n in order to output a resulting scope so that input scopes are\n contained within it (i.e., input scopes are subscopes of the\n output scope). Also, if necessary, it outputs permutations of\n the input scopes so that tensor broadcasting invariants are\n not violated.\n\n Note:\n For more information on broadcasting, please report to\n NumPy's official documentation available at the following URLs:\n 1. https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html\n 2. https://docs.scipy.org/doc/numpy/reference/generated/numpy.broadcast.html\n\n Args:\n s1: A fluent's scope.\n s2: A fluent's scope.\n\n Returns:\n A tuple with the output scope and permutations of the input scopes.\n '''\n if len(s1) == 0:\n return s2, [], []\n if len(s2) == 0:\n return s1, [], []\n\n subscope = list(set(s1) & set(s2))\n if len(subscope) == len(s1):\n subscope = s1\n elif len(subscope) == len(s2):\n subscope = s2\n\n perm1 = []\n if s1[-len(subscope):] != subscope:\n i = 0\n for var in s1:\n if var not in subscope:\n perm1.append(i)\n i += 1\n else:\n j = subscope.index(var)\n perm1.append(len(s1) - len(subscope) + j)\n perm2 = []\n if s2[-len(subscope):] != subscope:\n i = 0\n for var in s2:\n if var not in subscope:\n perm2.append(i)\n i += 1\n else:\n j = subscope.index(var)\n perm2.append(len(s2) - len(subscope) + j)\n\n scope = [] # type: ParamsList\n if len(s1) >= len(s2):\n if perm1 == []:\n scope = s1\n else:\n for i in range(len(s1)):\n scope.append(s1[perm1.index(i)])\n else:\n if perm2 == []:\n scope = s2\n else:\n for i in range(len(s2)):\n scope.append(s2[perm2.index(i)])\n\n return (scope, perm1, perm2)", "response": "This function broadcasts the smaller scope over the larger scope. It broadcasts the smaller scope over the larger scope. It handles scope intersection as well as differences in scopes\n and outputs a resulting scope so that input scopes are subscopes of the resulting scope."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_baseparser_extended_df(sample, bp_lines, ref, alt):\n columns = \"chrom\\tpos\\tref\\tcov\\tA\\tC\\tG\\tT\\t*\\t-\\t+\".split()\n if bp_lines is None:\n return None\n\n # change baseparser output to get most common maf per indel\n bpdf = pd.DataFrame([[sample] + l.rstrip('\\n').split(\"\\t\") for l in bp_lines if len(l) > 0],\n columns=[\"sample\"] + columns, dtype=np.object)\n bpdf[bpdf == \"\"] = None\n\n # remove zero coverage rows\n bpdf = bpdf[bpdf[\"cov\"].astype(int) > 0]\n\n if len(bpdf) == 0:\n return None\n\n if ref and alt:\n # add columns for validation allele\n bpdf = pd.concat([bpdf, pd.DataFrame({\"val_ref\": pd.Series(ref), \"val_alt\": pd.Series(alt)})], axis=1)\n bpdf = pd.concat([bpdf, bpdf.apply(_val_al, axis=1)], axis=1)\n\n bpdf = pd.concat([bpdf, bpdf.apply(_most_common_indel, axis=1)], axis=1)\n bpdf = pd.concat([bpdf, bpdf.apply(_most_common_al, axis=1)], axis=1)\n bpdf[\"most_common_count\"] = bpdf.apply(lambda x: max([x.most_common_al_count, x.most_common_indel_count]), axis=1)\n bpdf[\"most_common_maf\"] = bpdf.apply(lambda x: max([x.most_common_al_maf, x.most_common_indel_maf]), axis=1)\n\n return bpdf", "response": "Turn baseParser results into a dataframe"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfilter out mutations that are in normal.", "response": "def filter_out_mutations_in_normal(tumordf, normaldf, most_common_maf_min=0.2,\n most_common_count_maf_threshold=20,\n most_common_count_min=1):\n \"\"\"Remove mutations that are in normal\"\"\"\n df = tumordf.merge(normaldf, on=[\"chrom\", \"pos\"], suffixes=(\"_T\", \"_N\"))\n\n # filters\n common_al = (df.most_common_al_count_T == df.most_common_count_T) & (df.most_common_al_T == df.most_common_al_N)\n common_indel = (df.most_common_indel_count_T == df.most_common_count_T) & \\\n (df.most_common_indel_T == df.imost_common_indel_N)\n normal_criteria = ((df.most_common_count_N >= most_common_count_maf_threshold) &\n (df.most_common_maf_N > most_common_maf_min)) | \\\n ((df.most_common_count_N < most_common_count_maf_threshold) &\n (df.most_common_count_N > most_common_count_min))\n df = df[~(common_al | common_indel) & normal_criteria]\n\n # restore column names of tumor\n for c in df.columns:\n if c.endswith(\"_N\"):\n del df[c]\n df.columns = [c[:-2] if c.endswith(\"_T\") else c for c in df.columns]\n\n return df"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nselects only mutations that revert the given mutations in a single event.", "response": "def select_only_revertant_mutations(bpdf, snv=None, ins=None, dlt=None):\n \"\"\"\n Selects only mutations that revert the given mutations in a single event.\n \"\"\"\n if sum([bool(snv), bool(ins), bool(dlt)]) != 1:\n raise(Exception(\"Should be either snv, ins or del\".format(snv)))\n\n if snv:\n if snv not in [\"A\", \"C\", \"G\", \"T\"]:\n raise(Exception(\"snv {} should be A, C, G or T\".format(snv)))\n return bpdf[(bpdf.most_common_al == snv) & (bpdf.most_common_al_count == bpdf.most_common_count)]\n elif bool(ins):\n return \\\n bpdf[((bpdf.most_common_indel.apply(lambda x: len(x) + len(ins) % 3 if x else None) == 0) &\n (bpdf.most_common_indel_type == \"+\") & (bpdf.most_common_count == bpdf.most_common_indel_count)) |\n ((bpdf.most_common_indel.apply(lambda x: len(ins) - len(x) % 3 if x else None) == 0) &\n (bpdf.most_common_indel_type == \"-\") & (bpdf.most_common_count == bpdf.most_common_indel_count))]\n elif bool(dlt):\n return \\\n bpdf[((bpdf.most_common_indel.apply(lambda x: len(x) - len(dlt) % 3 if x else None) == 0) &\n (bpdf.most_common_indel_type == \"+\") & (bpdf.most_common_count == bpdf.most_common_indel_count)) |\n ((bpdf.most_common_indel.apply(lambda x: -len(dlt) - len(x) % 3 if x else None) == 0) &\n (bpdf.most_common_indel_type == \"-\") & (bpdf.most_common_count == bpdf.most_common_indel_count))]\n else:\n # should never happen\n raise(Exception(\"No mutation given?\"))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nvalidate mutations in vcf file.", "response": "def validate_mutations(vcffile, bams, reffa, chr_reffa, samples, output_format, outfile,\n mpileup_parameters=mpileup_parser.MPILEUP_DEFAULT_PARAMS):\n \"\"\"Check if mutations in vcf are in bam\"\"\"\n output_header = \"sample chrom pos ref cov A C G T * - + \" \\\n \"val_ref val_alt val_al_type val_al_count val_maf \"\\\n \"most_common_indel most_common_indel_count most_common_indel_maf most_common_indel_type most_common_al \" \\\n \"most_common_al_count most_common_al_maf most_common_count most_common_maf\".split()\n\n # for backwards compatibility\n # if bam or samples is a string, convert to list instead\n if isinstance(samples, six.string_types):\n samples = [samples]\n if isinstance(bams, six.string_types):\n bams = [bams]\n\n if output_format == 'vcf':\n vcf_reader = vcf.Reader(open(vcffile))\n vcf_reader.samples = samples\n vcf_reader.formats['GT'] = vcf.parser._Format(id='GT', num=1, type='String', desc=\"Genotype\")\n vcf_reader.formats['AD'] = vcf.parser._Format(id='AD', num='R', type='Integer', desc=\"Allelic depth\")\n vcf_reader.formats['DP'] = vcf.parser._Format(id='DP', num=1, type='Integer', desc=\"Depth\")\n vcf_writer = vcf.Writer(outfile, vcf_reader)\n else:\n vcf_reader = open(vcffile)\n\n\n if output_format == \"sufam\":\n outfile.write(\"\\t\".join(output_header))\n outfile.write(\"\\n\")\n for record in vcf_reader:\n if output_format != 'vcf':\n line = record\n if line.startswith(\"#CHROM\"):\n header = line[1:].rstrip('\\n').split(\"\\t\")\n # create spoof pyvcf record if vcf_reader is not used\n _Record = namedtuple('Record', header)\n if line.startswith(\"#\"):\n continue\n if len(header) == 0:\n raise(Exception(\"No header found in vcf file #CHROM not found\"))\n # zip all column values, except alt (needs to be list in pyvcf)\n record_args = dict(zip(header, line.rstrip('\\n').split(\"\\t\")))\n record_args['ALT'] = [record_args['ALT']]\n record = _Record(**record_args)\n\n # determine type of mutation\n record_type = \"snv\"\n if len(record.ALT) > 1:\n warnings.warn(\"Multiple ALT in one record is not implemented - using first\")\n if len(record.REF) > len(record.ALT[0]):\n record_type = \"deletion\"\n elif len(record.ALT[0]) > len(record.REF):\n record_type = \"insertion\"\n\n # no coverage results\n no_cov = pd.Series({\n \"chrom\": str(record.CHROM), \"pos\": str(record.POS),\n \"ref\": str(record.REF),\n \"cov\": 0, \"A\": 0, \"C\": 0, \"G\": 0, \"T\": 0,\n \"val_ref\": str(record.REF), \"val_alt\": str(record.ALT[0]),\n \"val_al_type\": record_type, \"val_al_count\": 0, \"val_maf\": 0})\n\n # collect mpileup baseparser results per bam\n bps = []\n for i, bam in enumerate(bams):\n sample = samples[i]\n no_cov['sample'] = sample\n bp_lines = mpileup_parser.run_and_parse(bam, str(record.CHROM), str(record.POS), str(record.POS), reffa,\n chr_reffa, mpileup_parameters)\n bpdf = get_baseparser_extended_df(sample, bp_lines, str(record.REF), str(record.ALT[0]))\n if bpdf is None:\n bp = no_cov\n else:\n bp = bpdf.ix[0, :]\n bps += [bp]\n\n # output call\n if output_format == \"vcf\":\n _write_bp_vcf(outfile, bps, vcf_writer, record)\n else:\n # only one bam file supported for outputs other than vcf\n _write_bp(outfile, bps[0], output_header, output_format)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _prep_clients(self, clients):\n for pool_id, client in enumerate(clients):\n # Tag it with an id we'll use to identify it in the pool\n if hasattr(client, \"pool_id\"):\n raise ValueError(\"%r is already part of a pool.\", client)\n setattr(client, \"pool_id\", pool_id)\n # Wrap all public functions\n self._wrap_functions(client)\n return clients", "response": "Prepare a list of clients by tagging it with and id and wrapping methods."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nwrap public functions to catch ConnectionError and TimeoutError.", "response": "def _wrap_functions(self, client):\n \"\"\"Wrap public functions to catch ConnectionError.\n\n When an error happens, it puts the client in the penalty box\n so that it won't be retried again for a little while.\n \"\"\"\n\n def wrap(fn):\n def wrapper(*args, **kwargs):\n \"\"\"Simple wrapper for to catch dead clients.\"\"\"\n try:\n return fn(*args, **kwargs)\n except (ConnectionError, TimeoutError): # TO THE PENALTY BOX!\n self._penalize_client(client)\n raise\n\n return functools.update_wrapper(wrapper, fn)\n\n for name in dir(client):\n if name.startswith(\"_\"):\n continue\n # Some things aren't wrapped\n if name in (\"echo\", \"execute_command\", \"parse_response\"):\n continue\n obj = getattr(client, name)\n if not callable(obj):\n continue\n log.debug(\"Wrapping %s\", name)\n setattr(client, name, wrap(obj))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nrestores clients that have reconnected.", "response": "def _prune_penalty_box(self):\n \"\"\"Restores clients that have reconnected.\n\n This function should be called first for every public method.\n \"\"\"\n added = False\n for client in self.penalty_box.get():\n log.info(\"Client %r is back up.\", client)\n self.active_clients.append(client)\n added = True\n if added:\n self._sort_clients()"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_client(self, shard_key):\n self._prune_penalty_box()\n\n if len(self.active_clients) == 0:\n raise ClusterEmptyError(\"All clients are down.\")\n\n # So that hashing is consistent when a node is down, check against\n # the initial client list. Only use the active client list when\n # the desired node is down.\n # N.B.: I know this is not technically \"consistent hashing\" as\n # academically defined. It's a hack so that keys which need to\n # go elsewhere do, while the rest stay on the same instance.\n if not isinstance(shard_key, bytes):\n shard_key = shard_key.encode(\"utf-8\")\n hashed = mmh3.hash(shard_key)\n pos = hashed % len(self.initial_clients)\n if self.initial_clients[pos] in self.active_clients:\n return self.initial_clients[pos]\n else:\n pos = hashed % len(self.active_clients)\n return self.active_clients[pos]", "response": "Get the client for a given shard."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nplace client in the penalty box.", "response": "def _penalize_client(self, client):\n \"\"\"Place client in the penalty box.\n\n :param client: Client object\n \"\"\"\n if client in self.active_clients: # hasn't been removed yet\n log.warning(\"%r marked down.\", client)\n self.active_clients.remove(client)\n self.penalty_box.add(client)\n else:\n log.info(\"%r not in active client list.\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting the zrevrangebyscore across the cluster.", "response": "def zrevrange_with_int_score(self, key, max_score, min_score):\n \"\"\"Get the zrevrangebyscore across the cluster.\n Highest score for duplicate element is returned.\n A faster method should be written if scores are not needed.\n \"\"\"\n self._prune_penalty_box()\n\n if len(self.active_clients) == 0:\n raise ClusterEmptyError(\"All clients are down.\")\n\n element__score = defaultdict(int)\n for client in self.active_clients:\n revrange = client.zrevrangebyscore(\n key, max_score, min_score, withscores=True, score_cast_func=int\n )\n\n for element, count in revrange:\n element__score[element] = max(element__score[element], int(count))\n\n return element__score"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntaking files from React side creates SolveBio Object containing signed S3 URL.", "response": "def generate_s3_url(files):\n \"\"\"Takes files from React side, creates\n SolveBio Object containing signed S3 URL.\"\"\"\n if files:\n vault = g.client.Vault.get_personal_vault()\n files = json.loads(files)\n objects = []\n for i in xrange(len(files)):\n obj = g.client.Object.create(\n vault_id=vault.id,\n object_type='file',\n filename=files[i].get('filename'),\n mimetype=files[i].get('mimetype'),\n size=files[i].get('size')\n )\n objects.append({\n 'id': obj.id,\n 'filename': obj.filename,\n 'upload_url': obj.upload_url\n })\n return json.dumps(objects)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nhandle downstream processes using metadata about the uploaded files from React side.", "response": "def handle_uploaded_files(uploaded_files):\n \"\"\"Handles downstream processes using metadata\n about the uploaded files from React side.\"\"\"\n if uploaded_files:\n uploaded_files = json.loads(uploaded_files)[0]\n _id = uploaded_files.get('id')\n # Strip extension from filename\n _filename = os.path.splitext(uploaded_files.get('filename'))[0]\n\n # Create a dataset\n dataset = g.client.Dataset.get_or_create_by_full_path('~/' + _filename)\n\n # Import the file into the dataset\n g.client.DatasetImport.create(\n dataset_id=dataset.id,\n object_id=_id\n )\n\n # Wait until activity is completed\n dataset.activity(follow=True)\n\n SELECTED_COLS = ['col_a', 'col_b', 'col_c']\n query = dataset.query(fields=SELECTED_COLS)\n\n return html.Div(\n dt.DataTable(\n id='data-table',\n rows=list(query),\n columns=SELECTED_COLS\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a custom camera panel.", "response": "def create_camera_panel(document_controller, panel_id, properties):\n \"\"\"Create a custom camera panel.\n\n The camera panel type is specified in the 'camera_panel_type' key in the properties dict.\n\n The camera panel type must match a the 'camera_panel_type' of a camera panel factory in the Registry.\n\n The matching camera panel factory must return a ui_handler for the panel which is used to produce the UI.\n \"\"\"\n camera_panel_type = properties.get(\"camera_panel_type\")\n for component in Registry.get_components_by_type(\"camera_panel\"):\n if component.camera_panel_type == camera_panel_type:\n hardware_source_id = properties[\"hardware_source_id\"]\n hardware_source = HardwareSource.HardwareSourceManager().get_hardware_source_for_hardware_source_id(hardware_source_id)\n camera_device = getattr(hardware_source, \"camera\", None)\n camera_settings = getattr(hardware_source, \"camera_settings\", None)\n ui_handler = component.get_ui_handler(api_broker=PlugInManager.APIBroker(), event_loop=document_controller.event_loop, hardware_source_id=hardware_source_id, camera_device=camera_device, camera_settings=camera_settings)\n panel = Panel.Panel(document_controller, panel_id, properties)\n panel.widget = Declarative.DeclarativeWidget(document_controller.ui, document_controller.event_loop, ui_handler)\n return panel\n return None"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ninitializing the state of the class.", "response": "def initialize_state(self):\n \"\"\" Call this to initialize the state of the UI after everything has been connected. \"\"\"\n if self.__hardware_source:\n self.__profile_changed_event_listener = self.__hardware_source.profile_changed_event.listen(self.__update_profile_index)\n self.__frame_parameters_changed_event_listener = self.__hardware_source.frame_parameters_changed_event.listen(self.__update_frame_parameters)\n self.__data_item_states_changed_event_listener = self.__hardware_source.data_item_states_changed_event.listen(self.__data_item_states_changed)\n self.__acquisition_state_changed_event_listener = self.__hardware_source.acquisition_state_changed_event.listen(self.__acquisition_state_changed)\n self.__log_messages_event_listener = self.__hardware_source.log_messages_event.listen(self.__log_messages)\n if self.on_display_name_changed:\n self.on_display_name_changed(self.display_name)\n if self.on_binning_values_changed:\n self.on_binning_values_changed(self.__hardware_source.binning_values)\n if self.on_monitor_button_state_changed:\n has_monitor = self.__hardware_source and self.__hardware_source.features.get(\"has_monitor\", False)\n self.on_monitor_button_state_changed(has_monitor, has_monitor)\n self.__update_buttons()\n if self.on_profiles_changed:\n profile_items = self.__hardware_source.modes\n self.on_profiles_changed(profile_items)\n self.__update_profile_index(self.__hardware_source.selected_profile_index)\n if self.on_data_item_states_changed:\n self.on_data_item_states_changed(list())"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncall this when the user clicks the play or pause button.", "response": "def handle_play_pause_clicked(self):\n \"\"\" Call this when the user clicks the play/pause button. \"\"\"\n if self.__hardware_source:\n if self.is_playing:\n self.__hardware_source.stop_playing()\n else:\n self.__hardware_source.start_playing()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef resolve_expression(self, query=None, allow_joins=True, reuse=None, summarize=False, for_save=False):\n c = self.copy()\n c.is_summary = summarize\n c.for_save = for_save\n final_points = []\n for i, p in enumerate(self.params):\n try:\n float(p)\n except:\n _, source, _, join_list, last = query.setup_joins(\n six.text_type(p).split('__'), query.model._meta, query.get_initial_alias())[:5]\n target, alias, _ = query.trim_joins(source, join_list, last)\n final_points.append(\"%s.%s\" % (alias, target[0].get_attname_column()[1]))\n else:\n final_points.append(six.text_type(p))\n c.params = final_points\n return c", "response": "Setup any data here this method will be called before final SQL is generated"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef resolve_expression(self, query=None, allow_joins=True, reuse=None, summarize=False, for_save=False):\n c = self.copy()\n c.is_summary = summarize\n c.for_save = for_save\n for pos, expression in enumerate(self.expressions):\n c.expressions[pos] = expression.resolve_expression(query, allow_joins, reuse, summarize)\n return c", "response": "Prepare SQL from inner funcions."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfiltering rows inside a circunference of radius distance distance", "response": "def in_distance(self, distance, fields, points, annotate='_ed_distance'):\n \"\"\"Filter rows inside a circunference of radius distance `distance`\n\n :param distance: max distance to allow\n :param fields: `tuple` with the fields to filter (latitude, longitude)\n :param points: center of the circunference (latitude, longitude)\n :param annotate: name where the distance will be annotated\n\n \"\"\"\n clone = self._clone()\n return clone.annotate(\n **{annotate: EarthDistance([\n LlToEarth(fields), LlToEarth(points)])\n }).filter(**{'{0}__lte'.format(annotate): distance})"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets a value for the specified configuration key.", "response": "def get(self, path, default=_NoDefault, as_type=None, resolve_references=True):\n \"\"\"\n Gets a value for the specified path.\n\n :param path: the configuration key to fetch a value for, steps\n separated by the separator supplied to the constructor (default\n ``.``)\n :param default: a value to return if no value is found for the\n supplied path (``None`` is allowed)\n :param as_type: an optional callable to apply to the value found for\n the supplied path (possibly raising exceptions of its own if the\n value can not be coerced to the expected type)\n :param resolve_references: whether to resolve references in values\n :return: the value associated with the supplied configuration key, if\n available, or a supplied default value if the key was not found\n :raises ConfigurationError: when no value was found for *path* and\n *default* was not provided or a reference could not be resolved\n \"\"\"\n value = self._source\n steps_taken = []\n try:\n # walk through the values dictionary\n for step in path.split(self._separator):\n steps_taken.append(step)\n value = value[step]\n\n if as_type:\n return as_type(value)\n elif isinstance(value, Mapping):\n # create an instance of our current type, copying 'configured' properties / policies\n namespace = type(self)(separator=self._separator, missing=self._missing)\n namespace._source = value\n # carry the root object from namespace to namespace, references are always resolved from root\n namespace._root = self._root\n return namespace\n elif resolve_references and isinstance(value, str):\n # only resolve references in str-type values (the only way they can be expressed)\n return self._resolve(value)\n else:\n return value\n except ConfiguredReferenceError:\n # also a KeyError, but this one should bubble to caller\n raise\n except KeyError as e:\n if default is not _NoDefault:\n return default\n else:\n missing_key = self._separator.join(steps_taken)\n raise NotConfiguredError('no configuration for key {}'.format(missing_key), key=missing_key) from e"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef record_async(self, callback_fn):\n assert callable(callback_fn)\n\n def record_thread():\n current_frame_time = self.get_current_frame_time()\n\n def handle_finished(xdatas):\n callback_fn(xdatas)\n\n self.start_recording(current_frame_time, finished_callback_fn=handle_finished)\n\n self.__thread = threading.Thread(target=record_thread)\n self.__thread.start()", "response": "Call this when the user clicks the record button."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_buffer_data(self, start: int, count: int) -> typing.Optional[typing.List[typing.List[typing.Dict]]]:\n if hasattr(self.__device, \"get_buffer_data\"):\n buffer_data = self.__device.get_buffer_data(start, count)\n\n enabled_channel_states = list()\n for channel_index in range(self.channel_count):\n channel_state = self.get_channel_state(channel_index)\n if channel_state.enabled:\n enabled_channel_states.append(channel_state)\n\n scan_id = uuid.uuid4()\n\n for data_element_group in buffer_data:\n for channel_index, (data_element, channel_state) in enumerate(zip(data_element_group, enabled_channel_states)):\n channel_name = channel_state.name\n channel_id = channel_state.channel_id\n if self.subscan_enabled:\n channel_id += \"_subscan\"\n properties = data_element[\"properties\"]\n update_autostem_properties(data_element, self.__stem_controller)\n update_calibration_metadata(data_element, None, data_element[\"data\"].shape, scan_id, None, channel_name, channel_id, properties, None, 0)\n data_element[\"properties\"][\"channel_index\"] = channel_index\n data_element[\"properties\"][\"hardware_source_name\"] = self.display_name\n data_element[\"properties\"][\"hardware_source_id\"] = self.hardware_source_id\n\n return buffer_data\n\n return None", "response": "Get the recently acquired buffer data."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_plain_logname(base_name, root_dir, enable_json):\n if enable_json:\n nested_dir = os.path.join(root_dir, 'plain')\n if os.path.exists(root_dir) and not os.path.exists(nested_dir):\n os.mkdir(nested_dir)\n root_dir = nested_dir\n return os.path.join(root_dir, '{}.log'.format(base_name))", "response": "Get the log name for a plain log file."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef setup_logger(\n base_name,\n root_dir=None,\n enable_json=True,\n json_formatter='tipsi_tools.tipsi_logging.JSFormatter',\n loggers={},\n):\n \"\"\"\n json_formatter:\n 'fan.contrib.django.span_formatter.SpanFormatter' - add INSTALLATION_ID, SPAN and etc\n \"\"\"\n if not root_dir:\n root_dir = os.environ.get('LOG_DIR')\n assert root_dir, 'You should pass root_dir parameter or set env LOG_DIR'\n\n JSON_FORMATTER = {\n '()': json_formatter,\n 'env_vars': ['HOST_TYPE', 'TIPSI_CONFIG', 'TIPSI_BRANCH', 'CONTAINER_TYPE'],\n }\n\n default_loggers = {\n '': {'handlers': ['default'], 'level': 'DEBUG', 'propagate': True},\n 'googleapicliet.discovery_cache': {'level': 'ERROR'},\n 'boto3': {'level': 'INFO'},\n 'botocore': {'level': 'INFO'},\n 'kazoo': {'level': 'INFO'},\n 'urllib3': {'level': 'INFO'},\n }\n\n LOGGING = {\n 'version': 1,\n 'disable_existing_loggers': False,\n 'formatters': {\n 'json': JSON_FORMATTER,\n 'standard': {'format': '%(asctime)s [%(levelname)s] %(name)s: %(message)s'},\n },\n 'handlers': {'default': base_handler(get_plain_logname(base_name, root_dir, enable_json))},\n 'loggers': {**default_loggers, **loggers},\n }\n if enable_json:\n LOGGING['handlers']['json'] = base_handler(\n os.path.join(root_dir, '{}.json_log'.format(base_name)), formatter='json'\n )\n LOGGING['loggers']['']['handlers'].append('json')\n logging.config.dictConfig(LOGGING)", "response": "Setup the logger for the given base_name root_dir and loggers."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef run(command):\n '''\n Run command in shell, accepts command construction from list\n Return (return_code, stdout, stderr)\n stdout and stderr - as list of strings\n '''\n if isinstance(command, list):\n command = ' '.join(command)\n out = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n return (out.returncode, _prepare(out.stdout), _prepare(out.stderr))", "response": "Run command in shell accepts command construction from list\n Return return code stdout and stderr - as list of strings"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef succ(cmd, check_stderr=True, stdout=None, stderr=None):\n '''\n Alias to run with check return code and stderr\n '''\n code, out, err = run(cmd)\n\n # Because we're raising error, sometimes we want to process stdout/stderr after catching error\n # so we're copying these outputs if required\n if stdout is not None:\n stdout[:] = out\n if stderr is not None:\n stderr[:] = err\n\n if code != 0:\n for l in out:\n print(l)\n assert code == 0, 'Return: {} {}\\nStderr: {}'.format(code, cmd, err)\n if check_stderr:\n assert err == [], 'Error: {} {}'.format(err, code)\n return code, out, err", "response": "Run a command and return the output and error."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef wait_socket(host, port, timeout=120):\n '''\n Wait for socket opened on remote side. Return False after timeout\n '''\n return wait_result(lambda: check_socket(host, port), True, timeout)", "response": "Wait for socket opened on remote side. Return False after timeout."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef interpolate_sysenv(line, defaults={}):\n '''\n Format line system environment variables + defaults\n '''\n map = ChainMap(os.environ, defaults)\n return line.format(**map)", "response": "Interpolate line system environment variables + defaults"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef source(fname):\n '''\n Acts similar to bash 'source' or '.' commands.\n '''\n rex = re.compile('(?:export |declare -x )?(.*?)=\"(.*?)\"')\n out = call_out('source {} && export'.format(fname))\n out = [x for x in out if 'export' in x or 'declare' in x]\n out = {k: v for k, v in [rex.match(x).groups() for x in out if rex.match(x)]}\n for k, v in out.items():\n os.environ[k] = v", "response": "Acts similar to bash source or. commands.\n Acts similar to bash source or. commands.\n Acts similar to bash source or. commands.\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef cd(dir_name):\n old_path = os.path.abspath('.')\n os.chdir(dir_name)\n try:\n yield\n os.chdir(old_path)\n except Exception:\n os.chdir(old_path)\n raise", "response": "cd - yield the current directory and return back after block ended"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns True if identifier has not been seen before.", "response": "def _is_new(self, identifier):\n \"\"\"Returns True if identifier hasn't been seen before.\"\"\"\n if identifier in self.tracker:\n return False\n else:\n self.tracker.append(identifier)\n self.tracker.pop(0)\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncalls when a GET request is sent to the room", "response": "def on_get(self, request, response, room_alias=None):\n \"\"\"Called when a GET request is sent to /rooms/{room_alias}\"\"\"\n response.body = \"{}\"\n if self.handler(room_alias):\n response.status = falcon.HTTP_200\n self.api.create_room(alias=room_alias)\n else:\n response.status = falcon.HTTP_404"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nrespond to PUT request containing events.", "response": "def on_put(self, request, response, txn_id=None):\n \"\"\"Responds to PUT request containing events.\"\"\"\n response.body = \"{}\"\n\n # Check whether repeat txn_id\n if not self._is_new(txn_id):\n response.status = falcon.HTTP_200\n return\n\n request.context[\"body\"] = request.stream.read()\n try:\n events = json.loads(request.context[\"body\"].decode(\"utf-8\"))[\"events\"]\n except(KeyError, ValueError, UnicodeDecodeError):\n response.status = falcon.HTTP_400\n response.body = \"Malformed request body\"\n return\n\n if self.handler(EventStream(events, self.Api)):\n response.status = falcon.HTTP_200\n else:\n response.status = falcon.HTTP_400"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nresponds to GET request for users.", "response": "def on_get(self, request, response, user_id=None):\n \"\"\"Responds to GET request for users.\"\"\"\n response.body = \"{}\"\n if self.handler(user_id):\n response.status = falcon.HTTP_200\n self.api.register(utils.mxid2localpart(user_id))\n else:\n response.status = falcon.HTTP_404"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting the frame parameters with the settings index and fire the frame parameters changed event.", "response": "def set_frame_parameters(self, profile_index: int, frame_parameters) -> None:\n \"\"\"Set the frame parameters with the settings index and fire the frame parameters changed event.\n\n If the settings index matches the current settings index, call set current frame parameters.\n\n If the settings index matches the record settings index, call set record frame parameters.\n \"\"\"\n self.frame_parameters_changed_event.fire(profile_index, frame_parameters)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef verlet(dfun, xzero, vzero, timerange, timestep):\n '''Verlet method integration. This function wraps the Verlet class.\n\n :param dfun:\n second derivative function of the system.\n The differential system arranged as a series of second-order\n equations: \\ddot{X} = dfun(t, x)\n :param xzero:\n the initial condition of the system\n :param vzero:\n the initial condition of first derivative of the system\n :param timerange:\n the start and end times as (starttime, endtime)\n :param timestep:\n the timestep\n :returns: t, x, v:\n as lists.\n '''\n return zip(*list(Verlet(dfun, xzero, vzero, timerange, timestep)))", "response": "This function is used to wrap the Verlet class in order to generate a list of verlets for a given differential system."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef add_request(self, request):\n\n # Duplicate request name?\n if request.name in self.requests:\n raise ValueError('redefinition of request \"{0}\"'.format(request.name))\n self.requests[request.name] = request\n\n # Add the request URLs\n for method, url in request.urls:\n\n # URL with arguments?\n if RE_URL_ARG.search(url):\n request_regex = '^' + RE_URL_ARG_ESC.sub(r'/(?P<\\1>[^/]+)', re.escape(url)) + '$'\n self.__request_regex.append((method, re.compile(request_regex), request))\n else:\n request_key = (method, url)\n if request_key in self.__request_urls:\n raise ValueError('redefinition of request URL \"{0}\"'.format(url))\n self.__request_urls[request_key] = request", "response": "Add a request object to the internal list of requests and URLs."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef add_header(self, key, value):\n\n assert isinstance(key, str), 'header key must be of type str'\n assert isinstance(value, str), 'header value must be of type str'\n self.headers[key] = value", "response": "Add a response header to the internal dictionary."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef response(self, status, content_type, content, headers=None):\n\n assert not isinstance(content, (str, bytes)), 'response content cannot be of type str or bytes'\n response_headers = [('Content-Type', content_type)]\n if headers:\n response_headers.extend(headers)\n self.start_response(status, response_headers)\n return content", "response": "Send an HTTP response to the application."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nsend a plain - text response.", "response": "def response_text(self, status, text=None, content_type='text/plain', encoding='utf-8', headers=None):\n \"\"\"\n Send a plain-text response\n \"\"\"\n\n if text is None:\n if isinstance(status, str):\n text = status\n else:\n text = status.phrase\n return self.response(status, content_type, [text.encode(encoding)], headers=headers)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsending a JSON response", "response": "def response_json(self, status, response, content_type='application/json', encoding='utf-8', headers=None, jsonp=None):\n \"\"\"\n Send a JSON response\n \"\"\"\n\n encoder = JSONEncoder(\n check_circular=self.app.validate_output,\n allow_nan=False,\n sort_keys=True,\n indent=2 if self.app.pretty_output else None,\n separators=(',', ': ') if self.app.pretty_output else (',', ':')\n )\n content = encoder.encode(response)\n if jsonp:\n content_list = [jsonp.encode(encoding), b'(', content.encode(encoding), b');']\n else:\n content_list = [content.encode(encoding)]\n return self.response(status, content_type, content_list, headers=headers)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreconstruct the URL using the algorithm provided by PEP3333_get_url_from_environ.", "response": "def reconstruct_url(self, path_info=None, query_string=None, relative=False):\n \"\"\"\n Reconstructs the request URL using the algorithm provided by PEP3333\n \"\"\"\n\n environ = self.environ\n if relative:\n url = ''\n else:\n url = environ['wsgi.url_scheme'] + '://'\n\n if environ.get('HTTP_HOST'):\n url += environ['HTTP_HOST']\n else:\n url += environ['SERVER_NAME']\n\n if environ['wsgi.url_scheme'] == 'https':\n if environ['SERVER_PORT'] != '443':\n url += ':' + environ['SERVER_PORT']\n else:\n if environ['SERVER_PORT'] != '80':\n url += ':' + environ['SERVER_PORT']\n\n url += quote(environ.get('SCRIPT_NAME', ''))\n if path_info is None:\n url += quote(environ.get('PATH_INFO', ''))\n else:\n url += path_info\n if query_string is None:\n if environ.get('QUERY_STRING'):\n url += '?' + environ['QUERY_STRING']\n else:\n if query_string:\n if isinstance(query_string, str):\n url += '?' + query_string\n else:\n url += '?' + encode_query_string(query_string)\n\n return url"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nread from files found in XDG - specified system - wide configuration directories.", "response": "def read_xdg_config_dirs(name, extension):\n \"\"\"\n Read from files found in XDG-specified system-wide configuration paths,\n defaulting to ``/etc/xdg``. Depends on ``XDG_CONFIG_DIRS`` environment\n variable.\n\n :param name: application or configuration set name\n :param extension: file extension to look for\n :return: a `.Configuration` instance with values read from XDG-specified\n directories\n \"\"\"\n # find optional value of ${XDG_CONFIG_DIRS}\n config_dirs = environ.get('XDG_CONFIG_DIRS')\n if config_dirs:\n # PATH-like env vars operate in decreasing precedence, reverse this path set to mimic the end result\n config_dirs = reversed(config_dirs.split(path.pathsep))\n else:\n # XDG spec: \"If $XDG_CONFIG_DIRS is either not set or empty, a value equal to /etc/xdg should be used.\"\n config_dirs = ['/etc/xdg']\n\n # load a file from all config dirs, default to NotConfigured\n fname = '{name}.{extension}'.format(name=name, extension=extension)\n return loadf(*(path.join(config_dir, fname) for config_dir in config_dirs),\n default=NotConfigured)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nread from file found in XDG - specified configuration home directory.", "response": "def read_xdg_config_home(name, extension):\n \"\"\"\n Read from file found in XDG-specified configuration home directory,\n expanding to ``${HOME}/.config/name.extension`` by default. Depends on\n ``XDG_CONFIG_HOME`` or ``HOME`` environment variables.\n\n :param name: application or configuration set name\n :param extension: file extension to look for\n :return: a `.Configuration` instance, possibly `.NotConfigured`\n \"\"\"\n # find optional value of ${XDG_CONFIG_HOME}\n config_home = environ.get('XDG_CONFIG_HOME')\n if not config_home:\n # XDG spec: \"If $XDG_CONFIG_HOME is either not set or empty, a default equal to $HOME/.config should be used.\"\n # see https://specifications.freedesktop.org/basedir-spec/latest/ar01s03.html\n config_home = path.expanduser('~/.config')\n\n # expand to full path to configuration file in XDG config path\n return loadf(path.join(config_home, '{name}.{extension}'.format(name=name, extension=extension)),\n default=NotConfigured)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nread environment variables starting with NAME_ and returns a Configuration instance.", "response": "def read_envvars(name, extension):\n \"\"\"\n Read environment variables starting with ``NAME_``, where subsequent\n underscores are interpreted as namespaces. Underscores can be retained as\n namespaces by doubling them up, e.g. ``NAME_SPA__CE_KEY`` would be\n accessible in the resulting `.Configuration` as\n ``c.spa_ce.key``, where ``c`` is the `.Configuration` instance.\n\n .. note::\n\n Environment variables are always `str`s, this function makes no effort\n to changes this. All values read from command line variables will be\n `str` instances.\n\n :param name: environment variable prefix to look for (without the ``_``)\n :param extension: *(unused)*\n :return: a `.Configuration` instance, possibly `.NotConfigured`\n \"\"\"\n prefix = '{}_'.format(name)\n prefix_len = len(prefix)\n envvar_file = '{}_config_file'.format(name)\n # create a new mapping from environment values starting with the prefix (but stripped of that prefix)\n values = {var.lower()[prefix_len:]: value\n for var, value in environ.items()\n # TODO: document ignoring envvar_file\n if var.lower().startswith(prefix) and var.lower() != envvar_file}\n # TODO: envvar values can only be str, how do we configure non-str values?\n if not values:\n return NotConfigured\n\n def dotted(name):\n # replace 'regular' underscores (those between alphanumeric characters) with dots first\n name = re.sub(r'([0-9A-Za-z])_([0-9A-Za-z])', r'\\1.\\2', name)\n # unescape double underscores back to a single one\n return re.sub(r'__', '_', name)\n\n return Configuration({dotted(name): value for name, value in values.items()})"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef read_envvar_file(name, extension):\n envvar_file = environ.get('{}_config_file'.format(name).upper())\n if envvar_file:\n # envvar set, load value as file\n return loadf(envvar_file)\n else:\n # envvar not set, return an empty source\n return NotConfigured", "response": "Read values from a file provided as a environment variable\n ."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef read_envvar_dir(envvar, name, extension):\n config_dir = environ.get(envvar)\n if not config_dir:\n return NotConfigured\n\n # envvar is set, construct full file path, expanding user to allow the envvar containing a value like ~/config\n config_path = path.join(path.expanduser(config_dir), '{name}.{extension}'.format(name=name, extension=extension))\n return loadf(config_path, default=NotConfigured)", "response": "Read values from a file located in a directory specified by an environment variable."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngenerating a generator that yields the available locality loaders for the specified locality.", "response": "def loaders(*specifiers):\n \"\"\"\n Generates loaders in the specified order.\n\n Arguments can be `.Locality` instances, producing the loader(s) available\n for that locality, `str` instances (used as file path templates) or\n `callable`s. These can be mixed:\n\n .. code-block:: python\n\n # define a load order using predefined user-local locations,\n # an explicit path, a template and a user-defined function\n load_order = loaders(Locality.user,\n '/etc/defaults/hard-coded.yaml',\n '/path/to/{name}.{extension}',\n my_loader)\n\n # load configuration for name 'my-application' using the load order\n # defined above\n config = load_name('my-application', load_order=load_order)\n\n :param specifiers:\n :return: a `generator` of configuration loaders in the specified order\n \"\"\"\n for specifier in specifiers:\n if isinstance(specifier, Locality):\n # localities can carry multiple loaders, flatten this\n yield from _LOADERS[specifier]\n else:\n # something not a locality, pass along verbatim\n yield specifier"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreads a. Configuration instance from file - like objects.", "response": "def load(*fps, missing=Missing.silent):\n \"\"\"\n Read a `.Configuration` instance from file-like objects.\n\n :param fps: file-like objects (supporting ``.read()``)\n :param missing: policy to be used when a configured key is missing, either\n as a `.Missing` instance or a default value\n :return: a `.Configuration` instance providing values from *fps*\n :rtype: `.Configuration`\n \"\"\"\n return Configuration(*(yaml.safe_load(fp.read()) for fp in fps), missing=missing)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreading a. Configuration instance from named files.", "response": "def loadf(*fnames, default=_NoDefault, missing=Missing.silent):\n \"\"\"\n Read a `.Configuration` instance from named files.\n\n :param fnames: name of the files to ``open()``\n :param default: `dict` or `.Configuration` to use when a file does not\n exist (default is to raise a `FileNotFoundError`)\n :param missing: policy to be used when a configured key is missing, either\n as a `.Missing` instance or a default value\n :return: a `.Configuration` instance providing values from *fnames*\n :rtype: `.Configuration`\n \"\"\"\n def readf(fname):\n if default is _NoDefault or path.exists(fname):\n # (attempt to) open fname if it exists OR if we're expected to raise an error on a missing file\n with open(fname, 'r') as fp:\n # default to empty dict, yaml.safe_load will return None for an empty document\n return yaml.safe_load(fp.read()) or {}\n else:\n return default\n\n return Configuration(*(readf(path.expanduser(fname)) for fname in fnames), missing=missing)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreads a. Configuration instance from a list of strings.", "response": "def loads(*strings, missing=Missing.silent):\n \"\"\"\n Read a `.Configuration` instance from strings.\n\n :param strings: configuration contents\n :param missing: policy to be used when a configured key is missing, either\n as a `.Missing` instance or a default value\n :return: a `.Configuration` instance providing values from *strings*\n :rtype: `.Configuration`\n \"\"\"\n return Configuration(*(yaml.safe_load(string) for string in strings), missing=missing)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load_name(*names, load_order=DEFAULT_LOAD_ORDER, extension='yaml', missing=Missing.silent):\n def generate_sources():\n # argument order for product matters, for names \"foo\" and \"bar\":\n # /etc/foo.yaml before /etc/bar.yaml, but both of them before ~/.foo.yaml and ~/.bar.yaml\n for source, name in product(load_order, names):\n if callable(source):\n yield source(name, extension)\n else:\n # expand user to turn ~/.name.yaml into /home/user/.name.yaml\n candidate = path.expanduser(source.format(name=name, extension=extension))\n yield loadf(candidate, default=NotConfigured)\n\n return Configuration(*generate_sources(), missing=missing)", "response": "Read a. Configuration instance by name."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef acquire(self, block=True):\n while True:\n # Try to set the lock\n if self.redis.set(self.name, self.value, px=self.timeout, nx=True):\n # It's ours until the timeout now\n return True\n # Lock is taken\n if not block:\n return False\n # If blocking, try again in a bit\n time.sleep(self.sleep)", "response": "Acquire lock. Blocks until acquired if block is True otherwise returns False."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nrefreshes an existing lock.", "response": "def refresh(self):\n \"\"\"Refresh an existing lock to prevent it from expiring.\n Uses a LUA (EVAL) script to ensure only a lock which we own is being overwritten.\n Returns True if refresh succeeded, False if not.\"\"\"\n keys = [self.name]\n args = [self.value, self.timeout]\n # Redis docs claim EVALs are atomic, and I'm inclined to believe it.\n if hasattr(self, '_refresh_script'):\n return self._refresh_script(keys=keys, args=args) == 1\n else:\n keys_and_args = keys + args\n return self.redis.eval(self.lua_refresh, len(keys), *keys_and_args)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef run(self):\n\n log.info(\"Waiting for lock, currently held by %s\", self.lock.who())\n if self.lock.acquire():\n log.info(\"Lock '%s' acquired\", self.lockname)\n # We got the lock, so we make sure the process is running and keep refreshing the lock - if we ever stop for any reason, for example because our host died, the lock will soon expire.\n while True:\n if self.process is None: # Process not spawned yet\n self.process = self.spawn(self.command)\n log.info(\"Spawned PID %d\", self.process.pid)\n child_status = self.process.poll()\n if child_status is not None:\n # Oops, process died on us.\n log.error(\"Child died with exit code %d\", child_status)\n sys.exit(1)\n # Refresh lock and sleep\n if not self.lock.refresh():\n who = self.lock.who()\n if who is None:\n if self.lock.acquire(block=False):\n log.warning(\"Lock refresh failed, but successfully re-acquired unclaimed lock\")\n else:\n log.error(\"Lock refresh and subsequent re-acquire failed, giving up (Lock now held by %s)\", self.lock.who())\n self.cleanup()\n sys.exit(os.EX_UNAVAILABLE)\n else:\n log.error(\"Lock refresh failed, %s stole it - bailing out\", self.lock.who())\n self.cleanup()\n sys.exit(os.EX_UNAVAILABLE)\n time.sleep(self.sleep)", "response": "Run the process if nobody else is."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef cleanup(self):\n if self.process is None: # Process wasn't running yet, so nothing to worry about\n return\n if self.process.poll() is None:\n log.info(\"Sending TERM to %d\", self.process.pid)\n self.process.terminate()\n # Give process a second to terminate, if it didn't, kill it.\n start = time.clock()\n while time.clock() - start < 1.0:\n time.sleep(0.05)\n if self.process.poll() is not None:\n break\n else:\n log.info(\"Sending KILL to %d\", self.process.pid)\n self.process.kill()\n assert self.process.poll() is not None", "response": "Clean up making sure the process is stopped before we pack up and go home."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nvalidate the incoming data against the current node s node s node s schema.", "response": "def validate(data, schema, defined_keys=False):\n \"\"\"\n Main entry point for the validation engine.\n\n :param data: The incoming data, as a dictionary object.\n :param schema: The schema from which data will be validated against\n \"\"\"\n if isinstance(data, dict):\n validator = Validator(data, schema, defined_keys=defined_keys)\n validator.validate()\n else:\n raise TypeError('expected data to be of type dict, but got: %s' % type(data))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntraverses the dictionary and checks if the data is valid for a particular key value pair.", "response": "def traverser(self, data, schema, tree):\n \"\"\"\n Traverses the dictionary, recursing onto itself if\n it sees appropriate key/value pairs that indicate that\n there is a need for more validation in a branch below us.\n \"\"\"\n if hasattr(schema, '__validator_leaf__'):\n return schema(data, tree)\n\n if hasattr(schema, 'must_validate'): # cherry picking?\n if not len(schema.must_validate):\n reason = \"must_validate attribute must not be empty\"\n raise SchemaError(data, tree, reason=reason)\n data = sift(data, schema.must_validate)\n\n schema = self.sanitize_optionals(data, schema, tree)\n self.is_alpha_ordered(data, schema, tree)\n\n validated_indexes = []\n skip_missing_indexes = getattr(schema, 'must_validate', False)\n\n if len(data) < len(schema):\n # we have missing required items in data, but we don't know\n # which ones so find what may fail:\n data_keys = [v[0] for v in data.values()]\n schema_keys = [v[0] for v in schema.values()]\n\n def enforce_once(data_keys, schema_key):\n # XXX Go through all the data keys and try and see if they pass\n # validation against the schema. At this point it is impossible\n # to know which data key corresponds to what schema key\n # (because schema keys can be a function/callable) so it is\n # a *very* naive way to try and detect which one might be\n # missing\n for data_key in data_keys:\n failed = None\n try:\n enforce(data_key, schema_key, tree, pair='key')\n return\n except Invalid:\n failed = data_key, schema_key\n\n if failed:\n return failed\n\n # if there are no callables in the schema keys, just\n # find the missing data key directly\n if all([not is_callable(s) for s in schema_keys]):\n for schema_key in schema_keys:\n if schema_key not in data_keys:\n msg = \"required key in data is missing: %s\" % str(schema_key)\n raise Invalid(None, tree, reason=msg, pair='key')\n\n for schema_key in schema_keys:\n failure = enforce_once(data_keys, schema_key)\n if failure:\n _, failed_schema_key = failure\n msg = \"required key in data is missing: %s\" % str(failed_schema_key)\n raise Invalid(None, tree, reason=msg, pair='key')\n\n for index in range(len(data)):\n self.length_equality(data, schema, index, tree)\n key, value = data[index]\n skey, svalue = schema[index]\n tree.append(key)\n\n # Validate the key before anything, to prevent recursing\n self.key_leaf(data[index], schema[index], tree)\n\n # If a dict is a value we need to recurse.\n # XXX Should we check isinstance(value, ndict) ?\n if isinstance(value, dict) and len(value):\n self.traverser(value, svalue, tree)\n else:\n self.value_leaf(data[index], schema[index], tree)\n if tree:\n tree.pop()\n\n validated_indexes.append(index)\n\n # XXX There is a chance we might have missing items from\n # the incoming data that are labeled as required from the schema\n # we should make sure *here* that we account for that and raise\n # the appropriate exception. Since the loop finished and everything\n # seems to have passed, this lack of check will give false positives.\n missing_indexes = set(schema.keys()).difference(validated_indexes)\n if missing_indexes:\n if skip_missing_indexes:\n return\n for i in missing_indexes:\n if not hasattr(schema[i], 'is_optional'):\n required_key = schema[i][0]\n tree.append('item[%s]' % i)\n msg = \"required item in schema is missing: %s\" % str(required_key)\n raise Invalid(required_key, tree, reason=msg, pair='key')"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nvalidating that a key is a leaf of a tree.", "response": "def key_leaf(self, data, schema, tree):\n \"\"\"\n The deepest validation we can make in any given circumstance for a key.\n Does not recurse, it will just receive both values and the tree,\n passing them on to the :fun:`enforce` function.\n \"\"\"\n key, value = data\n schema_key, schema_value = schema\n enforce(key, schema_key, tree, 'key')"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef value_leaf(self, data, schema, tree):\n key, value = data\n schema_key, schema_value = schema\n\n if hasattr(schema_value, '__validator_leaf__'):\n return schema_value(value, tree)\n enforce(value, schema_value, tree, 'value')", "response": "Validate that a value is a leaf of a tree."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a string representation of a datetime. timedelta object. Works just like strftime", "response": "def strfdelta(tdelta, fmt):\n \"\"\"\n Used to format `datetime.timedelta` objects. Works just like `strftime`\n\n >>> strfdelta(duration, '%H:%M:%S')\n\n param tdelta: Time duration which is an instance of datetime.timedelta\n param fmt: The pattern to format the timedelta with\n rtype: str\n \"\"\"\n substitutes = dict()\n hours, rem = divmod(tdelta.total_seconds(), 3600)\n minutes, seconds = divmod(rem, 60)\n\n substitutes[\"H\"] = '{:02d}'.format(int(hours))\n substitutes[\"M\"] = '{:02d}'.format(int(minutes))\n substitutes[\"S\"] = '{:02d}'.format(int(seconds))\n return DeltaTemplate(fmt).substitute(**substitutes)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the user s information", "response": "def get_user_information():\n \"\"\"\n Returns the user's information\n\n :rtype: (str, int, str)\n \"\"\"\n try:\n import pwd\n _username = pwd.getpwuid(os.getuid())[0]\n _userid = os.getuid()\n _uname = os.uname()[1]\n except ImportError:\n import getpass\n _username = getpass.getuser()\n _userid = 0\n import platform\n _uname = platform.node()\n\n return _username, _userid, _uname"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nauthenticate the App using the token credentials.", "response": "def _login_azure_app_token(client_id=None, client_secret=None, tenant_id=None):\n \"\"\"\n Authenticate APP using token credentials:\n https://docs.microsoft.com/en-us/python/azure/python-sdk-azure-authenticate?view=azure-python\n :return: ~ServicePrincipalCredentials credentials\n \"\"\"\n client_id = os.getenv('AZURE_CLIENT_ID') if not client_id else client_id\n client_secret = os.getenv('AZURE_CLIENT_SECRET') if not client_secret else client_secret\n tenant_id = os.getenv('AZURE_TENANT_ID') if not tenant_id else tenant_id\n credentials = ServicePrincipalCredentials(\n client_id=client_id,\n secret=client_secret,\n tenant=tenant_id,\n )\n return credentials"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef exec_container(self,\n asset_url,\n algorithm_url,\n resource_group_name,\n account_name,\n account_key,\n location,\n share_name_input='compute',\n share_name_output='output',\n docker_image='python:3.6-alpine',\n memory=1.5,\n cpu=1):\n \"\"\"Prepare a docker image that will run in the cloud, mounting the asset and executing the algorithm.\n :param asset_url\n :param algorithm_url\n :param resource_group_name:\n :param account_name:\n :param account_key:\n :param share_name_input:\n :param share_name_output:\n :param location:\n \"\"\"\n try:\n container_group_name = 'compute' + str(int(time.time()))\n result_file = self._create_container_group(resource_group_name=resource_group_name,\n name=container_group_name,\n image=docker_image,\n location=location,\n memory=memory,\n cpu=cpu,\n algorithm=algorithm_url,\n asset=asset_url,\n input_mount_point='/input',\n output_moint_point='/output',\n account_name=account_name,\n account_key=account_key,\n share_name_input=share_name_input,\n share_name_output=share_name_output\n )\n while self.client.container_groups.get(resource_group_name,\n container_group_name).provisioning_state != 'Succeeded':\n logging.info(\"Waiting to resources \")\n while self.client.container_groups.get(resource_group_name, container_group_name). \\\n containers[0].instance_view.current_state.state != 'Terminated':\n logging.info(\"Waiting to terminate\")\n self.delete_vm(container_group_name, resource_group_name)\n return result_file\n except Exception:\n logging.error(\"There was a problem executing your container\")\n raise Exception", "response": "Prepare a docker image that will run in the cloud, mounting the asset and executing the algorithm.\n :param asset_url\n :param algorithm_url\n :param resource_group_name:\n :param account_name:\n :param account_key:\n :param share_name_input:\n :param share_name_output:\n :param location:"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef list_container_groups(self, resource_group_name):\n print(\"Listing container groups in resource group '{0}'...\".format(resource_group_name))\n\n container_groups = self.client.container_groups.list_by_resource_group(resource_group_name)\n\n for container_group in container_groups:\n print(\" {0}\".format(container_group.name))", "response": "Lists the container groups in the specified resource group."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a sequence of namedtuple objects from a fasta file with attributes ( id description seq", "response": "def fastalite(handle):\n \"\"\"Return a sequence of namedtuple objects from a fasta file with\n attributes (id, description, seq) given open file-like object\n ``handle``\n\n \"\"\"\n\n Seq = namedtuple('Seq', ['id', 'description', 'seq'])\n\n header, seq = '', []\n for line in handle:\n if line.startswith('>'):\n if header:\n yield Seq(header.split()[0], header, ''.join(seq))\n header, seq = line[1:].strip(), []\n else:\n seq.append(line.strip())\n\n if header and seq:\n yield Seq(header.split()[0], header, ''.join(seq))"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a sequence of namedtuple objects from a fastq file with attributes ( id description seq qual", "response": "def fastqlite(handle):\n \"\"\"Return a sequence of namedtuple objects from a fastq file with\n attributes (id, description, seq, qual) given open file-like\n object ``handle``. This parser assumes that lines corresponding to\n sequences and quality scores are not wrapped. Raises\n ``ValueError`` for malformed records.\n\n See https://doi.org/10.1093/nar/gkp1137 for a discussion of the\n fastq format.\n\n \"\"\"\n\n Seq = namedtuple('Seq', ['id', 'description', 'seq', 'qual'])\n for i, chunk in enumerate(grouper(handle, 4, '')):\n description, seq, plus, qual = chunk\n seq, qual = seq.strip(), qual.strip()\n\n checks = [description.startswith('@'), seq,\n plus.startswith('+'), qual, len(seq) == len(qual)]\n\n if not all(checks):\n raise ValueError('Malformed record around line {}'.format(i * 4))\n\n description = description[1:].strip()\n yield Seq(description.split()[0], description, seq, qual)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nyield extra tuples containing a field name and a callable that takes a row", "response": "def cauldron_extras(self):\n \"\"\" Yield extra tuples containing a field name and a callable that takes\n a row\n \"\"\"\n for extra in super(Dimension, self).cauldron_extras:\n yield extra\n\n if self.formatters:\n prop = self.id + '_raw'\n else:\n prop = self.id_prop\n\n yield self.id + '_id', lambda row: getattr(row, prop)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef make_column_suffixes(self):\n if self.column_suffixes:\n return self.column_suffixes\n\n if len(self.columns) == 0:\n return ()\n\n elif len(self.columns) == 1:\n if self.formatters:\n return '_raw',\n else:\n return '',\n\n elif len(self.columns) == 2:\n if self.formatters:\n return '_id', '_raw',\n else:\n return '_id', '',\n else:\n raise BadIngredient(\n 'column_suffixes must be supplied if there is '\n 'more than one column'\n )", "response": "Make sure we have the right column suffixes."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a SQLAlchemy clause from a condition.", "response": "def parse_condition(\n cond, selectable, aggregated=False, default_aggregation='sum'\n):\n \"\"\"Create a SQLAlchemy clause from a condition.\"\"\"\n if cond is None:\n return None\n\n else:\n if 'and' in cond:\n conditions = [\n parse_condition(\n c, selectable, aggregated, default_aggregation\n ) for c in cond['and']\n ]\n return and_(*conditions)\n elif 'or' in cond:\n conditions = [\n parse_condition(\n c, selectable, aggregated, default_aggregation\n ) for c in cond['or']\n ]\n return or_(*conditions)\n elif 'field' not in cond:\n raise BadIngredient('field must be defined in condition')\n field = parse_field(\n cond['field'],\n selectable,\n aggregated=aggregated,\n default_aggregation=default_aggregation\n )\n if 'in' in cond:\n value = cond['in']\n if isinstance(value, dict):\n raise BadIngredient('value for in must be a list')\n condition_expression = getattr(field, 'in_')(tuple(value))\n elif 'gt' in cond:\n value = cond['gt']\n if isinstance(value, (list, dict)):\n raise BadIngredient('conditional value must be a scalar')\n condition_expression = getattr(field, '__gt__')(value)\n elif 'gte' in cond:\n value = cond['gte']\n if isinstance(value, (list, dict)):\n raise BadIngredient('conditional value must be a scalar')\n condition_expression = getattr(field, '__ge__')(value)\n elif 'lt' in cond:\n value = cond['lt']\n if isinstance(value, (list, dict)):\n raise BadIngredient('conditional value must be a scalar')\n condition_expression = getattr(field, '__lt__')(value)\n elif 'lte' in cond:\n value = cond['lte']\n if isinstance(value, (list, dict)):\n raise BadIngredient('conditional value must be a scalar')\n condition_expression = getattr(field, '__le__')(value)\n elif 'eq' in cond:\n value = cond['eq']\n if isinstance(value, (list, dict)):\n raise BadIngredient('conditional value must be a scalar')\n condition_expression = getattr(field, '__eq__')(value)\n elif 'ne' in cond:\n value = cond['ne']\n if isinstance(value, (list, dict)):\n raise BadIngredient('conditional value must be a scalar')\n condition_expression = getattr(field, '__ne__')(value)\n else:\n raise BadIngredient('Bad condition')\n\n return condition_expression"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef tokenize(s):\n\n # Crude tokenization\n s = s.replace('+', ' PLUS ').replace('-', ' MINUS ') \\\n .replace('/', ' DIVIDE ').replace('*', ' MULTIPLY ')\n words = [w for w in s.split(' ') if w]\n return words", "response": "Tokenize a string by splitting it by + and - and return a list of words."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfind a column in a column collection by name or _label", "response": "def _find_in_columncollection(columns, name):\n \"\"\" Find a column in a column collection by name or _label\"\"\"\n for col in columns:\n if col.name == name or getattr(col, '_label', None) == name:\n return col\n return None"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfind a column named name in selectable", "response": "def find_column(selectable, name):\n \"\"\"\n Find a column named `name` in selectable\n\n :param selectable:\n :param name:\n :return: A column object\n \"\"\"\n from recipe import Recipe\n\n if isinstance(selectable, Recipe):\n selectable = selectable.subquery()\n\n # Selectable is a table\n if isinstance(selectable, DeclarativeMeta):\n col = getattr(selectable, name, None)\n if col is not None:\n return col\n\n col = _find_in_columncollection(selectable.__table__.columns, name)\n if col is not None:\n return col\n\n # Selectable is a sqlalchemy subquery\n elif hasattr(selectable, 'c'\n ) and isinstance(selectable.c, ImmutableColumnCollection):\n col = getattr(selectable.c, name, None)\n if col is not None:\n return col\n\n col = _find_in_columncollection(selectable.c, name)\n if col is not None:\n return col\n\n raise BadIngredient('Can not find {} in {}'.format(name, selectable))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nparsing a field object from yaml into a sqlalchemy expression", "response": "def parse_field(fld, selectable, aggregated=True, default_aggregation='sum'):\n \"\"\" Parse a field object from yaml into a sqlalchemy expression \"\"\"\n # An aggregation is a callable that takes a single field expression\n # None will perform no aggregation\n aggregation_lookup = {\n 'sum': func.sum,\n 'min': func.min,\n 'max': func.max,\n 'avg': func.avg,\n 'count': func.count,\n 'count_distinct': lambda fld: func.count(distinct(fld)),\n 'month': lambda fld: func.date_trunc('month', fld),\n 'week': lambda fld: func.date_trunc('week', fld),\n 'year': lambda fld: func.date_trunc('year', fld),\n 'quarter': lambda fld: func.date_trunc('quarter', fld),\n 'age': lambda fld: func.date_part('year', func.age(fld)),\n None: lambda fld: fld,\n }\n\n # Ensure that the dictionary contains:\n # {\n # 'value': str,\n # 'aggregation': str|None,\n # 'condition': dict|None\n # }\n if isinstance(fld, basestring):\n fld = {\n 'value': fld,\n }\n if not isinstance(fld, dict):\n raise BadIngredient('fields must be a string or a dict')\n if 'value' not in fld:\n raise BadIngredient('fields must contain a value')\n if not isinstance(fld['value'], basestring):\n raise BadIngredient('field value must be a string')\n\n # Ensure a condition\n if 'condition' in fld:\n if not isinstance(fld['condition'], dict) and \\\n not fld['condition'] is None:\n raise BadIngredient('condition must be null or an object')\n else:\n fld['condition'] = None\n\n # Ensure an aggregation\n initial_aggregation = default_aggregation if aggregated else None\n if 'aggregation' in fld:\n if not isinstance(fld['aggregation'], basestring) and \\\n not fld['aggregation'] is None:\n raise BadIngredient('aggregation must be null or an string')\n if fld['aggregation'] is None:\n fld['aggregation'] = initial_aggregation\n else:\n fld['aggregation'] = initial_aggregation\n\n value = fld.get('value', None)\n if value is None:\n raise BadIngredient('field value is not defined')\n\n field_parts = []\n for word in tokenize(value):\n if word in ('MINUS', 'PLUS', 'DIVIDE', 'MULTIPLY'):\n field_parts.append(word)\n else:\n field_parts.append(find_column(selectable, word))\n\n if len(field_parts) is None:\n raise BadIngredient('field is not defined.')\n # Fields should have an odd number of parts\n if len(field_parts) % 2 != 1:\n raise BadIngredient('field does not have the right number of parts')\n\n field = field_parts[0]\n if len(field_parts) > 1:\n # if we need to add and subtract from the field\n # join the field parts into pairs, for instance if field parts is\n # [MyTable.first, 'MINUS', MyTable.second, 'PLUS', MyTable.third]\n # we will get two pairs here\n # [('MINUS', MyTable.second), ('PLUS', MyTable.third)]\n for operator, other_field in zip(field_parts[1::2], field_parts[2::2]):\n if operator == 'PLUS':\n field = field.__add__(other_field)\n elif operator == 'MINUS':\n field = field.__sub__(other_field)\n elif operator == 'DIVIDE':\n field = field.__div__(other_field)\n elif operator == 'MULTIPLY':\n field = field.__mul__(other_field)\n else:\n raise BadIngredient('Unknown operator {}'.format(operator))\n\n # Handle the aggregator\n aggr = fld.get('aggregation', 'sum')\n if aggr is not None:\n aggr = aggr.strip()\n\n if aggr not in aggregation_lookup:\n raise BadIngredient('unknown aggregation {}'.format(aggr))\n\n aggregator = aggregation_lookup[aggr]\n\n condition = parse_condition(\n fld.get('condition', None),\n selectable,\n aggregated=False,\n default_aggregation=default_aggregation\n )\n\n if condition is not None:\n field = case([(condition, field)])\n\n return aggregator(field)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate an ingredient from a dictionary.", "response": "def ingredient_from_dict(ingr_dict, selectable):\n \"\"\"Create an ingredient from an dictionary.\n\n This object will be deserialized from yaml \"\"\"\n\n # TODO: This is deprecated in favor of\n # ingredient_from_validated_dict\n\n # Describe the required params for each kind of ingredient\n # The key is the parameter name, the value is one of\n # field: A parse_field with aggregation=False\n # aggregated_field: A parse_field with aggregation=True\n # condition: A parse_condition\n\n params_lookup = {\n 'Dimension': {\n 'field': 'field'\n },\n 'LookupDimension': {\n 'field': 'field'\n },\n 'IdValueDimension':\n OrderedDict(id_field='field', field='field'),\n 'Metric': {\n 'field': 'aggregated_field'\n },\n 'DivideMetric':\n OrderedDict(\n numerator_field='aggregated_field',\n denominator_field='aggregated_field'\n ),\n 'WtdAvgMetric':\n OrderedDict(field='field', weight='field')\n }\n\n format_lookup = {\n 'comma': ',.0f',\n 'dollar': '$,.0f',\n 'percent': '.0%',\n 'comma1': ',.1f',\n 'dollar1': '$,.1f',\n 'percent1': '.1%',\n 'comma2': ',.2f',\n 'dollar2': '$,.2f',\n 'percent2': '.2%',\n }\n\n kind = ingr_dict.pop('kind', 'Metric')\n IngredientClass = ingredient_class_for_name(kind)\n\n if IngredientClass is None:\n raise BadIngredient('Unknown ingredient kind')\n\n params = params_lookup.get(kind, {'field': 'field'})\n\n args = []\n for k, v in iteritems(params):\n # All the params must be in the dict\n if k not in ingr_dict:\n raise BadIngredient(\n '{} must be defined to make a {}'.format(k, kind)\n )\n if v == 'field':\n statement = parse_field(\n ingr_dict.pop(k, None), selectable, aggregated=False\n )\n elif v == 'aggregated_field':\n statement = parse_field(\n ingr_dict.pop(k, None), selectable, aggregated=True\n )\n elif v == 'condition':\n statement = parse_condition(\n ingr_dict.pop(k, None), selectable, aggregated=True\n )\n else:\n raise BadIngredient('Do not know what this is')\n\n args.append(statement)\n # Remaining properties in ingr_dict are treated as keyword args\n\n # If the format string exists in format_lookup, use the value otherwise\n # use the original format\n if 'format' in ingr_dict:\n ingr_dict['format'] = format_lookup.get(\n ingr_dict['format'], ingr_dict['format']\n )\n return IngredientClass(*args, **ingr_dict)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef parse_validated_field(fld, selectable):\n aggr_fn = IngredientValidator.aggregation_lookup[fld['aggregation']]\n\n field = find_column(selectable, fld['value'])\n\n for operator in fld.get('operators', []):\n op = operator['operator']\n other_field = parse_validated_field(operator['field'], selectable)\n field = IngredientValidator.operator_lookup[op](field)(other_field)\n\n condition = fld.get('condition', None)\n if condition:\n condition = parse_condition(condition, selectable)\n field = case([(condition, field)])\n\n field = aggr_fn(field)\n return field", "response": "Converts a validated field to sqlalchemy. Field references are kept in selectable"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate an ingredient from a dictionary.", "response": "def ingredient_from_validated_dict(ingr_dict, selectable):\n \"\"\" Create an ingredient from an dictionary.\n\n This object will be deserialized from yaml \"\"\"\n\n validator = IngredientValidator(schema=ingr_dict['kind'])\n if not validator.validate(ingr_dict):\n raise Exception(validator.errors)\n ingr_dict = validator.document\n\n kind = ingr_dict.pop('kind', 'Metric')\n IngredientClass = ingredient_class_for_name(kind)\n\n if IngredientClass is None:\n raise BadIngredient('Unknown ingredient kind')\n\n args = []\n for fld in ingr_dict.pop('_fields', []):\n args.append(parse_validated_field(ingr_dict.pop(fld), selectable))\n\n return IngredientClass(*args, **ingr_dict)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef AutomaticShelf(table):\n if hasattr(table, '__table__'):\n table = table.__table__\n config = introspect_table(table)\n return Shelf.from_config(config, table)", "response": "Auto generate a Shelf from a SQLAlchemy Table."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef introspect_table(table):\n d = {}\n for c in table.columns:\n if isinstance(c.type, String):\n d[c.name] = {'kind': 'Dimension', 'field': c.name}\n if isinstance(c.type, (Integer, Float)):\n d[c.name] = {'kind': 'Metric', 'field': c.name}\n return d", "response": "Given a SQLAlchemy Table object return a Shelf description suitable\n for passing to Shelf. from_config.\n "} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\npop an ingredient off of this shelf.", "response": "def pop(self, k, d=_POP_DEFAULT):\n \"\"\"Pop an ingredient off of this shelf.\"\"\"\n if d is _POP_DEFAULT:\n return self._ingredients.pop(k)\n else:\n return self._ingredients.pop(k, d)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the list of Dimension objects on this shelf in the order in which they were used.", "response": "def dimension_ids(self):\n \"\"\" Return the Dimensions on this shelf in the order in which\n they were used.\"\"\"\n return self._sorted_ingredients([\n d.id for d in self.values() if isinstance(d, Dimension)\n ])"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the list of Metrics on this shelf in the order in which they were used.", "response": "def metric_ids(self):\n \"\"\" Return the Metrics on this shelf in the order in which\n they were used. \"\"\"\n return self._sorted_ingredients([\n d.id for d in self.values() if isinstance(d, Metric)\n ])"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef filter_ids(self):\n return self._sorted_ingredients([\n d.id for d in self.values() if isinstance(d, Filter)\n ])", "response": "Return the IDs of all the Filtered Metrics on this shelf."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate a new Shelf from a dictionary containing a dictionary containing ingredients defined in obj.", "response": "def from_config(\n cls,\n obj,\n selectable,\n ingredient_constructor=ingredient_from_validated_dict,\n metadata=None\n ):\n \"\"\"Create a shelf using a dict shelf definition.\n\n :param obj: A Python dictionary describing a Shelf.\n :param selectable: A SQLAlchemy Table, a Recipe, a table name, or a\n SQLAlchemy join to select from.\n :param metadata: If `selectable` is passed as a table name, then in\n order to introspect its schema, we must have the SQLAlchemy\n MetaData object to associate it with.\n :return: A shelf that contains the ingredients defined in obj.\n \"\"\"\n from recipe import Recipe\n if isinstance(selectable, Recipe):\n selectable = selectable.subquery()\n elif isinstance(selectable, basestring):\n if '.' in selectable:\n schema, tablename = selectable.split('.')\n else:\n schema, tablename = None, selectable\n\n selectable = Table(\n tablename,\n metadata,\n schema=schema,\n extend_existing=True,\n autoload=True\n )\n\n d = {}\n for k, v in iteritems(obj):\n d[k] = ingredient_constructor(v, selectable)\n shelf = cls(d, select_from=selectable)\n return shelf"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a shelf from a yaml string containing the ingredients defined in the yaml file.", "response": "def from_yaml(cls, yaml_str, selectable, **kwargs):\n \"\"\"Create a shelf using a yaml shelf definition.\n\n :param yaml_str: A string containing yaml ingredient definitions.\n :param selectable: A SQLAlchemy Table, a Recipe, or a SQLAlchemy\n join to select from.\n :return: A shelf that contains the ingredients defined in yaml_str.\n \"\"\"\n obj = safe_load(yaml_str)\n return cls.from_config(\n obj,\n selectable,\n ingredient_constructor=ingredient_from_dict,\n **kwargs\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates a shelf from a validated yaml string.", "response": "def from_validated_yaml(cls, yaml_str, selectable, **kwargs):\n \"\"\"Create a shelf using a yaml shelf definition.\n\n :param yaml_str: A string containing yaml ingredient definitions.\n :param selectable: A SQLAlchemy Table, a Recipe, or a SQLAlchemy\n join to select from.\n :return: A shelf that contains the ingredients defined in yaml_str.\n \"\"\"\n obj = safe_load(yaml_str)\n return cls.from_config(obj, selectable, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfinds an Ingredient optionally using the shelf.", "response": "def find(self, obj, filter_to_class=Ingredient, constructor=None):\n \"\"\"\n Find an Ingredient, optionally using the shelf.\n\n :param obj: A string or Ingredient\n :param filter_to_class: The Ingredient subclass that obj must be an\n instance of\n :param constructor: An optional callable for building Ingredients\n from obj\n :return: An Ingredient of subclass `filter_to_class`\n \"\"\"\n if callable(constructor):\n obj = constructor(obj, shelf=self)\n\n if isinstance(obj, basestring):\n set_descending = obj.startswith('-')\n if set_descending:\n obj = obj[1:]\n\n if obj not in self:\n raise BadRecipe(\"{} doesn't exist on the shelf\".format(obj))\n\n ingredient = self[obj]\n if not isinstance(ingredient, filter_to_class):\n raise BadRecipe('{} is not a {}'.format(obj, filter_to_class))\n\n if set_descending:\n ingredient.ordering = 'desc'\n\n return ingredient\n elif isinstance(obj, filter_to_class):\n return obj\n else:\n raise BadRecipe('{} is not a {}'.format(obj, filter_to_class))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef brew_query_parts(self):\n columns, group_bys, filters, havings = [], [], set(), set()\n for ingredient in self.ingredients():\n if ingredient.query_columns:\n columns.extend(ingredient.query_columns)\n if ingredient.group_by:\n group_bys.extend(ingredient.group_by)\n if ingredient.filters:\n filters.update(ingredient.filters)\n if ingredient.havings:\n havings.update(ingredient.havings)\n\n return {\n 'columns': columns,\n 'group_bys': group_bys,\n 'filters': filters,\n 'havings': havings,\n }", "response": "Make columns group_bys filters havings\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef enchant(self, list, cache_context=None):\n enchantedlist = []\n if list:\n sample_item = list[0]\n\n # Extra fields to add to each row\n # With extra callables\n extra_fields, extra_callables = [], []\n\n for ingredient in self.values():\n if not isinstance(ingredient, (Dimension, Metric)):\n continue\n if cache_context:\n ingredient.cache_context += str(cache_context)\n for extra_field, extra_callable in ingredient.cauldron_extras:\n extra_fields.append(extra_field)\n extra_callables.append(extra_callable)\n\n # Mixin the extra fields\n keyed_tuple = lightweight_named_tuple(\n 'result', sample_item._fields + tuple(extra_fields)\n )\n\n # Iterate over the results and build a new namedtuple for each row\n for row in list:\n values = row + tuple(fn(row) for fn in extra_callables)\n enchantedlist.append(keyed_tuple(values))\n\n return enchantedlist", "response": "Add any calculated values to each row of a resultset generating a namedtuple containing the values of all the ingredients that have been added for that sample."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef apply_automatic_filters(self, value):\n if self.apply != value:\n self.dirty = True\n self.apply = value\n return self.recipe", "response": "Toggles whether automatic filters are applied to a recipe."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef automatic_filters(self, value):\n assert isinstance(value, dict)\n self._automatic_filters = value\n self.dirty = True\n return self.recipe", "response": "Sets the automatic_filters dictionary to apply to this recipe."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef modify_postquery_parts(self, postquery_parts):\n if self._summarize_over is None:\n return postquery_parts\n assert self._summarize_over in self.recipe.dimension_ids\n\n # Start with a subquery\n subq = postquery_parts['query'].subquery(name='summarize')\n\n summarize_over_dim = set((\n self._summarize_over, self._summarize_over + '_id',\n self._summarize_over + '_raw'\n ))\n dim_column_names = set(dim for dim in self.recipe.dimension_ids).union(\n set(dim + '_id' for dim in self.recipe.dimension_ids)\n ).union(set(dim + '_raw' for dim in self.recipe.dimension_ids))\n used_dim_column_names = dim_column_names - summarize_over_dim\n\n # Build a new query around the subquery\n group_by_columns = [\n col for col in subq.c if col.name in used_dim_column_names\n ]\n\n # Generate columns for the metric, remapping the aggregation function\n # count -> sum\n # sum -> sum\n # avg -> avg\n # Metrics can override the summary aggregation by providing a\n # metric.meta.summary_aggregation callable parameter\n metric_columns = []\n for col in subq.c:\n if col.name not in dim_column_names:\n met = self.recipe._cauldron.find(col.name, Metric)\n summary_aggregation = met.meta.get('summary_aggregation', None)\n if summary_aggregation is None:\n if str(met.expression).startswith(u'avg'):\n summary_aggregation = func.avg\n elif str(met.expression).startswith(u'count'):\n summary_aggregation = func.sum\n elif str(met.expression).startswith(u'sum'):\n summary_aggregation = func.sum\n\n if summary_aggregation is None:\n # We don't know how to aggregate this metric in a summary\n raise BadRecipe(\n u'Provide a summary_aggregation for metric'\n u' {}'.format(col.name)\n )\n metric_columns.append(summary_aggregation(col).label(col.name))\n\n # Find the ordering columns and apply them to the new query\n order_by_columns = []\n for col in postquery_parts['query']._order_by:\n subq_col = getattr(\n subq.c, col.name, getattr(subq.c, col.name + '_raw', None)\n )\n if subq_col is not None:\n order_by_columns.append(subq_col)\n\n postquery_parts['query'] = self.recipe._session.query(\n *(group_by_columns + metric_columns)\n ).group_by(*group_by_columns).order_by(*order_by_columns)\n\n # Remove the summarized dimension\n self.recipe._cauldron.pop(self._summarize_over, None)\n return postquery_parts", "response": "Modify the postquery_parts dictionary with the information from the recipe that has dimensions\n Resummarize it over one of the dimensions returning averages of the aracite metrics."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef anonymize(self, value):\n assert isinstance(value, bool)\n\n if self._anonymize != value:\n self.dirty = True\n self._anonymize = value\n\n # Builder pattern must return the recipe\n return self.recipe", "response": "Should this recipe be anonymized"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_ingredients(self):\n for ingredient in self.recipe._cauldron.values():\n if hasattr(ingredient.meta, 'anonymizer'):\n anonymizer = ingredient.meta.anonymizer\n\n # Build a FakerAnonymizer if we have a string\n if isinstance(anonymizer, basestring):\n\n # Check for extra parameters\n kwargs = {}\n anonymizer_locale = getattr(\n ingredient.meta, 'anonymizer_locale', None\n )\n anonymizer_postprocessor = getattr(\n ingredient.meta, 'anonymizer_postprocessor', None\n )\n if anonymizer_postprocessor is not None:\n kwargs['postprocessor'] = anonymizer_postprocessor\n if anonymizer_locale is not None:\n kwargs['locale'] = anonymizer_locale\n\n anonymizer = FakerAnonymizer(anonymizer, **kwargs)\n\n # Strip out all FakerAnonymizers\n ingredient.formatters = [\n f for f in ingredient.formatters\n if not isinstance(f, FakerAnonymizer)\n ]\n if self._anonymize:\n if ingredient.meta.anonymizer not in ingredient.formatters:\n ingredient.formatters.append(anonymizer)\n else:\n if ingredient.meta.anonymizer in ingredient.formatters:\n ingredient.formatters.remove(anonymizer)", "response": "Add anonymizers to the last position of formatters"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef blend(self, blend_recipe, join_base, join_blend):\n assert isinstance(blend_recipe, Recipe)\n\n self.blend_recipes.append(blend_recipe)\n self.blend_types.append('inner')\n self.blend_criteria.append((join_base, join_blend))\n self.dirty = True\n return self.recipe", "response": "Blends a base recipe into the base recipe."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef modify_postquery_parts(self, postquery_parts):\n if not self.blend_recipes:\n return postquery_parts\n\n for blend_recipe, blend_type, blend_criteria in \\\n zip(self.blend_recipes,\n self.blend_types,\n self.blend_criteria):\n join_base, join_blend = blend_criteria\n\n blend_subq = blend_recipe.subquery()\n\n # For all metrics in the blend recipe\n # Use the metric in the base recipe and\n # Add the metric columns to the base recipe\n for m in blend_recipe.metric_ids:\n met = blend_recipe._cauldron[m]\n self.recipe._cauldron.use(met)\n for suffix in met.make_column_suffixes():\n col = getattr(blend_subq.c, met.id, None)\n if col is not None:\n postquery_parts['query'] = postquery_parts[\n 'query'\n ].add_columns(col.label(met.id + suffix))\n else:\n raise BadRecipe(\n '{} could not be found in .blend() '\n 'recipe subquery'.format(id + suffix)\n )\n\n # For all dimensions in the blend recipe\n # Use the dimension in the base recipe and\n # Add the dimension columns and group_by to the base recipe\n # Ignore the join_blend dimension\n for d in blend_recipe.dimension_ids:\n if d == join_blend:\n continue\n dim = blend_recipe._cauldron[d]\n self.recipe._cauldron.use(dim)\n for suffix in dim.make_column_suffixes():\n col = getattr(blend_subq.c, dim.id, None)\n if col is not None:\n postquery_parts['query'] = postquery_parts[\n 'query'\n ].add_columns(col.label(dim.id + suffix))\n postquery_parts['query'] = postquery_parts[\n 'query'\n ].group_by(col)\n else:\n raise BadRecipe(\n '{} could not be found in .blend() '\n 'recipe subquery'.format(id + suffix)\n )\n\n base_dim = self.recipe._cauldron[join_base]\n blend_dim = blend_recipe._cauldron[join_blend]\n\n base_col = base_dim.columns[0]\n blend_col = getattr(blend_subq.c, blend_dim.id_prop, None)\n if blend_col is None:\n raise BadRecipe(\n 'Can\\'t find join property for {} dimension in \\\n blend recipe'.format(blend_dim.id_prop)\n )\n\n if blend_type == 'outer':\n postquery_parts['query'] = postquery_parts['query'] \\\n .outerjoin(blend_subq, base_col == blend_col)\n else:\n postquery_parts['query'] = postquery_parts['query'] \\\n .join(blend_subq, base_col == blend_col)\n\n return postquery_parts", "response": "Modify the postquery_parts dictionary with the values of the related recipe and the values of the related recipe."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef compare(self, compare_recipe, suffix='_compare'):\n assert isinstance(compare_recipe, Recipe)\n assert isinstance(suffix, basestring)\n self.compare_recipe.append(compare_recipe)\n self.suffix.append(suffix)\n self.dirty = True\n return self.recipe", "response": "Adds a comparison recipe to a base recipe."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef modify_postquery_parts(self, postquery_parts):\n if not self.compare_recipe:\n return postquery_parts\n\n for compare_recipe, compare_suffix in zip(\n self.compare_recipe, self.suffix\n ):\n comparison_subq = compare_recipe.subquery()\n\n # For all metrics in the comparison recipe\n # Use the metric in the base recipe and\n # Add the metric columns to the base recipe\n\n # Comparison metrics hoisted into the base recipe need an\n # aggregation function.The default is func.avg but\n # metrics can override this by provoding a\n # metric.meta.summary_aggregation callable parameter\n for m in compare_recipe.metric_ids:\n met = compare_recipe._cauldron[m]\n id = met.id\n met.id = id + compare_suffix\n summary_aggregation = met.meta.get(\n 'summary_aggregation', func.avg\n )\n self.recipe._cauldron.use(met)\n for suffix in met.make_column_suffixes():\n col = getattr(comparison_subq.c, id + suffix, None)\n if col is not None:\n postquery_parts['query'] = \\\n postquery_parts['query'].add_columns(\n summary_aggregation(col).label(\n met.id + suffix))\n else:\n raise BadRecipe(\n '{} could not be found in .compare() '\n 'recipe subquery'.format(id + suffix)\n )\n\n join_conditions = []\n for dim in compare_recipe.dimension_ids:\n if dim not in self.recipe.dimension_ids:\n raise BadRecipe(\n '{} dimension in comparison recipe must exist '\n 'in base recipe'\n )\n base_dim = self.recipe._cauldron[dim]\n compare_dim = compare_recipe._cauldron[dim]\n base_col = base_dim.columns[0]\n compare_col = getattr(\n comparison_subq.c, compare_dim.id_prop, None\n )\n if compare_col is None:\n raise BadRecipe(\n 'Can\\'t find join property for {} dimension in \\\n compare recipe'.format(compare_dim.id_prop)\n )\n join_conditions.append(base_col == compare_col)\n\n join_clause = text('1=1')\n if join_conditions:\n join_clause = and_(*join_conditions)\n\n postquery_parts['query'] = postquery_parts['query'] \\\n .outerjoin(comparison_subq, join_clause)\n\n return postquery_parts", "response": "Modify the postquery_parts dictionary with the values of the base recipe and the values of the base recipe."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nlists the blobs and files inside a container or share.", "response": "def list(self, container_or_share_name, container=None, account=None):\n \"\"\"List the blobs/files inside a container/share_name.\n Args:\n container_or_share_name(str): Name of the container/share_name where we want to list the blobs/files.\n container(bool): flag to know it you are listing files or blobs.\n account(str): The name of the storage account.\n \"\"\"\n key = self.storage_client.storage_accounts.list_keys(self.resource_group_name, account).keys[0].value\n if container:\n bs = BlockBlobService(account_name=account, account_key=key)\n container_list = []\n for i in bs.list_blobs(container_or_share_name).items:\n container_list.append(i.name)\n return container_list\n elif not container:\n fs = FileService(account_name=account, account_key=key)\n container_list = []\n for i in fs.list_directories_and_files(container_or_share_name).items:\n container_list.append(i.name)\n return container_list\n else:\n raise ValueError(\"You have to pass a value for container param\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngenerating a URL for the remote file to distribute.", "response": "def generate_url(self, remote_file):\n \"\"\"Sign a remote file to distribute. The azure url format is https://myaccount.blob.core.windows.net/mycontainer/myblob.\n Args:\n remote_file(str): The blob that we want to sign.\n \"\"\"\n parse_url = _parse_url(remote_file)\n key = self.storage_client.storage_accounts.list_keys(self.resource_group_name, parse_url.account).keys[0].value\n if parse_url.file_type == 'blob':\n bs = BlockBlobService(account_name=parse_url.account, account_key=key)\n\n sas_token = bs.generate_blob_shared_access_signature(parse_url.container_or_share_name,\n parse_url.file,\n permission=BlobPermissions.READ,\n expiry=datetime.utcnow() + timedelta(hours=24),\n )\n source_blob_url = bs.make_blob_url(container_name=parse_url.container_or_share_name,\n blob_name=parse_url.file,\n sas_token=sas_token)\n return source_blob_url\n elif parse_url.file_type == 'file':\n fs = FileService(account_name=parse_url.account, account_key=key)\n sas_token = fs.generate_file_shared_access_signature(share_name=parse_url.container_or_share_name,\n directory_name=parse_url.path,\n file_name=parse_url.file,\n permission=BlobPermissions.READ,\n expiry=datetime.utcnow() + timedelta(hours=24),\n )\n source_file_url = fs.make_file_url(share_name=parse_url.container_or_share_name,\n directory_name=parse_url.path,\n file_name=parse_url.file,\n sas_token=sas_token)\n return source_file_url\n else:\n raise ValueError(\"This azure storage type is not valid. It should be blob or file.\")"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef delete(self, remote_file):\n if 'core.windows.net' not in remote_file:\n self.logger.error(\"Source or destination must be a azure storage url (format \"\n \"https://myaccount.blob.core.windows.net/mycontainer/myblob\")\n raise OsmosisError\n parse_url = _parse_url(remote_file)\n key = self.storage_client.storage_accounts.list_keys(self.resource_group_name, parse_url.account).keys[0].value\n if parse_url.file_type == 'blob':\n bs = BlockBlobService(account_name=parse_url.account, account_key=key)\n return bs.delete_blob(parse_url.container_or_share_name, parse_url.file)\n elif parse_url.file_type == 'file':\n fs = FileService(account_name=parse_url.account, account_key=key)\n return fs.delete_file(parse_url.container_or_share_name, parse_url.path, parse_url.file)\n else:\n raise ValueError(\"This azure storage type is not valid. It should be blob or file.\")", "response": "Delete a file from the cloud."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncopies a file from one path to another.", "response": "def copy(self, source_path, dest_path, account=None, group_name=None):\n \"\"\"Copy file from a path to another path. The azure url format is https://myaccount.blob.core.windows.net/mycontainer/myblob.\n Args:\n source_path(str): The path of the file to be copied.\n dest_path(str): The destination path where the file is going to be allocated.\n Raises:\n :exc:`~..OsmosisError`: if the file is not uploaded correctly.\n \"\"\"\n if 'core.windows.net' not in source_path and 'core.windows.net' not in dest_path:\n self.logger.error(\"Source or destination must be a azure storage url (format \"\n \"https://myaccount.blob.core.windows.net/mycontainer/myblob\")\n raise OsmosisError\n\n # Check if source exists and can read\n if 'core.windows.net' in source_path:\n parse_url = _parse_url(source_path)\n key = self.storage_client.storage_accounts.list_keys(self.resource_group_name, parse_url.account).keys[\n 0].value\n if parse_url.file_type == 'blob':\n bs = BlockBlobService(account_name=parse_url.account, account_key=key)\n return bs.get_blob_to_path(parse_url.container_or_share_name, parse_url.file, dest_path)\n elif parse_url.file_type == 'file':\n fs = FileService(account_name=parse_url.account, account_key=key)\n return fs.get_file_to_path(parse_url.container_or_share_name, parse_url.path, parse_url.file, dest_path)\n else:\n raise ValueError(\"This azure storage type is not valid. It should be blob or file.\")\n else:\n parse_url = _parse_url(dest_path)\n key = self.storage_client.storage_accounts.list_keys(self.resource_group_name, parse_url.account).keys[\n 0].value\n if parse_url.file_type == 'blob':\n bs = BlockBlobService(account_name=parse_url.account, account_key=key)\n return bs.create_blob_from_path(parse_url.container_or_share_name, parse_url.file, source_path)\n elif parse_url.file_type == 'file':\n fs = FileService(account_name=parse_url.account, account_key=key)\n return fs.create_file_from_path(parse_url.container_or_share_name, parse_url.path, parse_url.file,\n source_path)\n else:\n raise ValueError(\"This azure storage type is not valid. It should be blob or file.\")"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a SQL expression string that can be used to render a SQLAlchemy query.", "response": "def prettyprintable_sql(statement, dialect=None, reindent=True):\n \"\"\"\n Generate an SQL expression string with bound parameters rendered inline\n for the given SQLAlchemy statement. The function can also receive a\n `sqlalchemy.orm.Query` object instead of statement.\n\n WARNING: Should only be used for debugging. Inlining parameters is not\n safe when handling user created data.\n \"\"\"\n if isinstance(statement, sqlalchemy.orm.Query):\n if dialect is None:\n dialect = statement.session.get_bind().dialect\n statement = statement.statement\n\n # Generate a class that can handle encoding\n if dialect:\n DialectKlass = dialect.__class__\n else:\n DialectKlass = DefaultDialect\n\n class LiteralDialect(DialectKlass):\n colspecs = {\n # prevent various encoding explosions\n String: StringLiteral,\n # teach SA about how to literalize a datetime\n DateTime: StringLiteral,\n Date: StringLiteral,\n # don't format py2 long integers to NULL\n NullType: StringLiteral,\n }\n\n compiled = statement.compile(\n dialect=LiteralDialect(), compile_kwargs={\n 'literal_binds': True\n }\n )\n return sqlparse.format(str(compiled), reindent=reindent)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _normalize_coerce_to_format_with_lookup(self, v):\n try:\n return self.format_lookup.get(v, v)\n except TypeError:\n # v is something we can't lookup (like a list)\n return v", "response": "Replace a format with a default value"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _normalize_coerce_to_field_dict(self, v):\n\n def tokenize(s):\n \"\"\" Tokenize a string by splitting it by + and -\n\n >>> tokenize('this + that')\n ['this', '+', 'that']\n\n >>> tokenize('this+that')\n ['this', '+', 'that']\n\n >>> tokenize('this+that-other')\n ['this', '+', 'that', '-', 'other]\n \"\"\"\n\n # Crude tokenization\n s = s.replace('+', ' + ').replace('-', ' - ') \\\n .replace('/', ' / ').replace('*', ' * ')\n words = [w for w in s.split(' ') if w]\n return words\n\n if isinstance(v, _str_type):\n field_parts = tokenize(v)\n field = field_parts[0]\n d = {'value': field}\n if len(field_parts) > 1:\n # if we need to add and subtract from the field join the field\n # parts into pairs, for instance if field parts is\n # [MyTable.first, '-', MyTable.second, '+', MyTable.third]\n # we will get two pairs here\n # [('-', MyTable.second), ('+', MyTable.third)]\n d['operators'] = []\n for operator, other_field in zip(\n field_parts[1::2], field_parts[2::2]\n ):\n d['operators'].append({\n 'operator': operator,\n 'field': {\n 'value': other_field\n }\n })\n return d\n else:\n return v", "response": "Coerce a string to a dict."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _validate_type_scalar(self, value):\n if isinstance(\n value, _int_types + (_str_type, float, date, datetime, bool)\n ):\n return True", "response": "Is not a list or dict"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconstruct a Recipe from a plain Python dictionary.", "response": "def from_config(cls, shelf, obj, **kwargs):\n \"\"\"\n Construct a Recipe from a plain Python dictionary.\n\n Most of the directives only support named ingredients, specified as\n strings, and looked up on the shelf. But filters can be specified as\n objects.\n\n Additionally, each RecipeExtension can extract and handle data from the\n configuration.\n \"\"\"\n def subdict(d, keys):\n new = {}\n for k in keys:\n if k in d:\n new[k] = d[k]\n return new\n\n core_kwargs = subdict(obj, recipe_schema['schema'].keys())\n core_kwargs = normalize_schema(recipe_schema, core_kwargs)\n core_kwargs['filters'] = [\n parse_condition(filter, shelf.Meta.select_from)\n if isinstance(filter, dict)\n else filter\n for filter in obj.get('filters', [])\n ]\n core_kwargs.update(kwargs)\n recipe = cls(shelf=shelf, **core_kwargs)\n\n # Now let extensions handle their own stuff\n for ext in recipe.recipe_extensions:\n additional_schema = getattr(ext, 'recipe_schema', None)\n if additional_schema is not None:\n ext_data = subdict(obj, additional_schema.keys())\n ext_data = normalize_dict(additional_schema, ext_data)\n recipe = ext.from_config(ext_data)\n return recipe"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef shelf(self, shelf=None):\n if shelf is None:\n self._shelf = Shelf({})\n elif isinstance(shelf, Shelf):\n self._shelf = shelf\n elif isinstance(shelf, dict):\n self._shelf = Shelf(shelf)\n else:\n raise BadRecipe('shelf must be a dict or recipe.shelf.Shelf')\n\n if self._select_from is None and \\\n self._shelf.Meta.select_from is not None:\n self._select_from = self._shelf.Meta.select_from\n return self", "response": "Defines a shelf to use for this recipe"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nadding a list of Metric ingredients to the recipe.", "response": "def metrics(self, *metrics):\n \"\"\" Add a list of Metric ingredients to the query. These can either be\n Metric objects or strings representing metrics on the shelf.\n\n The Metric expression will be added to the query's select statement.\n The metric value is a property of each row of the result.\n\n :param metrics: Metrics to add to the recipe. Metrics can\n either be keys on the ``shelf`` or\n Metric objects\n :type metrics: list\n \"\"\"\n for m in metrics:\n self._cauldron.use(self._shelf.find(m, Metric))\n self.dirty = True\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef dimensions(self, *dimensions):\n for d in dimensions:\n self._cauldron.use(self._shelf.find(d, Dimension))\n\n self.dirty = True\n return self", "response": "Add a list of Dimension ingredients to the recipe."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef filters(self, *filters):\n\n def filter_constructor(f, shelf=None):\n if isinstance(f, BinaryExpression):\n return Filter(f)\n else:\n return f\n\n for f in filters:\n self._cauldron.use(\n self._shelf.find(\n f, (Filter, Having), constructor=filter_constructor\n )\n )\n\n self.dirty = True\n return self", "response": "Add a list of filters to the recipe."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nadding a list of ingredients to order_by to the recipe.", "response": "def order_by(self, *order_bys):\n \"\"\" Add a list of ingredients to order by to the query. These can\n either be Dimension or Metric objects or strings representing\n order_bys on the shelf.\n\n The Order_by expression will be added to the query's order_by statement\n\n :param order_bys: Order_bys to add to the recipe. Order_bys can\n either be keys on the ``shelf`` or\n Dimension or Metric objects. If the\n key is prefixed by \"-\" the ordering will be\n descending.\n :type order_bys: list\n \"\"\"\n\n # Order bys shouldn't be added to the _cauldron\n self._order_bys = []\n for ingr in order_bys:\n order_by = self._shelf.find(ingr, (Dimension, Metric))\n self._order_bys.append(order_by)\n\n self.dirty = True\n return self"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nlimiting the number of rows returned from the database.", "response": "def limit(self, limit):\n \"\"\" Limit the number of rows returned from the database.\n\n :param limit: The number of rows to return in the recipe. 0 will\n return all rows.\n :type limit: int\n \"\"\"\n if self._limit != limit:\n self.dirty = True\n self._limit = limit\n return self"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef offset(self, offset):\n if self._offset != offset:\n self.dirty = True\n self._offset = offset\n return self", "response": "Offset the number of rows before returning rows from the database."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _is_postgres(self):\n if self._is_postgres_engine is None:\n is_postgres_engine = False\n try:\n dialect = self.session.bind.engine.name\n if 'redshift' in dialect or 'postg' in dialect or 'pg' in \\\n dialect:\n is_postgres_engine = True\n except:\n pass\n self._is_postgres_engine = is_postgres_engine\n return self._is_postgres_engine", "response": "Determine if the running engine is postgres."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _prepare_order_bys(self):\n order_bys = OrderedSet()\n if self._order_bys:\n for ingredient in self._order_bys:\n if isinstance(ingredient, Dimension):\n # Reverse the ordering columns so that dimensions\n # order by their label rather than their id\n columns = reversed(ingredient.columns)\n else:\n columns = ingredient.columns\n for c in columns:\n order_by = c.desc() if ingredient.ordering == 'desc' else c\n if str(order_by) not in [str(o) for o in order_bys]:\n order_bys.add(order_by)\n\n return list(order_bys)", "response": "Build a list of order by columns"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngenerating a SQLAlchemy query for the current set of ingredients for the current cauldron entry.", "response": "def query(self):\n \"\"\"\n Generates a query using the ingredients supplied by the recipe.\n\n :return: A SQLAlchemy query\n \"\"\"\n if len(self._cauldron.ingredients()) == 0:\n raise BadRecipe('No ingredients have been added to this recipe')\n if not self.dirty and self._query:\n return self._query\n\n # Step 1: Gather up global filters and user filters and\n # apply them as if they had been added to recipe().filters(...)\n\n for extension in self.recipe_extensions:\n extension.add_ingredients()\n\n # Step 2: Build the query (now that it has all the filters\n # and apply any blend recipes\n\n # Get the parts of the query from the cauldron\n # We don't need to regather order_bys\n recipe_parts = self._cauldron.brew_query_parts()\n recipe_parts['order_bys'] = self._prepare_order_bys()\n\n for extension in self.recipe_extensions:\n recipe_parts = extension.modify_recipe_parts(recipe_parts)\n\n # Start building the query\n query = self._session.query(*recipe_parts['columns'])\n if self._select_from is not None:\n query = query.select_from(self._select_from)\n recipe_parts['query'] = query \\\n .group_by(*recipe_parts['group_bys']) \\\n .order_by(*recipe_parts['order_bys']) \\\n .filter(*recipe_parts['filters'])\n\n if recipe_parts['havings']:\n for having in recipe_parts['havings']:\n recipe_parts['query'] = recipe_parts['query'].having(having)\n\n for extension in self.recipe_extensions:\n recipe_parts = extension.modify_prequery_parts(recipe_parts)\n\n if self._select_from is None and len(\n recipe_parts['query'].selectable.froms\n ) != 1:\n raise BadRecipe(\n 'Recipes must use ingredients that all come from '\n 'the same table. \\nDetails on this recipe:\\n{'\n '}'.format(str(self._cauldron))\n )\n\n for extension in self.recipe_extensions:\n recipe_parts = extension.modify_postquery_parts(recipe_parts)\n\n recipe_parts = run_hooks(\n recipe_parts, 'modify_query', self.dynamic_extensions\n )\n\n # Apply limit on the outermost query\n # This happens after building the comparison recipe\n if self._limit and self._limit > 0:\n recipe_parts['query'] = recipe_parts['query'].limit(self._limit)\n\n if self._offset and self._offset > 0:\n recipe_parts['query'] = recipe_parts['query'].offset(self._offset)\n\n # Step 5: Clear the dirty flag,\n # Patch the query if there's a comparison query\n # cache results\n\n self._query = recipe_parts['query']\n self.dirty = False\n return self._query"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef dirty(self):\n if self._dirty:\n return True\n else:\n for extension in self.recipe_extensions:\n if extension.dirty:\n return True\n return False", "response": "Returns True if the recipe is dirty False otherwise"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nset the recipe to dirty flag.", "response": "def dirty(self, value):\n \"\"\" If dirty is true set the recipe to dirty flag. If false,\n clear the recipe and all extension dirty flags \"\"\"\n if value:\n self._dirty = True\n else:\n self._dirty = False\n for extension in self.recipe_extensions:\n extension.dirty = False"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef as_table(self, name=None):\n if name is None:\n name = self._id\n return alias(self.subquery(), name=name)", "response": "Return an alias to a table name."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef all(self):\n starttime = fetchtime = enchanttime = time.time()\n fetched_from_cache = False\n\n if self.dirty or self.all_dirty:\n query = self.query()\n self._all = query.all()\n # If we're using a caching query and that query did not\n # save new values to cache, we got the cached results\n # This is not 100% accurate; it only reports if the caching query\n # attempts to save to cache not the internal state of the cache\n # and whether the cache save actually occurred.\n if not getattr(query, 'saved_to_cache', True):\n fetched_from_cache = True\n fetchtime = time.time()\n self._all = self._cauldron.enchant(\n self._all, cache_context=self.cache_context\n )\n enchanttime = time.time()\n\n self.all_dirty = False\n else:\n # In this case we are using the object self._all as cache\n fetched_from_cache = True\n\n self.stats.set_stats(\n len(self._all), fetchtime - starttime, enchanttime - fetchtime,\n fetched_from_cache\n )\n\n return self._all", "response": "Return a list of result objects."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _validate_condition_keys(self, field, value, error):\n if 'field' in value:\n operators = self.nonscalar_conditions + self.scalar_conditions\n matches = sum(1 for k in operators if k in value)\n if matches == 0:\n error(field, 'Must contain one of {}'.format(operators))\n return False\n elif matches > 1:\n error(\n field,\n 'Must contain no more than one of {}'.format(operators)\n )\n return False\n return True\n elif 'and' in value:\n for condition in value['and']:\n self._validate_condition_keys(field, condition, error)\n elif 'or' in value:\n for condition in value['or']:\n self._validate_condition_keys(field, condition, error)\n else:\n error(field, \"Must contain field + operator keys, 'and', or 'or'.\")\n return False", "response": "Validates that all of the keys in one of the sets of keys in value are defined as keys of value."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nrouting to the survival method returns survivors", "response": "def survival(self,parents,offspring,elite=None,elite_index=None,X=None,X_O=None,F=None,F_O=None):\n \"\"\"routes to the survival method, returns survivors\"\"\"\n if self.sel == 'tournament':\n survivors, survivor_index = self.tournament(parents + offspring, self.tourn_size, num_selections = len(parents))\n elif self.sel == 'lexicase':\n survivor_index = self.lexicase(np.vstack((F,F_O)), num_selections = len(parents), survival = True)\n survivors = [(parents+ offspring)[s] for s in survivor_index]\n elif self.sel == 'epsilon_lexicase':\n # survivors, survivor_index = self.epsilon_lexicase(parents + offspring, num_selections = len(parents), survival = True)\n if self.lex_size:\n sizes = [len(i.stack) for i in (parents + offspring)]\n survivor_index = self.epsilon_lexicase(np.vstack((F,F_O)), sizes, num_selections = F.shape[0], survival = True)\n survivors = [(parents+ offspring)[s] for s in survivor_index]\n else:\n survivor_index = self.epsilon_lexicase(np.vstack((F,F_O)), [], num_selections = F.shape[0], survival = True)\n survivors = [(parents+ offspring)[s] for s in survivor_index]\n elif self.sel == 'deterministic_crowding':\n survivors, survivor_index = self.deterministic_crowding(parents,offspring,X,X_O)\n elif self.sel == 'random':\n # pdb.set_trace()\n survivor_index = self.random_state.permutation(np.arange(2*len(parents)))[:len(parents)]\n survivors = [(parents + offspring)[s] for s in survivor_index]\n # elitism\n if self.elitism:\n if min([x.fitness for x in survivors]) > elite.fitness:\n # if the elite individual did not survive and elitism is on, replace worst individual with elite\n rep_index = np.argmax([x.fitness for x in survivors])\n survivors[rep_index] = elite\n survivor_index[rep_index] = elite_index\n # return survivors\n\n return survivors,survivor_index"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef tournament(self,individuals,tourn_size, num_selections=None):\n winners = []\n locs = []\n if num_selections is None:\n num_selections = len(individuals)\n\n for i in np.arange(num_selections):\n # sample pool with replacement\n pool_i = self.random_state.choice(len(individuals),size=tourn_size)\n pool = []\n for i in pool_i:\n pool.append(np.mean(individuals[i].fitness))\n # winner\n locs.append(pool_i[np.argmin(pool)])\n winners.append(copy.deepcopy(individuals[locs[-1]]))\n\n return winners,locs", "response": "conducts tournament selection of size tourn_size"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef lexicase(self,F, num_selections=None, survival = False):\n if num_selections is None:\n num_selections = F.shape[0]\n winners = []\n locs = []\n\n individual_locs = np.arange(F.shape[0])\n \n for i in np.arange(num_selections):\n can_locs = individual_locs\n cases = list(np.arange(F.shape[1]))\n self.random_state.shuffle(cases)\n # pdb.set_trace()\n while len(cases) > 0 and len(can_locs) > 1:\n # get best fitness for case among candidates\n best_val_for_case = np.min(F[can_locs,cases[0]])\n # filter individuals without an elite fitness on this case\n can_locs = [l for l in can_locs if F[l,cases[0]] <= best_val_for_case ]\n cases.pop(0)\n\n choice = self.random_state.randint(len(can_locs))\n locs.append(can_locs[choice])\n if survival: # filter out winners from remaining selection pool\n individual_locs = [i for i in individual_locs if i != can_locs[choice]]\n\n while len(locs) < num_selections:\n locs.append(individual_locs[0])\n\n return locs", "response": "conducts lexicase selection for de - aggregated fitness vectors"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconduct epsilon lexicase selection for de - aggregated fitness vectors", "response": "def epsilon_lexicase(self, F, sizes, num_selections=None, survival = False):\n \"\"\"conducts epsilon lexicase selection for de-aggregated fitness vectors\"\"\"\n # pdb.set_trace()\n if num_selections is None:\n num_selections = F.shape[0]\n\n if self.c: # use c library\n # define c types\n locs = np.empty(num_selections,dtype='int32',order='F')\n # self.lib.epsilon_lexicase(F,F.shape[0],F.shape[1],num_selections,locs)\n if self.lex_size:\n ep_lex(F,F.shape[0],F.shape[1],num_selections,locs,self.lex_size,np.array(sizes))\n else:\n ep_lex(F,F.shape[0],F.shape[1],num_selections,locs,self.lex_size,np.array([]))\n return locs\n else: # use python version\n \n locs = []\n individual_locs = np.arange(F.shape[0])\n # calculate epsilon thresholds based on median absolute deviation (MAD)\n mad_for_case = np.array([self.mad(f) for f in F.transpose()])\n for i in np.arange(num_selections):\n\n can_locs = individual_locs\n cases = list(np.arange(F.shape[1]))\n self.random_state.shuffle(cases)\n # pdb.set_trace()\n while len(cases) > 0 and len(can_locs) > 1:\n # get best fitness for case among candidates\n best_val_for_case = np.min(F[can_locs,cases[0]])\n # filter individuals without an elite fitness on this case\n can_locs = [l for l in can_locs if F[l,cases[0]] <= best_val_for_case + mad_for_case[cases[0]]]\n cases.pop(0)\n\n choice = self.random_state.randint(len(can_locs))\n locs.append(can_locs[choice])\n if survival: # filter out winners from remaining selection pool\n individual_locs = [i for i in individual_locs if i != can_locs[choice]]\n\n while len(locs) < num_selections:\n locs.append(individual_locs[0])\n\n return locs"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef mad(self,x, axis=None):\n return np.median(np.abs(x - np.median(x, axis)), axis)", "response": "median absolute deviation statistic"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef deterministic_crowding(self,parents,offspring,X_parents,X_offspring):\n # get children locations produced from crossover\n cross_children = [i for i,o in enumerate(offspring) if len(o.parentid) > 1]\n # order offspring so that they are lined up with their most similar parent\n for c1,c2 in zip(cross_children[::2], cross_children[1::2]):\n # get parent locations\n p_loc = [j for j,p in enumerate(parents) if p.id in offspring[c1].parentid]\n if len(p_loc) != 2:\n continue\n # if child is more correlated with its non-root parent\n if r2_score(X_parents[p_loc[0]],X_offspring[c1]) + r2_score(X_parents[p_loc[1]],X_offspring[c2]) < r2_score(X_parents[p_loc[0]],X_offspring[c2]) + r2_score(X_parents[p_loc[1]],X_offspring[c1]):\n # swap offspring\n offspring[c1],offspring[c2] = offspring[c2],offspring[c1]\n\n survivors = []\n survivor_index = []\n\n for i,(p,o) in enumerate(zip(parents,offspring)):\n if p.fitness >= o.fitness:\n survivors.append(copy.deepcopy(p))\n survivor_index.append(i)\n else:\n survivors.append(copy.deepcopy(o))\n survivor_index.append(i+len(parents))\n\n # return survivors along with their indices\n return survivors, survivor_index", "response": "deterministic crowding implementation (for non-steady state).\n offspring compete against the parent they are most similar to, here defined as\n the parent they are most correlated with.\n the offspring only replace their parent if they are more fit."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nperform variation operators on parents.", "response": "def variation(self,parents):\n \"\"\"performs variation operators on parents.\"\"\"\n # downselect to features that are important\n if (self.valid(parents) and\n self.ml_type != 'SVC' and self.ml_type != 'SVR'):\n # this is needed because svm has a bug that throws valueerror on\n # attribute check\n if hasattr(self.pipeline.named_steps['ml'],'coef_'):\n # for l1 regularization, filter individuals with 0 coefficients\n if self.weight_parents:\n weights = self.pipeline.named_steps['ml'].coef_\n if len(weights.shape)>1: # handle multi-coefficient models\n weights = [np.mean(abs(c)) for c in weights.transpose()]\n # softmax transformation of the weights\n weights = np.exp(weights)/np.sum(np.exp(weights))\n offspring = copy.deepcopy(\n list(self.random_state.choice(self.valid(parents),\n self.population_size, p=weights)))\n else:\n offspring = copy.deepcopy(list(\n x for i,x in zip(self.pipeline.named_steps['ml'].coef_,\n self.valid(parents)) if (i != 0).any()))\n elif hasattr(self.pipeline.named_steps['ml'],'feature_importances_'):\n # for tree methods, filter our individuals with 0 feature importance\n if self.weight_parents:\n weights = self.pipeline.named_steps['ml'].feature_importances_\n # softmax transformation of the weights\n weights = np.exp(weights)/np.sum(np.exp(weights))\n offspring = copy.deepcopy(list(\n self.random_state.choice(self.valid(parents),\n self.population_size, p=weights)))\n else:\n offspring = copy.deepcopy(list(\n x for i,x in zip(self.pipeline.named_steps['ml'].feature_importances_,\n self.valid(parents)) if i != 0))\n else:\n offspring = copy.deepcopy(self.valid(parents))\n else:\n offspring = copy.deepcopy(self.valid(parents))\n\n if self.elitism: # keep a copy of the elite individual\n elite_index = np.argmin([x.fitness for x in parents])\n elite = copy.deepcopy(parents[elite_index])\n\n # Apply crossover and mutation on the offspring\n if self.verbosity > 2: print(\"variation...\")\n for child1, child2 in it.zip_longest(offspring[::2], offspring[1::2],\n fillvalue=None):\n\n if self.random_state.rand() < self.crossover_rate and child2 != None:\n # crossover\n self.cross(child1.stack, child2.stack, self.max_depth)\n # update ids\n child1.parentid = [child1.id,child2.id]\n child1.id = uuid.uuid4()\n child2.parentid = [child1.id,child2.id]\n child2.id = uuid.uuid4()\n # set default fitness\n child1.fitness = -1\n child2.fitness = -1\n elif child2 == None:\n # single mutation\n self.mutate(child1.stack,self.func_set,self.term_set)\n # update ids\n child1.parentid = [child1.id]\n child1.id = uuid.uuid4()\n # set default fitness\n child1.fitness = -1\n else:\n #double mutation\n self.mutate(child1.stack,self.func_set,self.term_set)\n self.mutate(child2.stack,self.func_set,self.term_set)\n # update ids\n child1.parentid = [child1.id]\n child1.id = uuid.uuid4()\n child2.parentid = [child2.id]\n child2.id = uuid.uuid4()\n # set default fitness\n child1.fitness = -1\n child2.fitness = -1\n\n while len(offspring) < self.population_size:\n #make new offspring to replace the invalid ones\n offspring.append(Ind())\n self.make_program(offspring[-1].stack,self.func_set,self.term_set,\n self.random_state.randint(self.min_depth,\n self.max_depth+1),\n self.otype)\n offspring[-1].stack = list(reversed(offspring[-1].stack))\n \n return offspring,elite,elite_index"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef cross(self,p_i,p_j, max_depth = 2):\n # only choose crossover points for out_types available in both programs\n # pdb.set_trace()\n # determine possible outttypes\n types_p_i = [t for t in [p.out_type for p in p_i]]\n types_p_j = [t for t in [p.out_type for p in p_j]]\n types = set(types_p_i).intersection(types_p_j)\n\n # grab subtree of p_i\n p_i_sub = [i for i,n in enumerate(p_i) if n.out_type in types]\n x_i_end = self.random_state.choice(p_i_sub)\n x_i_begin = x_i_end\n arity_sum = p_i[x_i_end].arity[p_i[x_i_end].in_type]\n # print(\"x_i_end:\",x_i_end)\n # i = 0\n while (arity_sum > 0): #and i < 1000:\n if x_i_begin == 0:\n print(\"arity_sum:\",arity_sum,\"x_i_begin:\",x_i_begin,\"x_i_end:\",x_i_end)\n x_i_begin -= 1\n arity_sum += p_i[x_i_begin].arity[p_i[x_i_begin].in_type]-1\n # i += 1\n # if i == 1000:\n # print(\"in variation\")\n # pdb.set_trace()\n\n # grab subtree of p_j with matching out_type to p_i[x_i_end]\n p_j_sub = [i for i,n in enumerate(p_j) if n.out_type == p_i[x_i_end].out_type]\n x_j_end = self.random_state.choice(p_j_sub)\n x_j_begin = x_j_end\n arity_sum = p_j[x_j_end].arity[p_j[x_j_end].in_type]\n # i = 0\n while (arity_sum > 0): #and i < 1000:\n if x_j_begin == 0:\n print(\"arity_sum:\",arity_sum,\"x_j_begin:\",x_j_begin,\"x_j_end:\",x_j_end)\n print(\"p_j:\",p_j)\n x_j_begin -= 1\n arity_sum += p_j[x_j_begin].arity[p_j[x_j_begin].in_type]-1\n # i += 1\n # if i == 1000:\n # print(\"in variation\")\n # pdb.set_trace()\n #swap subtrees\n tmpi = p_i[:]\n tmpj = p_j[:]\n tmpi[x_i_begin:x_i_end+1:],tmpj[x_j_begin:x_j_end+1:] = \\\n tmpj[x_j_begin:x_j_end+1:],tmpi[x_i_begin:x_i_end+1:]\n\n if not self.is_valid_program(p_i) or not self.is_valid_program(p_j):\n # pdb.set_trace()\n print(\"parent 1:\",p_i,\"x_i_begin:\",x_i_begin,\"x_i_end:\",x_i_end)\n print(\"parent 2:\",p_j,\"x_j_begin:\",x_j_begin,\"x_j_end:\",x_j_end)\n print(\"child 1:\",tmpi)\n print(\"child 2:\",tmpj)\n raise ValueError('Crossover produced an invalid program.')\n\n # size check, then assignment\n if len(tmpi) <= 2**max_depth-1:\n p_i[:] = tmpi\n if len(tmpj) <= 2**max_depth-1:\n p_j[:] = tmpj", "response": "subtree - like swap crossover between programs p_i and p_j."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nmutating a set of functions and terms", "response": "def mutate(self,p_i,func_set,term_set): #, max_depth=2\n \"\"\"point mutation, addition, removal\"\"\"\n self.point_mutate(p_i,func_set,term_set)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef point_mutate(self,p_i,func_set,term_set):\n # point mutation\n x = self.random_state.randint(len(p_i))\n arity = p_i[x].arity[p_i[x].in_type]\n # find eligible replacements based on arity and type\n reps = [n for n in func_set+term_set\n if n.arity[n.in_type]==arity and n.out_type==p_i[x].out_type\n and n.in_type==p_i[x].in_type]\n\n tmp = reps[self.random_state.randint(len(reps))]\n tmp_p = p_i[:]\n p_i[x] = tmp\n if not self.is_valid_program(p_i):\n print(\"old:\",tmp_p)\n print(\"new:\",p_i)\n raise ValueError('Mutation produced an invalid program.')", "response": "point mutation on individual p_i"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking whether a program p makes a syntactically valid tree.", "response": "def is_valid_program(self,p):\n \"\"\"checks whether program p makes a syntactically valid tree.\n\n checks that the accumulated program length is always greater than the\n accumulated arities, indicating that the appropriate number of arguments is\n alway present for functions. It then checks that the sum of arties +1\n exactly equals the length of the stack, indicating that there are no\n missing arguments.\n \"\"\"\n # print(\"p:\",p)\n arities = list(a.arity[a.in_type] for a in p)\n accu_arities = list(accumulate(arities))\n accu_len = list(np.arange(len(p))+1)\n check = list(a < b for a,b in zip(accu_arities,accu_len))\n # print(\"accu_arities:\",accu_arities)\n # print(\"accu_len:\",accu_len)\n # print(\"accu_arities < accu_len:\",accu_arities<accu_len)\n return all(check) and sum(a.arity[a.in_type] for a in p) +1 == len(p) and len(p)>0"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef run_MDR(n,stack_float,labels=None):\n # need to check that tmp is categorical\n\n x1 = stack_float.pop()\n x2 = stack_float.pop()\n # check data is categorical\n if len(np.unique(x1))<=3 and len(np.unique(x2))<=3:\n tmp = np.vstack((x1,x2)).transpose()\n\n if labels is None: # prediction\n return n.model.transform(tmp)[:,0]\n else: # training\n out = n.model.fit_transform(tmp,labels)[:,0]\n\n return out\n else:\n return np.zeros(x1.shape[0])", "response": "run utility function for MDR nodes."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns equation string for program stack", "response": "def stack_2_eqn(self,p):\n \"\"\"returns equation string for program stack\"\"\"\n stack_eqn = []\n if p: # if stack is not empty\n for n in p.stack:\n self.eval_eqn(n,stack_eqn)\n return stack_eqn[-1]\n return []"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef stacks_2_eqns(self,stacks):\n if stacks:\n return list(map(lambda p: self.stack_2_eqn(p), stacks))\n else:\n return []", "response": "returns equation strings from stacks"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nmaking a program stack", "response": "def make_program(self,stack,func_set,term_set,max_d,ntype):\n \"\"\"makes a program stack\"\"\"\n # print(\"stack:\",stack,\"max d:\",max_d)\n if max_d == 0:\n ts = [t for t in term_set if t.out_type==ntype]\n\n if not ts:\n raise ValueError('no ts. ntype:'+ntype+'. term_set out_types:'+\n ','.join([t.out_type for t in term_set]))\n\n stack.append(ts[self.random_state.choice(len(ts))])\n else:\n fs = [f for f in func_set if (f.out_type==ntype\n and (f.in_type=='f' or max_d>1))]\n if len(fs)==0:\n print('ntype:',ntype,'\\nfunc_set:',[f.name for f in func_set])\n stack.append(fs[self.random_state.choice(len(fs))])\n tmp = copy.copy(stack[-1])\n\n for i in np.arange(tmp.arity['f']):\n self.make_program(stack,func_set,term_set,max_d-1,'f')\n for i in np.arange(tmp.arity['b']):\n self.make_program(stack,func_set,term_set,max_d-1,'b')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef init_pop(self):\n pop = Pop(self.population_size)\n seed_with_raw_features = False\n # make programs\n if self.seed_with_ml:\n # initial population is the components of the default ml model\n if (self.ml_type == 'SVC' or self.ml_type == 'SVR'):\n # this is needed because svm has a bug that throws valueerror\n #on attribute check\n seed_with_raw_features=True\n elif (hasattr(self.pipeline.named_steps['ml'],'coef_') or\n hasattr(self.pipeline.named_steps['ml'],'feature_importances_')):\n # add model components with non-zero coefficients to initial\n # population, in order of coefficient size\n coef = (self.pipeline.named_steps['ml'].coef_ if\n hasattr(self.pipeline.named_steps['ml'],'coef_') else\n self.pipeline.named_steps['ml'].feature_importances_)\n # compress multiple coefficients for each feature into single\n # numbers (occurs with multiclass classification)\n if len(coef.shape)>1:\n coef = [np.mean(abs(c)) for c in coef.transpose()]\n\n # remove zeros\n coef = [c for c in coef if c!=0]\n # sort feature locations based on importance/coefficient\n locs = np.arange(len(coef))\n locs = locs[np.argsort(np.abs(coef))[::-1]]\n for i,p in enumerate(pop.individuals):\n if i < len(locs):\n p.stack = [node('x',loc=locs[i])]\n else:\n # make program if pop is bigger than n_features\n self.make_program(p.stack,self.func_set,self.term_set,\n self.random_state.randint(self.min_depth,\n self.max_depth+1),\n self.otype)\n p.stack = list(reversed(p.stack))\n else:\n seed_with_raw_features = True\n # seed with random features if no importance info available\n if seed_with_raw_features:\n for i,p in enumerate(pop.individuals):\n if i < self.n_features:\n p.stack = [node('x',\n loc=self.random_state.randint(self.n_features))]\n else:\n # make program if pop is bigger than n_features\n self.make_program(p.stack,self.func_set,self.term_set,\n self.random_state.randint(self.min_depth,\n self.max_depth+1),\n self.otype)\n p.stack = list(reversed(p.stack))\n\n # print initial population\n if self.verbosity > 2:\n print(\"seeded initial population:\",\n self.stacks_2_eqns(pop.individuals))\n\n else: # don't seed with ML\n for I in pop.individuals:\n depth = self.random_state.randint(self.min_depth,self.max_depth_init)\n self.make_program(I.stack,self.func_set,self.term_set,depth,\n self.otype)\n #print(I.stack)\n I.stack = list(reversed(I.stack))\n\n return pop", "response": "initializes population of features as GP stacks."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef main():\n parser = argparse.ArgumentParser(description='A feature engineering wrapper'\n ' for machine learning algorithms.',\n add_help=False)\n\n parser.add_argument('INPUT_FILE', type=str,\n help='Data file to run FEW on; ensure that the '\n 'target/label column is labeled as \"label\" or \"class\".')\n\n parser.add_argument('-h', '--help', action='help',\n help='Show this help message and exit.')\n\n parser.add_argument('-is', action='store',dest='INPUT_SEPARATOR',\n default=None,type=str,\n help='Character separating columns in the input file.')\n\n parser.add_argument('-o', action='store', dest='OUTPUT_FILE', default='',\n type=str, help='File to export the final model.')\n\n parser.add_argument('-g', action='store', dest='GENERATIONS', default=100,\n type=positive_integer,\n help='Number of generations to run FEW.')\n\n parser.add_argument('-p', action='store', dest='POPULATION_SIZE',default=50,\n help='Number of individuals in the GP population. '\n 'Follow the number with x to set population size as a'\n 'multiple of raw feature size.')\n\n parser.add_argument('-mr', action='store', dest='MUTATION_RATE',default=0.5,\n type=float_range,\n help='GP mutation rate in the range [0.0, 1.0].')\n\n parser.add_argument('-xr', action='store', dest='CROSSOVER_RATE',\n default=0.5,type=float_range,\n help='GP crossover rate in the range [0.0, 1.0].')\n\n parser.add_argument('-ml', action='store', dest='MACHINE_LEARNER',\n default=None,\n choices = ['lasso','svr','lsvr','lr','svc','rfc','rfr',\n 'dtc','dtr','dc','knc','knr','sgd'],\n type=str, help='ML algorithm to pair with features. '\n 'Default: Lasso (regression), LogisticRegression '\n '(classification)')\n\n parser.add_argument('-min_depth', action='store', dest='MIN_DEPTH',\n default=1,type=positive_integer,\n help='Minimum length of GP programs.')\n\n parser.add_argument('-max_depth', action='store', dest='MAX_DEPTH',\n default=2,type=positive_integer,\n help='Maximum number of nodes in GP programs.')\n\n parser.add_argument('-max_depth_init', action='store',dest='MAX_DEPTH_INIT',\n default=2,type=positive_integer,\n help='Maximum nodes in initial programs.')\n\n parser.add_argument('-op_weight', action='store',dest='OP_WEIGHT',default=1,\n type=bool, help='Weight attributes for incuded in'\n ' features based on ML scores. Default: off')\n\n parser.add_argument('-ms', action='store', dest='MAX_STALL',default=100,\n type=positive_integer, help='If model CV does not '\n 'improve for this many generations, end optimization.')\n\n parser.add_argument('--weight_parents', action='store_true',\n dest='WEIGHT_PARENTS',default=True,\n help='Feature importance weights parent selection.')\n\n parser.add_argument('--lex_size', action='store_true',dest='LEX_SIZE',default=False,\n help='Size mediated parent selection for lexicase survival.')\n\n parser.add_argument('-sel', action='store', dest='SEL',\n default='epsilon_lexicase',\n choices = ['tournament','lexicase','epsilon_lexicase',\n 'deterministic_crowding','random'],\n type=str, help='Selection method (Default: tournament)')\n\n parser.add_argument('-tourn_size', action='store', dest='TOURN_SIZE',\n default=2, type=positive_integer,\n help='Tournament size (Default: 2)')\n\n parser.add_argument('-fit', action='store', dest='FIT_CHOICE', default=None,\n choices = ['mse','mae','r2','vaf','mse_rel','mae_rel',\n 'r2_rel','vaf_rel','silhouette','inertia',\n 'separation','fisher','random','relief'],\n type=str,\n help='Fitness metric (Default: dependent on ml used)')\n\n parser.add_argument('--no_seed', action='store_false', dest='SEED_WITH_ML',\n default=True,\n help='Turn off initial GP population seeding.')\n\n parser.add_argument('--elitism', action='store_true', dest='ELITISM',\n default=False,\n help='Force survival of best feature in GP population.')\n\n parser.add_argument('--erc', action='store_true', dest='ERC', default=False,\n help='Use random constants in GP feature construction.')\n\n parser.add_argument('--bool', action='store_true', dest='BOOLEAN',\n default=False,\n help='Include boolean operators in features.')\n\n parser.add_argument('-otype', action='store', dest='OTYPE', default='f',\n choices=['f','b'],\n type=str,\n help='Feature output type. f: float, b: boolean.')\n\n parser.add_argument('-ops', action='store', dest='OPS', default=None,\n type=str,\n help='Specify operators separated by commas')\n \n parser.add_argument('-dtypes', action='store', dest='DTYPES', default=None,\n type=str,\n help='Specify datafile types separated by a comma')\n \n parser.add_argument('--class', action='store_true', dest='CLASSIFICATION',\n default=False,\n help='Conduct classification rather than regression.')\n\n parser.add_argument('--mdr', action='store_true',dest='MDR',default=False,\n help='Use MDR nodes.')\n\n parser.add_argument('--nonorm', action='store_false',dest='NORMALIZE',default=True,\n help='Disable standard scaler preprocessor.')\n\n parser.add_argument('--diversity', action='store_true',\n dest='TRACK_DIVERSITY', default=False,\n help='Store diversity of feature transforms each gen.')\n\n parser.add_argument('--clean', action='store_true', dest='CLEAN',\n default=False,\n help='Clean input data of missing values.')\n\n parser.add_argument('--no_lib', action='store_false', dest='c',\n default=True,\n help='Don''t use optimized c libraries.')\n\n parser.add_argument('-s', action='store', dest='RANDOM_STATE',\n default=None,\n type=int,\n help='Random number generator seed for reproducibility.'\n 'Note that using multi-threading may make exact results'\n ' impossible to reproduce.')\n\n parser.add_argument('-v', action='store', dest='VERBOSITY', default=1,\n choices=[0, 1, 2, 3], type=int,\n help='How much information FEW communicates while it is'\n ' running: 0 = none, 1 = minimal, 2 = lots, 3 = all.')\n\n parser.add_argument('--no-update-check', action='store_true',\n dest='DISABLE_UPDATE_CHECK', default=False,\n help='Don''t check the FEW version.')\n\n parser.add_argument('--version', action='version',\n version='FEW {version}'.format(version=__version__),\n help='Show FEW\\'s version number and exit.')\n\n args = parser.parse_args()\n\n # if args.VERBOSITY >= 2:\n # print('\\nFEW settings:')\n # for arg in sorted(args.__dict__):\n # if arg == 'DISABLE_UPDATE_CHECK':\n # continue\n # print('{}\\t=\\t{}'.format(arg, args.__dict__[arg]))\n # print('')\n\n # load data from csv file\n if args.INPUT_SEPARATOR is None:\n input_data = pd.read_csv(args.INPUT_FILE, sep=args.INPUT_SEPARATOR,\n engine='python')\n else: # use c engine for read_csv is separator is specified\n input_data = pd.read_csv(args.INPUT_FILE, sep=args.INPUT_SEPARATOR)\n\n # if 'Label' in input_data.columns.values:\n input_data.rename(columns={'Label': 'label','Class':'label','class':'label',\n 'target':'label'}, inplace=True)\n\n RANDOM_STATE = args.RANDOM_STATE\n\n train_i, test_i = train_test_split(input_data.index,\n stratify = None,\n #stratify=input_data['label'].values,\n train_size=0.75,\n test_size=0.25,\n random_state=RANDOM_STATE)\n\n training_features = input_data.loc[train_i].drop('label', axis=1).values\n training_labels = input_data.loc[train_i, 'label'].values\n \n testing_features = input_data.loc[test_i].drop('label', axis=1).values\n testing_labels = input_data.loc[test_i, 'label'].values\n\n learner = FEW(generations=args.GENERATIONS,\n population_size=args.POPULATION_SIZE,\n mutation_rate=args.MUTATION_RATE,\n crossover_rate=args.CROSSOVER_RATE,\n ml = ml_dict[args.MACHINE_LEARNER],\n min_depth = args.MIN_DEPTH,max_depth = args.MAX_DEPTH,\n sel = args.SEL, tourn_size = args.TOURN_SIZE,\n seed_with_ml = args.SEED_WITH_ML, op_weight = args.OP_WEIGHT,\n max_stall = args.MAX_STALL,\n erc = args.ERC, random_state=args.RANDOM_STATE,\n verbosity=args.VERBOSITY,\n disable_update_check=args.DISABLE_UPDATE_CHECK,\n fit_choice = args.FIT_CHOICE,boolean=args.BOOLEAN,\n classification=args.CLASSIFICATION,clean = args.CLEAN,\n track_diversity=args.TRACK_DIVERSITY,mdr=args.MDR,\n otype=args.OTYPE,c=args.c, lex_size = args.LEX_SIZE,\n weight_parents = args.WEIGHT_PARENTS,operators=args.OPS,\n normalize=args.NORMALIZE, dtypes = args.DTYPES)\n\n learner.fit(training_features, training_labels)\n # pdb.set_trace()\n if args.VERBOSITY >= 1:\n print('\\nTraining accuracy: {:1.3f}'.format(\n learner.score(training_features, training_labels)))\n print('Test accuracy: {:1.3f}'.format(\n learner.score(testing_features, testing_labels)))\n\n if args.OUTPUT_FILE != '':\n learner.export(args.OUTPUT_FILE)", "response": "This function is called by the command line to run the FEW."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fit(self, features, labels):\n\n # setup data\n # imputation\n if self.clean:\n features = self.impute_data(features)\n # save the number of features\n self.n_features = features.shape[1]\n self.n_samples = features.shape[0]\n # set population size\n if type(self.population_size) is str:\n if 'x' in self.population_size: # set pop size prop to features\n self.population_size = int(\n float(self.population_size[:-1])*features.shape[1])\n else:\n self.population_size = int(self.population_size)\n\n if self.verbosity >0: print(\"population size:\",self.population_size)\n \n\n # re-initialize pipeline (needs to be here rather than init for GridSearchCV)\n if self.normalize:\n self.pipeline = Pipeline([('standardScaler',StandardScaler()), ('ml', self.ml)])\n else:\n self.pipeline = Pipeline([('ml',self.ml)])\n\n # set variable names if they haven't been set\n if self.names is None:\n self.names = ['x_'+str(i) for i in np.arange(features.shape[1])]\n # set variable data types if they haven't been set\n if self.dtypes is None:\n self.dtypes = ['f' for i in np.arange(features.shape[1])]\n \n # create terminal set\n for i in np.arange(self.n_features):\n self.term_set.append(node('x',loc=i,otype=self.dtypes[i])) # features\n # add ephemeral random constants if flag\n if self.erc: # ephemeral random constants\n self.term_set.append(node('k',value=self.random_state.rand()))\n\n # edit function set if boolean\n if self.boolean or self.otype=='b': # include boolean functions\n self.func_set += [node('!'), node('&'), node('|'), node('=='),\n node('>_f'), node('<_f'), node('>=_f'), node('<=_f'),\n node('>_b'), node('<_b'), node('>=_b'), node('<=_b'),\n node('xor_b'), node('xor_f')]\n\n # add mdr if specified\n if self.mdr:\n self.func_set += [node('mdr2')]\n\n # print few settings\n if self.verbosity > 1:\n for arg in self.get_params():\n print('{}\\t=\\t{}'.format(arg, self.get_params()[arg]))\n print('')\n \n ######################################################### initial model\n # fit to original data\n with warnings.catch_warnings():\n warnings.simplefilter(\"ignore\")\n if self.scoring_function == roc_auc_score:\n self._best_score = self.roc_auc_cv(features,labels)\n else:\n self._best_score = np.mean(\n [self.scoring_function(labels[test],\n self.pipeline.fit(features[train],labels[train]).\n predict(features[test]))\n for train, test in KFold().split(features,\n labels)])\n\n initial_score = self._best_score\n if self.verbosity > 0:\n print(\"initial ML CV: {:1.3f}\".format(self._best_score))\n\n \n ############################################# Create initial population\n # for now, force seed_with_ml to be off if otype is 'b', since data\n # types are assumed to be float\n if self.otype=='b':\n self.seed_with_ml = False\n self.pop = self.init_pop()\n # check that uuids are unique in population\n uuids = [p.id for p in self.pop.individuals]\n if len(uuids) != len(set(uuids)):\n pdb.set_trace()\n # Evaluate the entire population\n # X represents a matrix of the population outputs (number of samples x\n # population size)\n # single thread\n self.X = self.transform(features,self.pop.individuals,labels).transpose()\n # pdb.set_trace()\n # parallel:\n # X = np.asarray(Parallel(n_jobs=-1)(\n # delayed(out)(I,features,self.otype,labels) for I in self.pop.individuals),\n # order = 'F')\n\n # calculate fitness of individuals\n # fitnesses = list(map(lambda I: fitness(I,labels,self.pipeline),X))\n self.F = self.calc_fitness(self.X,labels,self.fit_choice,self.sel)\n #pdb.set_trace()\n #with Parallel(n_jobs=10) as parallel:\n ####################\n\n self.diversity=[]\n # progress bar\n pbar = tqdm(total=self.generations,disable = self.verbosity==0,\n desc='Internal CV: {:1.3f}'.format(self._best_score))\n stall_count = 0\n ########################################################### main GP loop\n for g in np.arange(self.generations):\n if stall_count == self.max_stall:\n if self.verbosity > 0: print('max stall count reached.')\n break\n\n if self.track_diversity:\n self.get_diversity(self.X)\n\n # mid verbosity printouts\n if self.verbosity > 1:\n \n print(\"generation\", str(g)) \n \n print(\"median fitness pop: %0.2f\" % np.median(\n [np.mean(f) for f in self.F]))\n \n print(\"best fitness pop: %0.2f\" % np.min(\n [np.mean(f) for f in self.F]))\n if self.track_diversity:\n print(\"feature diversity: %0.2f\" % self.diversity[-1])\n\n # high verbosity printouts \n if self.verbosity > 2:\n eqns = self.stacks_2_eqns(self.pop.individuals)\n fs = [np.mean(f) for f in self.F] \n print(\"population:\",[(\"%0.2f\" % f, eqns[i]) for f,i in \n zip(np.sort(fs), np.argsort(fs))])\n #print(\"pop fitnesses:\", [\"%0.2f\" % np.mean(f) for f in self.F])\n \n ####################################################### fit ml model\n if self.verbosity > 1: print(\"ml fitting...\")\n\n tmp_score=0\n with warnings.catch_warnings():\n warnings.simplefilter(\"ignore\")\n try:\n if self.valid_loc():\n if self.scoring_function == roc_auc_score:\n tmp_score = self.roc_auc_cv(self.X[self.valid_loc(),:].transpose(),\n labels)\n else:\n tmp_score = np.mean(\n [self.scoring_function(labels[test], self.pipeline.fit(\n self.X[self.valid_loc(),:].transpose()[train], labels[train]).\n predict(self.X[self.valid_loc(),:].transpose()[test]))\n for train, test in KFold().split(features,\n labels)])\n\n except ValueError as detail:\n print(\"warning: ValueError in ml fit. X.shape:\",\n self.X[:,self.valid_loc()].transpose().shape,\n \"labels shape:\",labels.shape)\n print(\"First ten entries X:\",\n self.X[self.valid_loc(),:].transpose()[:10])\n print(\"First ten entries labels:\",labels[:10])\n print(\"equations:\",self.stacks_2_eqns(self.pop.individuals))\n print(\"FEW parameters:\",self.get_params())\n print(\"---\\ndetailed error message:\",\n detail)\n raise(detail)\n \n if self.verbosity > 1:\n print(\"current ml validation score:\",tmp_score)\n\n #################################################### save best model\n if self.valid_loc() and tmp_score > self._best_score:\n self._best_estimator = copy.deepcopy(self.pipeline)\n self._best_score = tmp_score\n stall_count = 0;\n self._best_inds = copy.deepcopy(self.valid())\n if self.verbosity > 1:\n print(\"updated best internal CV:\",self._best_score)\n else:\n stall_count = stall_count + 1\n\n ########################################################## variation\n if self.verbosity > 2:\n print(\"variation...\")\n offspring,elite,elite_index = self.variation(self.pop.individuals)\n\n ################################################# evaluate offspring\n if self.verbosity > 2:\n print(\"output...\")\n X_offspring = self.transform(features,offspring).transpose()\n\n if self.verbosity > 2:\n print(\"fitness...\")\n F_offspring = self.calc_fitness(X_offspring,\n labels,self.fit_choice,self.sel)\n\n ########################################################### survival\n if self.verbosity > 2: print(\"survival..\")\n survivors,survivor_index = self.survival(self.pop.individuals, offspring, \n elite, elite_index,\n X = self.X, X_O=X_offspring,\n F=self.F, F_O=F_offspring)\n # set survivors\n self.pop.individuals[:] = survivors\n self.X = np.vstack((self.X, X_offspring))[survivor_index]\n if 'lexicase' in self.sel:\n self.F = np.asarray(\n np.vstack((self.F, F_offspring))[survivor_index],\n order='F')\n else:\n self.F = np.asarray(\n np.hstack((self.F,F_offspring))[survivor_index],\n order='F')\n\n if self.verbosity > 2:\n print(\"median fitness survivors: %0.2f\" % np.median(\n [np.mean(f) for f in self.F]))\n if self.verbosity>2:\n print(\"best features:\",\n self.stacks_2_eqns(self._best_inds) if self._best_inds\n else 'original')\n pbar.set_description('Internal CV: {:1.3f}'.format(self._best_score))\n pbar.update(1)\n # end of main GP loop\n ####################\n if self.verbosity > 0: print('finished. best internal val score:'\n ' {:1.3f}'.format(self._best_score))\n if self.verbosity > 0: print(\"final model:\\n\",self.print_model())\n\n if not self._best_estimator:\n # if no better model found, just return underlying method fit to the\n # training data\n with warnings.catch_warnings():\n warnings.simplefilter(\"ignore\")\n self._best_estimator = self.pipeline.fit(features,labels)\n else:\n # fit final estimator to all the training data\n with warnings.catch_warnings():\n warnings.simplefilter(\"ignore\")\n self._best_estimator.fit(self.transform(features),labels)\n return self", "response": "Fit the model to data"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a transformation of x using population outputs", "response": "def transform(self,x,inds=None,labels = None):\n \"\"\"return a transformation of x using population outputs\"\"\"\n if inds:\n # return np.asarray(Parallel(n_jobs=10)(delayed(self.out)(I,x,labels,self.otype) \n # for I in inds)).transpose()\n return np.asarray(\n [self.out(I,x,labels,self.otype) for I in inds]).transpose()\n elif self._best_inds:\n # return np.asarray(Parallel(n_jobs=10)(delayed(self.out)(I,x,labels,self.otype) \n # for I in self._best_inds)).transpose()\n return np.asarray(\n [self.out(I,x,labels,self.otype) for I in self._best_inds]).transpose()\n else:\n return x"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nimputes data set containing Nan values", "response": "def impute_data(self,x):\n \"\"\"Imputes data set containing Nan values\"\"\"\n imp = Imputer(missing_values='NaN', strategy='mean', axis=0)\n return imp.fit_transform(x)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nremove nan and inf rows from x", "response": "def clean(self,x):\n \"\"\"remove nan and inf rows from x\"\"\"\n return x[~np.any(np.isnan(x) | np.isinf(x),axis=1)]"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsets nan and inf rows from x to zero", "response": "def clean_with_zeros(self,x):\n \"\"\" set nan and inf rows from x to zero\"\"\"\n x[~np.any(np.isnan(x) | np.isinf(x),axis=1)] = 0\n return x"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\npredict on a holdout data set.", "response": "def predict(self, testing_features):\n \"\"\"predict on a holdout data set.\"\"\"\n # print(\"best_inds:\",self._best_inds)\n # print(\"best estimator size:\",self._best_estimator.coef_.shape)\n if self.clean:\n testing_features = self.impute_data(testing_features)\n\n if self._best_inds:\n X_transform = self.transform(testing_features)\n try:\n return self._best_estimator.predict(self.transform(testing_features))\n except ValueError as detail:\n # pdb.set_trace()\n print('shape of X:',testing_features.shape)\n print('shape of X_transform:',X_transform.transpose().shape)\n print('best inds:',self.stacks_2_eqns(self._best_inds))\n print('valid locs:',self.valid_loc(self._best_inds))\n raise ValueError(detail)\n else:\n return self._best_estimator.predict(testing_features)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nestimates accuracy on testing set", "response": "def score(self, testing_features, testing_labels):\n \"\"\"estimates accuracy on testing set\"\"\"\n # print(\"test features shape:\",testing_features.shape)\n # print(\"testing labels shape:\",testing_labels.shape)\n yhat = self.predict(testing_features)\n return self.scoring_function(testing_labels,yhat)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef export(self, output_file_name):\n if self._best_estimator is None:\n raise ValueError('A model has not been optimized. Please call fit()'\n ' first.')\n\n # Write print_model() to file\n with open(output_file_name, 'w') as output_file:\n output_file.write(self.print_model())\n # if decision tree, print tree into dot file\n if 'DecisionTree' in self.ml_type:\n export_graphviz(self._best_estimator,\n out_file=output_file_name+'.dot',\n feature_names = self.stacks_2_eqns(self._best_inds)\n if self._best_inds else None,\n class_names=['True','False'],\n filled=False,impurity = True,rotate=True)", "response": "Exports engineered features of the current model to a file."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef print_model(self,sep='\\n'):\n model = ''\n # print('ml type:',self.ml_type)\n # print('ml:',self._best_estimator)\n\n if self._best_inds:\n\n if self.ml_type == 'GridSearchCV':\n ml = self._best_estimator.named_steps['ml'].best_estimator_\n else:\n ml = self._best_estimator.named_steps['ml']\n\n if self.ml_type != 'SVC' and self.ml_type != 'SVR':\n # this is need because svm has a bug that throws valueerror on\n # attribute check\n \n if hasattr(ml,'coef_'):\n if len(ml.coef_.shape)==1:\n s = np.argsort(np.abs(ml.coef_))[::-1]\n scoef = ml.coef_[s]\n bi = [self._best_inds[k] for k in s]\n model = (' +' + sep).join(\n [str(round(c,3))+'*'+self.stack_2_eqn(f)\n for i,(f,c) in enumerate(zip(bi,scoef))\n if round(scoef[i],3) != 0])\n else:\n # more than one decision function is fit. print all.\n for j,coef in enumerate(ml.coef_):\n s = np.argsort(np.abs(coef))[::-1]\n scoef = coef[s]\n bi =[self._best_inds[k] for k in s]\n model += sep + 'class'+str(j)+' :'+' + '.join(\n [str(round(c,3))+'*'+self.stack_2_eqn(f)\n for i,(f,c) in enumerate(zip(bi,coef))\n if coef[i] != 0])\n elif hasattr(ml,'feature_importances_'):\n s = np.argsort(ml.feature_importances_)[::-1]\n sfi = ml.feature_importances_[s]\n bi = [self._best_inds[k] for k in s]\n # model = 'importance:feature'+sep\n\n model += sep.join(\n [str(round(c,3))+':'+self.stack_2_eqn(f)\n for i,(f,c) in enumerate(zip(bi,sfi))\n if round(sfi[i],3) != 0])\n else:\n return sep.join(self.stacks_2_eqns(self._best_inds))\n else:\n return sep.join(self.stacks_2_eqns(self._best_inds))\n else:\n return 'original features'\n\n return model", "response": "prints model contained in best inds if ml has a coefficient property. otherwise prints the features generated by FEW."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef valid_loc(self,F=None):\n if F is not None:\n return [i for i,f in enumerate(F) if np.all(f < self.max_fit) and np.all(f >= 0)]\n else:\n return [i for i,f in enumerate(self.F) if np.all(f < self.max_fit) and np.all(f >= 0)]", "response": "returns the indices of individuals with valid fitness."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the sublist of individuals with valid fitness.", "response": "def valid(self,individuals=None,F=None):\n \"\"\"returns the sublist of individuals with valid fitness.\"\"\"\n if F:\n valid_locs = self.valid_loc(F)\n else:\n valid_locs = self.valid_loc(self.F)\n\n if individuals:\n return [ind for i,ind in enumerate(individuals) if i in valid_locs]\n else:\n return [ind for i,ind in enumerate(self.pop.individuals) if i in valid_locs]"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncompute mean diversity of individual outputs", "response": "def get_diversity(self,X):\n \"\"\"compute mean diversity of individual outputs\"\"\"\n # diversity in terms of cosine distances between features\n feature_correlations = np.zeros(X.shape[0]-1)\n for i in np.arange(1,X.shape[0]-1):\n feature_correlations[i] = max(0.0,r2_score(X[0],X[i]))\n # pdb.set_trace()\n self.diversity.append(1-np.mean(feature_correlations))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn an roc auc score depending on the underlying estimator.", "response": "def roc_auc_cv(self,features,labels):\n \"\"\"returns an roc auc score depending on the underlying estimator.\"\"\" \n if callable(getattr(self.ml, \"decision_function\", None)):\n return np.mean([self.scoring_function(labels[test],\n self.pipeline.fit(features[train],labels[train]).\n decision_function(features[test]))\n for train, test in KFold().split(features, labels)])\n elif callable(getattr(self.ml, \"predict_proba\", None)):\n return np.mean([self.scoring_function(labels[test],\n self.pipeline.fit(features[train],labels[train]).\n predict_proba(features[test])[:,1])\n for train, test in KFold().split(features, labels)]) \n else:\n raise ValueError(\"ROC AUC score won't work with \" + self.ml_type + \". No \"\n \"decision_function or predict_proba method found for this learner.\")"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef divs(x,y):\n tmp = np.ones(x.shape)\n nonzero_y = y != 0\n tmp[nonzero_y] = x[nonzero_y]/y[nonzero_y]\n return tmp", "response": "safe division of x and y"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn non - aggregate version of r2 score.", "response": "def r2_score_vec(y_true,y_pred):\n \"\"\" returns non-aggregate version of r2 score.\n\n based on r2_score() function from sklearn (http://sklearn.org)\n \"\"\"\n\n numerator = (y_true - y_pred) ** 2\n denominator = (y_true - np.average(y_true)) ** 2\n\n nonzero_denominator = denominator != 0\n nonzero_numerator = numerator != 0\n valid_score = nonzero_denominator & nonzero_numerator\n output_scores = np.ones([y_true.shape[0]])\n output_scores[valid_score] = 1 - (numerator[valid_score] /\n denominator[valid_score])\n # arbitrary set to zero to avoid -inf scores, having a constant\n # y_true is not interesting for scoring a regression anyway\n output_scores[nonzero_numerator & ~nonzero_denominator] = 0.\n\n return output_scores"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncalculates the within - class squared distance from the centroid", "response": "def inertia(X,y,samples=False):\n \"\"\" return the within-class squared distance from the centroid\"\"\"\n # pdb.set_trace()\n if samples:\n # return within-class distance for each sample\n inertia = np.zeros(y.shape)\n for label in np.unique(y):\n inertia[y==label] = (X[y==label] - np.mean(X[y==label])) ** 2\n\n else: # return aggregate score\n inertia = 0\n for i,label in enumerate(np.unique(y)):\n inertia += np.sum((X[y==label] - np.mean(X[y==label])) ** 2)/len(y[y==label])\n inertia = inertia/len(np.unique(y))\n\n return inertia"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the sum of the between - class squared distance", "response": "def separation(X,y,samples=False):\n \"\"\" return the sum of the between-class squared distance\"\"\"\n # pdb.set_trace()\n num_classes = len(np.unique(y))\n total_dist = (X.max()-X.min())**2\n if samples:\n # return intra-class distance for each sample\n separation = np.zeros(y.shape)\n for label in np.unique(y):\n for outsider in np.unique(y[y!=label]):\n separation[y==label] += (X[y==label] - np.mean(X[y==outsider])) ** 2\n\n #normalize between 0 and 1\n print('separation:',separation)\n print('num_classes:',num_classes)\n print('total_dist:',total_dist)\n separation = separation#/separation.max()\n\n print('separation after normalization:',separation)\n\n else:\n # return aggregate score\n separation = 0\n for i,label in enumerate(np.unique(y)):\n for outsider in np.unique(y[y!=label]):\n separation += np.sum((X[y==label] - np.mean(X[y==outsider])) ** 2)/len(y[y==label])\n separation = separation/len(np.unique(y))\n\n return separation"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef fisher(yhat,y,samples=False):\n classes = np.unique(y)\n mu = np.zeros(len(classes))\n v = np.zeros(len(classes))\n # pdb.set_trace()\n for c in classes.astype(int):\n mu[c] = np.mean(yhat[y==c])\n v[c] = np.var(yhat[y==c])\n\n if not samples:\n fisher = 0\n for c1,c2 in pairwise(classes.astype(int)):\n fisher += np.abs(mu[c1] - mu[c2])/np.sqrt(v[c1]+v[c2])\n else:\n # lexicase version\n fisher = np.zeros(len(yhat))\n # get closests classes to each class (min mu distance)\n mu_d = pairwise_distances(mu.reshape(-1,1))\n min_mu=np.zeros(len(classes),dtype=int)\n for i in np.arange(len(min_mu)):\n min_mu[i] = np.argsort(mu_d[i])[1]\n # for c1, pairwise(classes.astype(int)):\n # min_mu[c1] = np.argmin()\n for i,l in enumerate(yhat.astype(int)):\n fisher[i] = np.abs(l - mu[min_mu[y[i]]])/np.sqrt(v[y[i]]+v[min_mu[y[i]]])\n\n # pdb.set_trace()\n return fisher", "response": "Fisher criterion for the given set of classes y."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef proper(self,x):\n x[x < 0] = self.max_fit\n x[np.isnan(x)] = self.max_fit\n x[np.isinf(x)] = self.max_fit\n return x", "response": "cleans fitness vector x"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef safe(self,x):\n x[np.isinf(x)] = 1\n x[np.isnan(x)] = 1\n return x", "response": "removes nans and infs from outputs"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef evaluate(self,n, features, stack_float, stack_bool,labels=None):\n np.seterr(all='ignore')\n if len(stack_float) >= n.arity['f'] and len(stack_bool) >= n.arity['b']:\n if n.out_type == 'f':\n stack_float.append(\n self.safe(self.eval_dict[n.name](n,features,stack_float,\n stack_bool,labels)))\n if (np.isnan(stack_float[-1]).any() or\n np.isinf(stack_float[-1]).any()):\n print(\"problem operator:\",n)\n else:\n stack_bool.append(self.safe(self.eval_dict[n.name](n,features,\n stack_float,\n stack_bool,\n labels)))\n if np.isnan(stack_bool[-1]).any() or np.isinf(stack_bool[-1]).any():\n print(\"problem operator:\",n)", "response": "evaluate node in program"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef all_finite(self,X):\n # Adapted from sklearn utils: _assert_all_finite(X)\n # First try an O(n) time, O(1) space solution for the common case that\n # everything is finite; fall back to O(n) space np.isfinite to prevent\n # false positives from overflow in sum method.\n # Note: this is basically here because sklearn tree.py uses float32 internally,\n # and float64's that are finite are not finite in float32.\n if (X.dtype.char in np.typecodes['AllFloat']\n and not np.isfinite(np.asarray(X,dtype='float32').sum())\n and not np.isfinite(np.asarray(X,dtype='float32')).all()):\n return False\n return True", "response": "returns true if X is finite false otherwise"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncompute the output for individual I", "response": "def out(self,I,features,labels=None,otype='f'):\n \"\"\"computes the output for individual I\"\"\"\n stack_float = []\n stack_bool = []\n # print(\"stack:\",I.stack)\n # evaulate stack over rows of features,labels\n # pdb.set_trace()\n for n in I.stack:\n self.evaluate(n,features,stack_float,stack_bool,labels)\n # print(\"stack_float:\",stack_float)\n if otype=='f':\n return (stack_float[-1] if self.all_finite(stack_float[-1])\n else np.zeros(len(features)))\n else:\n return (stack_bool[-1].astype(float) if self.all_finite(stack_bool[-1])\n else np.zeros(len(features)))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef calc_fitness(self,X,labels,fit_choice,sel):\n\n if 'lexicase' in sel:\n # return list(map(lambda yhat: self.f_vec[fit_choice](labels,yhat),X))\n return np.asarray(\n [self.proper(self.f_vec[fit_choice](labels,\n yhat)) for yhat in X],\n order='F')\n # return list(Parallel(n_jobs=-1)(delayed(self.f_vec[fit_choice])(labels,yhat) for yhat in X))\n else:\n # return list(map(lambda yhat: self.f[fit_choice](labels,yhat),X))\n return np.asarray([self.f[fit_choice](labels,yhat) for yhat in X],\n order='F').reshape(-1)", "response": "computes fitness of individual output yhat."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert the ImageData dictionary imdict to an nd image.", "response": "def imagedatadict_to_ndarray(imdict):\n \"\"\"\n Converts the ImageData dictionary, imdict, to an nd image.\n \"\"\"\n arr = imdict['Data']\n im = None\n if isinstance(arr, parse_dm3.array.array):\n im = numpy.asarray(arr, dtype=arr.typecode)\n elif isinstance(arr, parse_dm3.structarray):\n t = tuple(arr.typecodes)\n im = numpy.frombuffer(\n arr.raw_data,\n dtype=structarray_to_np_map[t])\n # print \"Image has dmimagetype\", imdict[\"DataType\"], \"numpy type is\", im.dtype\n assert dm_image_dtypes[imdict[\"DataType\"]][1] == im.dtype\n assert imdict['PixelDepth'] == im.dtype.itemsize\n im = im.reshape(imdict['Dimensions'][::-1])\n if imdict[\"DataType\"] == 23: # RGB\n im = im.view(numpy.uint8).reshape(im.shape + (-1, ))[..., :-1] # strip A\n # NOTE: RGB -> BGR would be [:, :, ::-1]\n return im"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting the numpy array nparr into a suitable ImageList entry dictionary.", "response": "def ndarray_to_imagedatadict(nparr):\n \"\"\"\n Convert the numpy array nparr into a suitable ImageList entry dictionary.\n Returns a dictionary with the appropriate Data, DataType, PixelDepth\n to be inserted into a dm3 tag dictionary and written to a file.\n \"\"\"\n ret = {}\n dm_type = None\n for k, v in iter(dm_image_dtypes.items()):\n if v[1] == nparr.dtype.type:\n dm_type = k\n break\n if dm_type is None and nparr.dtype == numpy.uint8 and nparr.shape[-1] in (3, 4):\n ret[\"DataType\"] = 23\n ret[\"PixelDepth\"] = 4\n if nparr.shape[2] == 4:\n rgb_view = nparr.view(numpy.int32).reshape(nparr.shape[:-1]) # squash the color into uint32\n else:\n assert nparr.shape[2] == 3\n rgba_image = numpy.empty(nparr.shape[:-1] + (4,), numpy.uint8)\n rgba_image[:,:,0:3] = nparr\n rgba_image[:,:,3] = 255\n rgb_view = rgba_image.view(numpy.int32).reshape(rgba_image.shape[:-1]) # squash the color into uint32\n ret[\"Dimensions\"] = list(rgb_view.shape[::-1])\n ret[\"Data\"] = parse_dm3.array.array(platform_independent_char(rgb_view.dtype), rgb_view.flatten())\n else:\n ret[\"DataType\"] = dm_type\n ret[\"PixelDepth\"] = nparr.dtype.itemsize\n ret[\"Dimensions\"] = list(nparr.shape[::-1])\n if nparr.dtype.type in np_to_structarray_map:\n types = np_to_structarray_map[nparr.dtype.type]\n ret[\"Data\"] = parse_dm3.structarray(types)\n ret[\"Data\"].raw_data = bytes(numpy.array(nparr, copy=False).data)\n else:\n ret[\"Data\"] = parse_dm3.array.array(platform_independent_char(nparr.dtype), numpy.array(nparr, copy=False).flatten())\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef load_image(file) -> DataAndMetadata.DataAndMetadata:\n if isinstance(file, str) or isinstance(file, str):\n with open(file, \"rb\") as f:\n return load_image(f)\n dmtag = parse_dm3.parse_dm_header(file)\n dmtag = fix_strings(dmtag)\n # display_keys(dmtag)\n img_index = -1\n image_tags = dmtag['ImageList'][img_index]\n data = imagedatadict_to_ndarray(image_tags['ImageData'])\n calibrations = []\n calibration_tags = image_tags['ImageData'].get('Calibrations', dict())\n for dimension in calibration_tags.get('Dimension', list()):\n origin, scale, units = dimension.get('Origin', 0.0), dimension.get('Scale', 1.0), dimension.get('Units', str())\n calibrations.append((-origin * scale, scale, units))\n calibrations = tuple(reversed(calibrations))\n if len(data.shape) == 3 and data.dtype != numpy.uint8:\n if image_tags['ImageTags'].get('Meta Data', dict()).get(\"Format\", str()).lower() in (\"spectrum\", \"spectrum image\"):\n if data.shape[1] == 1:\n data = numpy.squeeze(data, 1)\n data = numpy.moveaxis(data, 0, 1)\n data_descriptor = DataAndMetadata.DataDescriptor(False, 1, 1)\n calibrations = (calibrations[2], calibrations[0])\n else:\n data = numpy.moveaxis(data, 0, 2)\n data_descriptor = DataAndMetadata.DataDescriptor(False, 2, 1)\n calibrations = tuple(calibrations[1:]) + (calibrations[0],)\n else:\n data_descriptor = DataAndMetadata.DataDescriptor(False, 1, 2)\n elif len(data.shape) == 4 and data.dtype != numpy.uint8:\n # data = numpy.moveaxis(data, 0, 2)\n data_descriptor = DataAndMetadata.DataDescriptor(False, 2, 2)\n elif data.dtype == numpy.uint8:\n data_descriptor = DataAndMetadata.DataDescriptor(False, 0, len(data.shape[:-1]))\n else:\n data_descriptor = DataAndMetadata.DataDescriptor(False, 0, len(data.shape))\n brightness = calibration_tags.get('Brightness', dict())\n origin, scale, units = brightness.get('Origin', 0.0), brightness.get('Scale', 1.0), brightness.get('Units', str())\n intensity = -origin * scale, scale, units\n timestamp = None\n timezone = None\n timezone_offset = None\n title = image_tags.get('Name')\n properties = dict()\n if 'ImageTags' in image_tags:\n voltage = image_tags['ImageTags'].get('ImageScanned', dict()).get('EHT', dict())\n if voltage:\n properties.setdefault(\"hardware_source\", dict())[\"autostem\"] = { \"high_tension_v\": float(voltage) }\n dm_metadata_signal = image_tags['ImageTags'].get('Meta Data', dict()).get('Signal')\n if dm_metadata_signal and dm_metadata_signal.lower() == \"eels\":\n properties.setdefault(\"hardware_source\", dict())[\"signal_type\"] = dm_metadata_signal\n if image_tags['ImageTags'].get('Meta Data', dict()).get(\"Format\", str()).lower() in (\"spectrum\", \"spectrum image\"):\n data_descriptor.collection_dimension_count += data_descriptor.datum_dimension_count - 1\n data_descriptor.datum_dimension_count = 1\n if image_tags['ImageTags'].get('Meta Data', dict()).get(\"IsSequence\", False) and data_descriptor.collection_dimension_count > 0:\n data_descriptor.is_sequence = True\n data_descriptor.collection_dimension_count -= 1\n timestamp_str = image_tags['ImageTags'].get(\"Timestamp\")\n if timestamp_str:\n timestamp = get_datetime_from_timestamp_str(timestamp_str)\n timezone = image_tags['ImageTags'].get(\"Timezone\")\n timezone_offset = image_tags['ImageTags'].get(\"TimezoneOffset\")\n # to avoid having duplicate copies in Swift, get rid of these tags\n image_tags['ImageTags'].pop(\"Timestamp\", None)\n image_tags['ImageTags'].pop(\"Timezone\", None)\n image_tags['ImageTags'].pop(\"TimezoneOffset\", None)\n # put the image tags into properties\n properties.update(image_tags['ImageTags'])\n dimensional_calibrations = [Calibration.Calibration(c[0], c[1], c[2]) for c in calibrations]\n while len(dimensional_calibrations) < data_descriptor.expected_dimension_count:\n dimensional_calibrations.append(Calibration.Calibration())\n intensity_calibration = Calibration.Calibration(intensity[0], intensity[1], intensity[2])\n return DataAndMetadata.new_data_and_metadata(data,\n data_descriptor=data_descriptor,\n dimensional_calibrations=dimensional_calibrations,\n intensity_calibration=intensity_calibration,\n metadata=properties,\n timestamp=timestamp,\n timezone=timezone,\n timezone_offset=timezone_offset)", "response": "Loads the image from the file - like object or string file."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef save_image(xdata: DataAndMetadata.DataAndMetadata, file):\n # we need to create a basic DM tree suitable for an image\n # we'll try the minimum: just an data list\n # doesn't work. Do we need a ImageSourceList too?\n # and a DocumentObjectList?\n\n data = xdata.data\n data_descriptor = xdata.data_descriptor\n dimensional_calibrations = xdata.dimensional_calibrations\n intensity_calibration = xdata.intensity_calibration\n metadata = xdata.metadata\n modified = xdata.timestamp\n timezone = xdata.timezone\n timezone_offset = xdata.timezone_offset\n needs_slice = False\n is_sequence = False\n\n if len(data.shape) == 3 and data.dtype != numpy.uint8 and data_descriptor.datum_dimension_count == 1:\n data = numpy.moveaxis(data, 2, 0)\n dimensional_calibrations = (dimensional_calibrations[2],) + tuple(dimensional_calibrations[0:2])\n if len(data.shape) == 2 and data.dtype != numpy.uint8 and data_descriptor.datum_dimension_count == 1:\n is_sequence = data_descriptor.is_sequence\n data = numpy.moveaxis(data, 1, 0)\n data = numpy.expand_dims(data, axis=1)\n dimensional_calibrations = (dimensional_calibrations[1], Calibration.Calibration(), dimensional_calibrations[0])\n data_descriptor = DataAndMetadata.DataDescriptor(False, 2, 1)\n needs_slice = True\n data_dict = ndarray_to_imagedatadict(data)\n ret = {}\n ret[\"ImageList\"] = [{\"ImageData\": data_dict}]\n if dimensional_calibrations and len(dimensional_calibrations) == len(data.shape):\n dimension_list = data_dict.setdefault(\"Calibrations\", dict()).setdefault(\"Dimension\", list())\n for dimensional_calibration in reversed(dimensional_calibrations):\n dimension = dict()\n if dimensional_calibration.scale != 0.0:\n origin = -dimensional_calibration.offset / dimensional_calibration.scale\n else:\n origin = 0.0\n dimension['Origin'] = origin\n dimension['Scale'] = dimensional_calibration.scale\n dimension['Units'] = dimensional_calibration.units\n dimension_list.append(dimension)\n if intensity_calibration:\n if intensity_calibration.scale != 0.0:\n origin = -intensity_calibration.offset / intensity_calibration.scale\n else:\n origin = 0.0\n brightness = data_dict.setdefault(\"Calibrations\", dict()).setdefault(\"Brightness\", dict())\n brightness['Origin'] = origin\n brightness['Scale'] = intensity_calibration.scale\n brightness['Units'] = intensity_calibration.units\n if modified:\n timezone_str = None\n if timezone_str is None and timezone:\n try:\n import pytz\n tz = pytz.timezone(timezone)\n timezone_str = tz.tzname(modified)\n except:\n pass\n if timezone_str is None and timezone_offset:\n timezone_str = timezone_offset\n timezone_str = \" \" + timezone_str if timezone_str is not None else \"\"\n date_str = modified.strftime(\"%x\")\n time_str = modified.strftime(\"%X\") + timezone_str\n ret[\"DataBar\"] = {\"Acquisition Date\": date_str, \"Acquisition Time\": time_str}\n # I think ImageSource list creates a mapping between ImageSourceIds and Images\n ret[\"ImageSourceList\"] = [{\"ClassName\": \"ImageSource:Simple\", \"Id\": [0], \"ImageRef\": 0}]\n # I think this lists the sources for the DocumentObjectlist. The source number is not\n # the indxe in the imagelist but is either the index in the ImageSourceList or the Id\n # from that list. We also need to set the annotation type to identify it as an data\n ret[\"DocumentObjectList\"] = [{\"ImageSource\": 0, \"AnnotationType\": 20}]\n # finally some display options\n ret[\"Image Behavior\"] = {\"ViewDisplayID\": 8}\n dm_metadata = copy.deepcopy(metadata)\n if metadata.get(\"hardware_source\", dict()).get(\"signal_type\", \"\").lower() == \"eels\":\n if len(data.shape) == 1 or (len(data.shape) == 2 and data.shape[0] == 1):\n dm_metadata.setdefault(\"Meta Data\", dict())[\"Format\"] = \"Spectrum\"\n dm_metadata.setdefault(\"Meta Data\", dict())[\"Signal\"] = \"EELS\"\n elif data_descriptor.collection_dimension_count == 2 and data_descriptor.datum_dimension_count == 1:\n dm_metadata.setdefault(\"Meta Data\", dict())[\"Format\"] = \"Spectrum image\"\n dm_metadata.setdefault(\"Meta Data\", dict())[\"Signal\"] = \"EELS\"\n elif data_descriptor.datum_dimension_count == 1:\n dm_metadata.setdefault(\"Meta Data\", dict())[\"Format\"] = \"Spectrum\"\n if (1 if data_descriptor.is_sequence else 0) + data_descriptor.collection_dimension_count == 1 or needs_slice:\n if data_descriptor.is_sequence or is_sequence:\n dm_metadata.setdefault(\"Meta Data\", dict())[\"IsSequence\"] = True\n ret[\"ImageSourceList\"] = [{\"ClassName\": \"ImageSource:Summed\", \"Do Sum\": True, \"Id\": [0], \"ImageRef\": 0, \"LayerEnd\": 0, \"LayerStart\": 0, \"Summed Dimension\": len(data.shape) - 1}]\n if needs_slice:\n ret[\"DocumentObjectList\"][0][\"AnnotationGroupList\"] = [{\"AnnotationType\": 23, \"Name\": \"SICursor\", \"Rectangle\": (0, 0, 1, 1)}]\n ret[\"DocumentObjectList\"][0][\"ImageDisplayType\"] = 1 # display as an image\n if modified:\n dm_metadata[\"Timestamp\"] = modified.isoformat()\n if timezone:\n dm_metadata[\"Timezone\"] = timezone\n if timezone_offset:\n dm_metadata[\"TimezoneOffset\"] = timezone_offset\n ret[\"ImageList\"][0][\"ImageTags\"] = dm_metadata\n ret[\"InImageMode\"] = True\n parse_dm3.parse_dm_header(file, ret)", "response": "Save the image data to a nparray file - like object or string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nparsing the DM file header and return a new file containing the tag_root entry.", "response": "def parse_dm_header(f, outdata=None):\n \"\"\"\n This is the start of the DM file. We check for some\n magic values and then treat the next entry as a tag_root\n\n If outdata is supplied, we write instead of read using the dictionary outdata as a source\n Hopefully parse_dm_header(newf, outdata=parse_dm_header(f)) copies f to newf\n \"\"\"\n # filesize is sizeondisk - 16. But we have 8 bytes of zero at the end of\n # the file.\n if outdata is not None: # this means we're WRITING to the file\n if verbose:\n print(\"write_dm_header start\", f.tell())\n ver, file_size, endianness = 3, -1, 1\n put_into_file(f, \"> l l l\", ver, file_size, endianness)\n start = f.tell()\n parse_dm_tag_root(f, outdata)\n end = f.tell()\n # start is end of 3 long header. We want to write 2nd long\n f.seek(start - 8)\n # the real file size. We started counting after 12-byte version,fs,end\n # and we need to subtract 16 total:\n put_into_file(f, \"> l\", end - start + 4)\n f.seek(end)\n enda, endb = 0, 0\n put_into_file(f, \"> l l\", enda, endb)\n if verbose:\n print(\"write_dm_header end\", f.tell())\n else:\n if verbose:\n print(\"read_dm_header start\", f.tell())\n ver = get_from_file(f, \"> l\")\n assert ver in [3,4], \"Version must be 3 or 4, not %s\" % ver\n # argh. why a global?\n global size_type, version\n if ver == 3:\n size_type = 'L' # may be Q?\n version = 3\n if ver == 4:\n size_type = 'Q' # may be Q?\n version = 4\n file_size, endianness = get_from_file(f, \">%c l\" % size_type)\n assert endianness == 1, \"Endianness must be 1, not %s\"%endianness\n start = f.tell()\n ret = parse_dm_tag_root(f, outdata)\n end = f.tell()\n # print(\"fs\", file_size, end - start, (end-start)%8)\n # mfm 2013-07-11 the file_size value is not always\n # end-start, sometimes there seems to be an extra 4 bytes,\n # other times not. Let's just ignore it for the moment\n # assert(file_size == end - start)\n enda, endb = get_from_file(f, \"> l l\")\n assert(enda == endb == 0)\n if verbose:\n print(\"read_dm_header end\", f.tell())\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the dmtype for the python typeorobject typeorobj.", "response": "def get_structdmtypes_for_python_typeorobject(typeorobj):\n \"\"\"\n Return structchar, dmtype for the python (or numpy)\n type or object typeorobj.\n For more complex types we only return the dm type\n \"\"\"\n # not isinstance is probably a bit more lenient than 'is'\n # ie isinstance(x,str) is nicer than type(x) is str.\n # hence we use isinstance when available\n if isinstance(typeorobj, type):\n comparer = lambda test: test is typeorobj\n else:\n comparer = lambda test: isinstance(typeorobj, test)\n\n if comparer(int) and not -2**31 < typeorobj < 2**31 - 1:\n return 'q', 11\n\n for key, name, sc, types in dm_simple_names:\n for t in types:\n if comparer(t):\n return sc, key\n if comparer(str):\n return None, get_dmtype_for_name('array') # treat all strings as arrays!\n elif comparer(unicode_type):\n return None, get_dmtype_for_name('array') # treat all strings as arrays!\n elif comparer(array.array):\n return None, get_dmtype_for_name('array')\n elif comparer(tuple):\n return None, get_dmtype_for_name('struct')\n elif comparer(structarray):\n return None, get_dmtype_for_name('array')\n logging.warn(\"No appropriate DMType found for %s, %s\", typeorobj, type(typeorobj))\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef imread(files, **kwargs):\n kwargs_file = parse_kwargs(kwargs, 'is_ome', 'multifile', '_useframes',\n 'name', 'offset', 'size',\n 'multifile_close', 'fastij', 'movie') # legacy\n kwargs_seq = parse_kwargs(kwargs, 'pattern')\n\n if kwargs.get('pages', None) is not None:\n if kwargs.get('key', None) is not None:\n raise TypeError(\n \"the 'pages' and 'key' arguments cannot be used together\")\n log.warning(\"imread: the 'pages' argument is deprecated\")\n kwargs['key'] = kwargs.pop('pages')\n\n if isinstance(files, basestring) and any(i in files for i in '?*'):\n files = glob.glob(files)\n if not files:\n raise ValueError('no files found')\n if not hasattr(files, 'seek') and len(files) == 1:\n files = files[0]\n\n if isinstance(files, basestring) or hasattr(files, 'seek'):\n with TiffFile(files, **kwargs_file) as tif:\n return tif.asarray(**kwargs)\n else:\n with TiffSequence(files, **kwargs_seq) as imseq:\n return imseq.asarray(**kwargs)", "response": "Read image data from TIFF files and return as numpy array."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef imwrite(file, data=None, shape=None, dtype=None, **kwargs):\n tifargs = parse_kwargs(kwargs, 'append', 'bigtiff', 'byteorder', 'imagej')\n if data is None:\n dtype = numpy.dtype(dtype)\n size = product(shape) * dtype.itemsize\n byteorder = dtype.byteorder\n else:\n try:\n size = data.nbytes\n byteorder = data.dtype.byteorder\n except Exception:\n size = 0\n byteorder = None\n bigsize = kwargs.pop('bigsize', 2**32-2**25)\n if 'bigtiff' not in tifargs and size > bigsize and not (\n tifargs.get('imagej', False) or tifargs.get('truncate', False)):\n tifargs['bigtiff'] = True\n if 'byteorder' not in tifargs:\n tifargs['byteorder'] = byteorder\n\n with TiffWriter(file, **tifargs) as tif:\n return tif.save(data, shape, dtype, **kwargs)", "response": "Write numpy array to TIFF file."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef memmap(filename, shape=None, dtype=None, page=None, series=0, mode='r+',\n **kwargs):\n \"\"\"Return memory-mapped numpy array stored in TIFF file.\n\n Memory-mapping requires data stored in native byte order, without tiling,\n compression, predictors, etc.\n If 'shape' and 'dtype' are provided, existing files will be overwritten or\n appended to depending on the 'append' parameter.\n Otherwise the image data of a specified page or series in an existing\n file will be memory-mapped. By default, the image data of the first page\n series is memory-mapped.\n Call flush() to write any changes in the array to the file.\n Raise ValueError if the image data in the file is not memory-mappable.\n\n Parameters\n ----------\n filename : str\n Name of the TIFF file which stores the array.\n shape : tuple\n Shape of the empty array.\n dtype : numpy.dtype\n Data-type of the empty array.\n page : int\n Index of the page which image data to memory-map.\n series : int\n Index of the page series which image data to memory-map.\n mode : {'r+', 'r', 'c'}\n The file open mode. Default is to open existing file for reading and\n writing ('r+').\n kwargs : dict\n Additional parameters passed to imwrite() or TiffFile().\n\n \"\"\"\n if shape is not None and dtype is not None:\n # create a new, empty array\n kwargs.update(data=None, shape=shape, dtype=dtype, returnoffset=True,\n align=TIFF.ALLOCATIONGRANULARITY)\n result = imwrite(filename, **kwargs)\n if result is None:\n # TODO: fail before creating file or writing data\n raise ValueError('image data are not memory-mappable')\n offset = result[0]\n else:\n # use existing file\n with TiffFile(filename, **kwargs) as tif:\n if page is not None:\n page = tif.pages[page]\n if not page.is_memmappable:\n raise ValueError('image data are not memory-mappable')\n offset, _ = page.is_contiguous\n shape = page.shape\n dtype = page.dtype\n else:\n series = tif.series[series]\n if series.offset is None:\n raise ValueError('image data are not memory-mappable')\n shape = series.shape\n dtype = series.dtype\n offset = series.offset\n dtype = tif.byteorder + dtype.char\n return numpy.memmap(filename, dtype, mode, offset, shape, 'C')", "response": "Return memory - mapped numpy array stored in TIFF file."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreads tags from the chain of IFDs and return as list of dicts.", "response": "def read_tags(fh, byteorder, offsetsize, tagnames, customtags=None,\n maxifds=None):\n \"\"\"Read tags from chain of IFDs and return as list of dicts.\n\n The file handle position must be at a valid IFD header.\n\n \"\"\"\n if offsetsize == 4:\n offsetformat = byteorder+'I'\n tagnosize = 2\n tagnoformat = byteorder+'H'\n tagsize = 12\n tagformat1 = byteorder+'HH'\n tagformat2 = byteorder+'I4s'\n elif offsetsize == 8:\n offsetformat = byteorder+'Q'\n tagnosize = 8\n tagnoformat = byteorder+'Q'\n tagsize = 20\n tagformat1 = byteorder+'HH'\n tagformat2 = byteorder+'Q8s'\n else:\n raise ValueError('invalid offset size')\n\n if customtags is None:\n customtags = {}\n if maxifds is None:\n maxifds = 2**32\n\n result = []\n unpack = struct.unpack\n offset = fh.tell()\n while len(result) < maxifds:\n # loop over IFDs\n try:\n tagno = unpack(tagnoformat, fh.read(tagnosize))[0]\n if tagno > 4096:\n raise TiffFileError('suspicious number of tags')\n except Exception:\n log.warning('read_tags: corrupted tag list at offset %i', offset)\n break\n\n tags = {}\n data = fh.read(tagsize * tagno)\n pos = fh.tell()\n index = 0\n for _ in range(tagno):\n code, type_ = unpack(tagformat1, data[index:index+4])\n count, value = unpack(tagformat2, data[index+4:index+tagsize])\n index += tagsize\n name = tagnames.get(code, str(code))\n try:\n dtype = TIFF.DATA_FORMATS[type_]\n except KeyError:\n raise TiffFileError('unknown tag data type %i' % type_)\n\n fmt = '%s%i%s' % (byteorder, count * int(dtype[0]), dtype[1])\n size = struct.calcsize(fmt)\n if size > offsetsize or code in customtags:\n offset = unpack(offsetformat, value)[0]\n if offset < 8 or offset > fh.size - size:\n raise TiffFileError('invalid tag value offset %i' % offset)\n fh.seek(offset)\n if code in customtags:\n readfunc = customtags[code][1]\n value = readfunc(fh, byteorder, dtype, count, offsetsize)\n elif type_ == 7 or (count > 1 and dtype[-1] == 'B'):\n value = read_bytes(fh, byteorder, dtype, count, offsetsize)\n elif code in tagnames or dtype[-1] == 's':\n value = unpack(fmt, fh.read(size))\n else:\n value = read_numpy(fh, byteorder, dtype, count, offsetsize)\n elif dtype[-1] == 'B' or type_ == 7:\n value = value[:size]\n else:\n value = unpack(fmt, value[:size])\n\n if code not in customtags and code not in TIFF.TAG_TUPLE:\n if len(value) == 1:\n value = value[0]\n if type_ != 7 and dtype[-1] == 's' and isinstance(value, bytes):\n # TIFF ASCII fields can contain multiple strings,\n # each terminated with a NUL\n try:\n value = bytes2str(stripascii(value).strip())\n except UnicodeDecodeError:\n log.warning(\n 'read_tags: coercing invalid ASCII to bytes (tag %i)',\n code)\n\n tags[name] = value\n\n result.append(tags)\n # read offset to next page\n fh.seek(pos)\n offset = unpack(offsetformat, fh.read(offsetsize))[0]\n if offset == 0:\n break\n if offset >= fh.size:\n log.warning('read_tags: invalid page offset (%i)', offset)\n break\n fh.seek(offset)\n\n if result and maxifds == 1:\n result = result[0]\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nread EXIF tags from file and return as dict.", "response": "def read_exif_ifd(fh, byteorder, dtype, count, offsetsize):\n \"\"\"Read EXIF tags from file and return as dict.\"\"\"\n exif = read_tags(fh, byteorder, offsetsize, TIFF.EXIF_TAGS, maxifds=1)\n for name in ('ExifVersion', 'FlashpixVersion'):\n try:\n exif[name] = bytes2str(exif[name])\n except Exception:\n pass\n if 'UserComment' in exif:\n idcode = exif['UserComment'][:8]\n try:\n if idcode == b'ASCII\\x00\\x00\\x00':\n exif['UserComment'] = bytes2str(exif['UserComment'][8:])\n elif idcode == b'UNICODE\\x00':\n exif['UserComment'] = exif['UserComment'][8:].decode('utf-16')\n except Exception:\n pass\n return exif"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef read_gps_ifd(fh, byteorder, dtype, count, offsetsize):\n return read_tags(fh, byteorder, offsetsize, TIFF.GPS_TAGS, maxifds=1)", "response": "Read GPS tags from file and return as dict."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreading Interoperability tags from file and return as dict.", "response": "def read_interoperability_ifd(fh, byteorder, dtype, count, offsetsize):\n \"\"\"Read Interoperability tags from file and return as dict.\"\"\"\n tag_names = {1: 'InteroperabilityIndex'}\n return read_tags(fh, byteorder, offsetsize, tag_names, maxifds=1)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreads tag data from file and return as byte string.", "response": "def read_bytes(fh, byteorder, dtype, count, offsetsize):\n \"\"\"Read tag data from file and return as byte string.\"\"\"\n dtype = 'B' if dtype[-1] == 's' else byteorder+dtype[-1]\n count *= numpy.dtype(dtype).itemsize\n data = fh.read(count)\n if len(data) != count:\n log.warning('read_bytes: failed to read all bytes (%i < %i)',\n len(data), count)\n return data"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nread tag data from file and return as unicode string.", "response": "def read_utf8(fh, byteorder, dtype, count, offsetsize):\n \"\"\"Read tag data from file and return as unicode string.\"\"\"\n return fh.read(count).decode('utf-8')"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nread tag data from file and return as numpy array.", "response": "def read_numpy(fh, byteorder, dtype, count, offsetsize):\n \"\"\"Read tag data from file and return as numpy array.\"\"\"\n dtype = 'b' if dtype[-1] == 's' else byteorder+dtype[-1]\n return fh.read_array(dtype, count)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef read_colormap(fh, byteorder, dtype, count, offsetsize):\n cmap = fh.read_array(byteorder+dtype[-1], count)\n cmap.shape = (3, -1)\n return cmap", "response": "Read ColorMap data from file and return as numpy array."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreads JSON tag data from file and return as object.", "response": "def read_json(fh, byteorder, dtype, count, offsetsize):\n \"\"\"Read JSON tag data from file and return as object.\"\"\"\n data = fh.read(count)\n try:\n return json.loads(unicode(stripnull(data), 'utf-8'))\n except ValueError:\n log.warning('read_json: invalid JSON')"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreads FluoView mm_header tag from file and return as dict.", "response": "def read_mm_header(fh, byteorder, dtype, count, offsetsize):\n \"\"\"Read FluoView mm_header tag from file and return as dict.\"\"\"\n mmh = fh.read_record(TIFF.MM_HEADER, byteorder=byteorder)\n mmh = recarray2dict(mmh)\n mmh['Dimensions'] = [\n (bytes2str(d[0]).strip(), d[1], d[2], d[3], bytes2str(d[4]).strip())\n for d in mmh['Dimensions']]\n d = mmh['GrayChannel']\n mmh['GrayChannel'] = (\n bytes2str(d[0]).strip(), d[1], d[2], d[3], bytes2str(d[4]).strip())\n return mmh"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreading MetaMorph STK UIC1Tag from file and return as dict.", "response": "def read_uic1tag(fh, byteorder, dtype, count, offsetsize, planecount=None):\n \"\"\"Read MetaMorph STK UIC1Tag from file and return as dict.\n\n Return empty dictionary if planecount is unknown.\n\n \"\"\"\n assert dtype in ('2I', '1I') and byteorder == '<'\n result = {}\n if dtype == '2I':\n # pre MetaMorph 2.5 (not tested)\n values = fh.read_array('<u4', 2*count).reshape(count, 2)\n result = {'ZDistance': values[:, 0] / values[:, 1]}\n elif planecount:\n for _ in range(count):\n tagid = struct.unpack('<I', fh.read(4))[0]\n if tagid in (28, 29, 37, 40, 41):\n # silently skip unexpected tags\n fh.read(4)\n continue\n name, value = read_uic_tag(fh, tagid, planecount, offset=True)\n result[name] = value\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreads MetaMorph STK UIC2Tag from file and return as dict.", "response": "def read_uic2tag(fh, byteorder, dtype, planecount, offsetsize):\n \"\"\"Read MetaMorph STK UIC2Tag from file and return as dict.\"\"\"\n assert dtype == '2I' and byteorder == '<'\n values = fh.read_array('<u4', 6*planecount).reshape(planecount, 6)\n return {\n 'ZDistance': values[:, 0] / values[:, 1],\n 'DateCreated': values[:, 2], # julian days\n 'TimeCreated': values[:, 3], # milliseconds\n 'DateModified': values[:, 4], # julian days\n 'TimeModified': values[:, 5]}"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nread MetaMorph STK UIC4Tag from file and return as dict.", "response": "def read_uic4tag(fh, byteorder, dtype, planecount, offsetsize):\n \"\"\"Read MetaMorph STK UIC4Tag from file and return as dict.\"\"\"\n assert dtype == '1I' and byteorder == '<'\n result = {}\n while True:\n tagid = struct.unpack('<H', fh.read(2))[0]\n if tagid == 0:\n break\n name, value = read_uic_tag(fh, tagid, planecount, offset=False)\n result[name] = value\n return result"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nread a single UIC tag from file and return tag name and value.", "response": "def read_uic_tag(fh, tagid, planecount, offset):\n \"\"\"Read a single UIC tag value from file and return tag name and value.\n\n UIC1Tags use an offset.\n\n \"\"\"\n def read_int(count=1):\n value = struct.unpack('<%iI' % count, fh.read(4*count))\n return value[0] if count == 1 else value\n\n try:\n name, dtype = TIFF.UIC_TAGS[tagid]\n except IndexError:\n # unknown tag\n return '_TagId%i' % tagid, read_int()\n\n Fraction = TIFF.UIC_TAGS[4][1]\n\n if offset:\n pos = fh.tell()\n if dtype not in (int, None):\n off = read_int()\n if off < 8:\n if dtype is str:\n return name, ''\n log.warning(\"read_uic_tag: invalid offset for tag '%s' (%i)\",\n name, off)\n return name, off\n fh.seek(off)\n\n if dtype is None:\n # skip\n name = '_' + name\n value = read_int()\n elif dtype is int:\n # int\n value = read_int()\n elif dtype is Fraction:\n # fraction\n value = read_int(2)\n value = value[0] / value[1]\n elif dtype is julian_datetime:\n # datetime\n value = julian_datetime(*read_int(2))\n elif dtype is read_uic_image_property:\n # ImagePropertyEx\n value = read_uic_image_property(fh)\n elif dtype is str:\n # pascal string\n size = read_int()\n if 0 <= size < 2**10:\n value = struct.unpack('%is' % size, fh.read(size))[0][:-1]\n value = bytes2str(stripnull(value))\n elif offset:\n value = ''\n log.warning(\"read_uic_tag: corrupt string in tag '%s'\", name)\n else:\n raise ValueError('read_uic_tag: invalid string size %i' % size)\n elif dtype == '%ip':\n # sequence of pascal strings\n value = []\n for _ in range(planecount):\n size = read_int()\n if 0 <= size < 2**10:\n string = struct.unpack('%is' % size, fh.read(size))[0][:-1]\n string = bytes2str(stripnull(string))\n value.append(string)\n elif offset:\n log.warning(\"read_uic_tag: corrupt string in tag '%s'\", name)\n else:\n raise ValueError('read_uic_tag: invalid string size: %i' %\n size)\n else:\n # struct or numpy type\n dtype = '<' + dtype\n if '%i' in dtype:\n dtype = dtype % planecount\n if '(' in dtype:\n # numpy type\n value = fh.read_array(dtype, 1)[0]\n if value.shape[-1] == 2:\n # assume fractions\n value = value[..., 0] / value[..., 1]\n else:\n # struct format\n value = struct.unpack(dtype, fh.read(struct.calcsize(dtype)))\n if len(value) == 1:\n value = value[0]\n\n if offset:\n fh.seek(pos + 4)\n\n return name, value"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef read_uic_image_property(fh):\n # TODO: test this\n size = struct.unpack('B', fh.read(1))[0]\n name = struct.unpack('%is' % size, fh.read(size))[0][:-1]\n flags, prop = struct.unpack('<IB', fh.read(5))\n if prop == 1:\n value = struct.unpack('II', fh.read(8))\n value = value[0] / value[1]\n else:\n size = struct.unpack('B', fh.read(1))[0]\n value = struct.unpack('%is' % size, fh.read(size))[0]\n return dict(name=name, flags=flags, value=value)", "response": "Read UIC ImagePropertyEx tag from file and return as dict."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreads CZ_LSMINFO tag from file and return as dict.", "response": "def read_cz_lsminfo(fh, byteorder, dtype, count, offsetsize):\n \"\"\"Read CZ_LSMINFO tag from file and return as dict.\"\"\"\n assert byteorder == '<'\n magic_number, structure_size = struct.unpack('<II', fh.read(8))\n if magic_number not in (50350412, 67127628):\n raise ValueError('invalid CZ_LSMINFO structure')\n fh.seek(-8, 1)\n\n if structure_size < numpy.dtype(TIFF.CZ_LSMINFO).itemsize:\n # adjust structure according to structure_size\n lsminfo = []\n size = 0\n for name, dtype in TIFF.CZ_LSMINFO:\n size += numpy.dtype(dtype).itemsize\n if size > structure_size:\n break\n lsminfo.append((name, dtype))\n else:\n lsminfo = TIFF.CZ_LSMINFO\n\n lsminfo = fh.read_record(lsminfo, byteorder=byteorder)\n lsminfo = recarray2dict(lsminfo)\n\n # read LSM info subrecords at offsets\n for name, reader in TIFF.CZ_LSMINFO_READERS.items():\n if reader is None:\n continue\n offset = lsminfo.get('Offset' + name, 0)\n if offset < 8:\n continue\n fh.seek(offset)\n try:\n lsminfo[name] = reader(fh)\n except ValueError:\n pass\n return lsminfo"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nread LSM sequence of float pairs from file and return as list.", "response": "def read_lsm_floatpairs(fh):\n \"\"\"Read LSM sequence of float pairs from file and return as list.\"\"\"\n size = struct.unpack('<i', fh.read(4))[0]\n return fh.read_array('<2f8', count=size)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef read_lsm_positions(fh):\n size = struct.unpack('<I', fh.read(4))[0]\n return fh.read_array('<2f8', count=size)", "response": "Read LSM positions from file and return as list."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef read_lsm_timestamps(fh):\n size, count = struct.unpack('<ii', fh.read(8))\n if size != (8 + 8 * count):\n log.warning('read_lsm_timestamps: invalid LSM TimeStamps block')\n return []\n # return struct.unpack('<%dd' % count, fh.read(8*count))\n return fh.read_array('<f8', count=count)", "response": "Read LSM time stamps from file and return as list."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreads LSM events from file and return as list of ( time type text ).", "response": "def read_lsm_eventlist(fh):\n \"\"\"Read LSM events from file and return as list of (time, type, text).\"\"\"\n count = struct.unpack('<II', fh.read(8))[1]\n events = []\n while count > 0:\n esize, etime, etype = struct.unpack('<IdI', fh.read(16))\n etext = bytes2str(stripnull(fh.read(esize - 16)))\n events.append((etime, etype, etext))\n count -= 1\n return events"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreads LSM ChannelColors structure from file and return as dict.", "response": "def read_lsm_channelcolors(fh):\n \"\"\"Read LSM ChannelColors structure from file and return as dict.\"\"\"\n result = {'Mono': False, 'Colors': [], 'ColorNames': []}\n pos = fh.tell()\n (size, ncolors, nnames,\n coffset, noffset, mono) = struct.unpack('<IIIIII', fh.read(24))\n if ncolors != nnames:\n log.warning(\n 'read_lsm_channelcolors: invalid LSM ChannelColors structure')\n return result\n result['Mono'] = bool(mono)\n # Colors\n fh.seek(pos + coffset)\n colors = fh.read_array('uint8', count=ncolors*4).reshape((ncolors, 4))\n result['Colors'] = colors.tolist()\n # ColorNames\n fh.seek(pos + noffset)\n buffer = fh.read(size - noffset)\n names = []\n while len(buffer) > 4:\n size = struct.unpack('<I', buffer[:4])[0]\n names.append(bytes2str(buffer[4:3+size]))\n buffer = buffer[4+size:]\n result['ColorNames'] = names\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef read_lsm_scaninfo(fh):\n block = {}\n blocks = [block]\n unpack = struct.unpack\n if struct.unpack('<I', fh.read(4))[0] != 0x10000000:\n # not a Recording sub block\n log.warning('read_lsm_scaninfo: invalid LSM ScanInfo structure')\n return block\n fh.read(8)\n while True:\n entry, dtype, size = unpack('<III', fh.read(12))\n if dtype == 2:\n # ascii\n value = bytes2str(stripnull(fh.read(size)))\n elif dtype == 4:\n # long\n value = unpack('<i', fh.read(4))[0]\n elif dtype == 5:\n # rational\n value = unpack('<d', fh.read(8))[0]\n else:\n value = 0\n if entry in TIFF.CZ_LSMINFO_SCANINFO_ARRAYS:\n blocks.append(block)\n name = TIFF.CZ_LSMINFO_SCANINFO_ARRAYS[entry]\n newobj = []\n block[name] = newobj\n block = newobj\n elif entry in TIFF.CZ_LSMINFO_SCANINFO_STRUCTS:\n blocks.append(block)\n newobj = {}\n block.append(newobj)\n block = newobj\n elif entry in TIFF.CZ_LSMINFO_SCANINFO_ATTRIBUTES:\n name = TIFF.CZ_LSMINFO_SCANINFO_ATTRIBUTES[entry]\n block[name] = value\n elif entry == 0xffffffff:\n # end sub block\n block = blocks.pop()\n else:\n # unknown entry\n block['Entry0x%x' % entry] = value\n if not blocks:\n break\n return block", "response": "Read LSM ScanInfo structure from file and return as dict."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef read_sis(fh, byteorder, dtype, count, offsetsize):\n result = {}\n\n (magic, _, minute, hour, day, month, year, _, name, tagcount\n ) = struct.unpack('<4s6shhhhh6s32sh', fh.read(60))\n\n if magic != b'SIS0':\n raise ValueError('invalid OlympusSIS structure')\n\n result['name'] = bytes2str(stripnull(name))\n try:\n result['datetime'] = datetime.datetime(1900+year, month+1, day,\n hour, minute)\n except ValueError:\n pass\n\n data = fh.read(8 * tagcount)\n for i in range(0, tagcount*8, 8):\n tagtype, count, offset = struct.unpack('<hhI', data[i:i+8])\n fh.seek(offset)\n if tagtype == 1:\n # general data\n (_, lenexp, xcal, ycal, _, mag, _, camname, pictype,\n ) = struct.unpack('<10shdd8sd2s34s32s', fh.read(112)) # 220\n m = math.pow(10, lenexp)\n result['pixelsizex'] = xcal * m\n result['pixelsizey'] = ycal * m\n result['magnification'] = mag\n result['cameraname'] = bytes2str(stripnull(camname))\n result['picturetype'] = bytes2str(stripnull(pictype))\n elif tagtype == 10:\n # channel data\n continue\n # TODO: does not seem to work?\n # (length, _, exptime, emv, _, camname, _, mictype,\n # ) = struct.unpack('<h22sId4s32s48s32s', fh.read(152)) # 720\n # result['exposuretime'] = exptime\n # result['emvoltage'] = emv\n # result['cameraname2'] = bytes2str(stripnull(camname))\n # result['microscopename'] = bytes2str(stripnull(mictype))\n\n return result", "response": "Read OlympusSIS structure and return as dict."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreads OlympusSIS INI string and return as dict.", "response": "def read_sis_ini(fh, byteorder, dtype, count, offsetsize):\n \"\"\"Read OlympusSIS INI string and return as dict.\"\"\"\n inistr = fh.read(count)\n inistr = bytes2str(stripnull(inistr))\n try:\n return olympusini_metadata(inistr)\n except Exception as exc:\n log.warning('olympusini_metadata: %s: %s', exc.__class__.__name__, exc)\n return {}"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreads TVIPS EM - Menu headers and return as dict.", "response": "def read_tvips_header(fh, byteorder, dtype, count, offsetsize):\n \"\"\"Read TVIPS EM-MENU headers and return as dict.\"\"\"\n result = {}\n header = fh.read_record(TIFF.TVIPS_HEADER_V1, byteorder=byteorder)\n for name, typestr in TIFF.TVIPS_HEADER_V1:\n result[name] = header[name].tolist()\n if header['Version'] == 2:\n header = fh.read_record(TIFF.TVIPS_HEADER_V2, byteorder=byteorder)\n if header['Magic'] != int(0xaaaaaaaa):\n log.warning('read_tvips_header: invalid TVIPS v2 magic number')\n return {}\n # decode utf16 strings\n for name, typestr in TIFF.TVIPS_HEADER_V2:\n if typestr.startswith('V'):\n s = header[name].tostring().decode('utf16', errors='ignore')\n result[name] = stripnull(s, null='\\0')\n else:\n result[name] = header[name].tolist()\n # convert nm to m\n for axis in 'XY':\n header['PhysicalPixelSize' + axis] /= 1e9\n header['PixelSize' + axis] /= 1e9\n elif header.version != 1:\n log.warning('read_tvips_header: unknown TVIPS header version')\n return {}\n return result"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef read_fei_metadata(fh, byteorder, dtype, count, offsetsize):\n result = {}\n section = {}\n data = bytes2str(stripnull(fh.read(count)))\n for line in data.splitlines():\n line = line.strip()\n if line.startswith('['):\n section = {}\n result[line[1:-1]] = section\n continue\n try:\n key, value = line.split('=')\n except ValueError:\n continue\n section[key] = astype(value)\n return result", "response": "Read FEI SFEG headers and return as dict."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef read_cz_sem(fh, byteorder, dtype, count, offsetsize):\n result = {'': ()}\n key = None\n data = bytes2str(stripnull(fh.read(count)))\n for line in data.splitlines():\n if line.isupper():\n key = line.lower()\n elif key:\n try:\n name, value = line.split('=')\n except ValueError:\n try:\n name, value = line.split(':', 1)\n except Exception:\n continue\n value = value.strip()\n unit = ''\n try:\n v, u = value.split()\n number = astype(v, (int, float))\n if number != v:\n value = number\n unit = u\n except Exception:\n number = astype(value, (int, float))\n if number != value:\n value = number\n if value in ('No', 'Off'):\n value = False\n elif value in ('Yes', 'On'):\n value = True\n result[key] = (name.strip(), value)\n if unit:\n result[key] += (unit,)\n key = None\n else:\n result[''] += (astype(line, (int, float)),)\n return result", "response": "Read Zeiss SEM tag and return as dict."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef read_nih_image_header(fh, byteorder, dtype, count, offsetsize):\n a = fh.read_record(TIFF.NIH_IMAGE_HEADER, byteorder=byteorder)\n a = a.newbyteorder(byteorder)\n a = recarray2dict(a)\n a['XUnit'] = a['XUnit'][:a['XUnitSize']]\n a['UM'] = a['UM'][:a['UMsize']]\n return a", "response": "Read NIH_IMAGE_HEADER tag from file and return as dict."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreading ScanImage BigTIFF v3 static and ROI metadata from open file.", "response": "def read_scanimage_metadata(fh):\n \"\"\"Read ScanImage BigTIFF v3 static and ROI metadata from open file.\n\n Return non-varying frame data as dict and ROI group data as JSON.\n\n The settings can be used to read image data and metadata without parsing\n the TIFF file.\n\n Raise ValueError if file does not contain valid ScanImage v3 metadata.\n\n \"\"\"\n fh.seek(0)\n try:\n byteorder, version = struct.unpack('<2sH', fh.read(4))\n if byteorder != b'II' or version != 43:\n raise Exception\n fh.seek(16)\n magic, version, size0, size1 = struct.unpack('<IIII', fh.read(16))\n if magic != 117637889 or version != 3:\n raise Exception\n except Exception:\n raise ValueError('not a ScanImage BigTIFF v3 file')\n\n frame_data = matlabstr2py(bytes2str(fh.read(size0)[:-1]))\n roi_data = read_json(fh, '<', None, size1, None) if size1 > 1 else {}\n return frame_data, roi_data"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef read_micromanager_metadata(fh):\n fh.seek(0)\n try:\n byteorder = {b'II': '<', b'MM': '>'}[fh.read(2)]\n except IndexError:\n raise ValueError('not a MicroManager TIFF file')\n\n result = {}\n fh.seek(8)\n (index_header, index_offset, display_header, display_offset,\n comments_header, comments_offset, summary_header, summary_length\n ) = struct.unpack(byteorder + 'IIIIIIII', fh.read(32))\n\n if summary_header != 2355492:\n raise ValueError('invalid MicroManager summary header')\n result['Summary'] = read_json(fh, byteorder, None, summary_length, None)\n\n if index_header != 54773648:\n raise ValueError('invalid MicroManager index header')\n fh.seek(index_offset)\n header, count = struct.unpack(byteorder + 'II', fh.read(8))\n if header != 3453623:\n raise ValueError('invalid MicroManager index header')\n data = struct.unpack(byteorder + 'IIIII'*count, fh.read(20*count))\n result['IndexMap'] = {'Channel': data[::5],\n 'Slice': data[1::5],\n 'Frame': data[2::5],\n 'Position': data[3::5],\n 'Offset': data[4::5]}\n\n if display_header != 483765892:\n raise ValueError('invalid MicroManager display header')\n fh.seek(display_offset)\n header, count = struct.unpack(byteorder + 'II', fh.read(8))\n if header != 347834724:\n raise ValueError('invalid MicroManager display header')\n result['DisplaySettings'] = read_json(fh, byteorder, None, count, None)\n\n if comments_header != 99384722:\n raise ValueError('invalid MicroManager comments header')\n fh.seek(comments_offset)\n header, count = struct.unpack(byteorder + 'II', fh.read(8))\n if header != 84720485:\n raise ValueError('invalid MicroManager comments header')\n result['Comments'] = read_json(fh, byteorder, None, count, None)\n\n return result", "response": "Read MicroManager non - TIFF metadata from open file and return as dict."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef imagej_metadata_tag(metadata, byteorder):\n header = [{'>': b'IJIJ', '<': b'JIJI'}[byteorder]]\n bytecounts = [0]\n body = []\n\n def _string(data, byteorder):\n return data.encode('utf-16' + {'>': 'be', '<': 'le'}[byteorder])\n\n def _doubles(data, byteorder):\n return struct.pack(byteorder+('d' * len(data)), *data)\n\n def _ndarray(data, byteorder):\n return data.tobytes()\n\n def _bytes(data, byteorder):\n return data\n\n metadata_types = (\n ('Info', b'info', 1, _string),\n ('Labels', b'labl', None, _string),\n ('Ranges', b'rang', 1, _doubles),\n ('LUTs', b'luts', None, _ndarray),\n ('Plot', b'plot', 1, _bytes),\n ('ROI', b'roi ', 1, _bytes),\n ('Overlays', b'over', None, _bytes))\n\n for key, mtype, count, func in metadata_types:\n if key.lower() in metadata:\n key = key.lower()\n elif key not in metadata:\n continue\n if byteorder == '<':\n mtype = mtype[::-1]\n values = metadata[key]\n if count is None:\n count = len(values)\n else:\n values = [values]\n header.append(mtype + struct.pack(byteorder+'I', count))\n for value in values:\n data = func(value, byteorder)\n body.append(data)\n bytecounts.append(len(data))\n\n if not body:\n return ()\n body = b''.join(body)\n header = b''.join(header)\n data = header + body\n bytecounts[0] = len(header)\n bytecounts = struct.pack(byteorder+('I' * len(bytecounts)), *bytecounts)\n return ((50839, 'B', len(data), data, True),\n (50838, 'I', len(bytecounts)//4, bytecounts, True))", "response": "Return IJMetadata and IJMetadataByteCounts tags from the metadata dict."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns IJMetadata tag value as dict.", "response": "def imagej_metadata(data, bytecounts, byteorder):\n \"\"\"Return IJMetadata tag value as dict.\n\n The 'Info' string can have multiple formats, e.g. OIF or ScanImage,\n that might be parsed into dicts using the matlabstr2py or\n oiffile.SettingsFile functions.\n\n \"\"\"\n def _string(data, byteorder):\n return data.decode('utf-16' + {'>': 'be', '<': 'le'}[byteorder])\n\n def _doubles(data, byteorder):\n return struct.unpack(byteorder+('d' * (len(data) // 8)), data)\n\n def _lut(data, byteorder):\n return numpy.frombuffer(data, 'uint8').reshape(-1, 256)\n\n def _bytes(data, byteorder):\n return data\n\n metadata_types = { # big-endian\n b'info': ('Info', _string),\n b'labl': ('Labels', _string),\n b'rang': ('Ranges', _doubles),\n b'luts': ('LUTs', _lut),\n b'plot': ('Plots', _bytes),\n b'roi ': ('ROI', _bytes),\n b'over': ('Overlays', _bytes)}\n metadata_types.update( # little-endian\n dict((k[::-1], v) for k, v in metadata_types.items()))\n\n if not bytecounts:\n raise ValueError('no ImageJ metadata')\n\n if not data[:4] in (b'IJIJ', b'JIJI'):\n raise ValueError('invalid ImageJ metadata')\n\n header_size = bytecounts[0]\n if header_size < 12 or header_size > 804:\n raise ValueError('invalid ImageJ metadata header size')\n\n ntypes = (header_size - 4) // 8\n header = struct.unpack(byteorder+'4sI'*ntypes, data[4:4+ntypes*8])\n pos = 4 + ntypes * 8\n counter = 0\n result = {}\n for mtype, count in zip(header[::2], header[1::2]):\n values = []\n name, func = metadata_types.get(mtype, (bytes2str(mtype), read_bytes))\n for _ in range(count):\n counter += 1\n pos1 = pos + bytecounts[counter]\n values.append(func(data[pos:pos1], byteorder))\n pos = pos1\n result[name.strip()] = values[0] if count == 1 else values\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef imagej_description_metadata(description):\n def _bool(val):\n return {'true': True, 'false': False}[val.lower()]\n\n result = {}\n for line in description.splitlines():\n try:\n key, val = line.split('=')\n except Exception:\n continue\n key = key.strip()\n val = val.strip()\n for dtype in (int, float, _bool):\n try:\n val = dtype(val)\n break\n except Exception:\n pass\n result[key] = val\n\n if 'ImageJ' not in result:\n raise ValueError('not a ImageJ image description')\n return result", "response": "Return metatata from ImageJ image description as dict."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef imagej_description(shape, rgb=None, colormaped=False, version=None,\n hyperstack=None, mode=None, loop=None, **kwargs):\n \"\"\"Return ImageJ image description from data shape.\n\n ImageJ can handle up to 6 dimensions in order TZCYXS.\n\n >>> imagej_description((51, 5, 2, 196, 171)) # doctest: +SKIP\n ImageJ=1.11a\n images=510\n channels=2\n slices=5\n frames=51\n hyperstack=true\n mode=grayscale\n loop=false\n\n \"\"\"\n if colormaped:\n raise NotImplementedError('ImageJ colormapping not supported')\n if version is None:\n version = '1.11a'\n shape = imagej_shape(shape, rgb=rgb)\n rgb = shape[-1] in (3, 4)\n\n result = ['ImageJ=%s' % version]\n append = []\n result.append('images=%i' % product(shape[:-3]))\n if hyperstack is None:\n hyperstack = True\n append.append('hyperstack=true')\n else:\n append.append('hyperstack=%s' % bool(hyperstack))\n if shape[2] > 1:\n result.append('channels=%i' % shape[2])\n if mode is None and not rgb:\n mode = 'grayscale'\n if hyperstack and mode:\n append.append('mode=%s' % mode)\n if shape[1] > 1:\n result.append('slices=%i' % shape[1])\n if shape[0] > 1:\n result.append('frames=%i' % shape[0])\n if loop is None:\n append.append('loop=false')\n if loop is not None:\n append.append('loop=%s' % bool(loop))\n for key, value in kwargs.items():\n append.append('%s=%s' % (key.lower(), value))\n\n return '\\n'.join(result + append + [''])", "response": "Return ImageJ image description from data shape."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns shape normalized to 6D ImageJ hyperstack TZCYXS.", "response": "def imagej_shape(shape, rgb=None):\n \"\"\"Return shape normalized to 6D ImageJ hyperstack TZCYXS.\n\n Raise ValueError if not a valid ImageJ hyperstack shape.\n\n >>> imagej_shape((2, 3, 4, 5, 3), False)\n (2, 3, 4, 5, 3, 1)\n\n \"\"\"\n shape = tuple(int(i) for i in shape)\n ndim = len(shape)\n if 1 > ndim > 6:\n raise ValueError('invalid ImageJ hyperstack: not 2 to 6 dimensional')\n if rgb is None:\n rgb = shape[-1] in (3, 4) and ndim > 2\n if rgb and shape[-1] not in (3, 4):\n raise ValueError('invalid ImageJ hyperstack: not a RGB image')\n if not rgb and ndim == 6 and shape[-1] != 1:\n raise ValueError('invalid ImageJ hyperstack: not a non-RGB image')\n if rgb or shape[-1] == 1:\n return (1, ) * (6 - ndim) + shape\n return (1, ) * (5 - ndim) + shape + (1,)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef json_description(shape, **metadata):\n metadata.update(shape=shape)\n return json.dumps(metadata)", "response": "Return JSON image description from data shape and other metadata."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef json_description_metadata(description):\n if description[:6] == 'shape=':\n # old-style 'shaped' description; not JSON\n shape = tuple(int(i) for i in description[7:-1].split(','))\n return dict(shape=shape)\n if description[:1] == '{' and description[-1:] == '}':\n # JSON description\n return json.loads(description)\n raise ValueError('invalid JSON image description', description)", "response": "Return metatata from JSON formated image description as dict."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef fluoview_description_metadata(description, ignoresections=None):\n if not description.startswith('['):\n raise ValueError('invalid FluoView image description')\n if ignoresections is None:\n ignoresections = {'Region Info (Fields)', 'Protocol Description'}\n\n result = {}\n sections = [result]\n comment = False\n for line in description.splitlines():\n if not comment:\n line = line.strip()\n if not line:\n continue\n if line[0] == '[':\n if line[-5:] == ' End]':\n # close section\n del sections[-1]\n section = sections[-1]\n name = line[1:-5]\n if comment:\n section[name] = '\\n'.join(section[name])\n if name[:4] == 'LUT ':\n a = numpy.array(section[name], dtype='uint8')\n a.shape = -1, 3\n section[name] = a\n continue\n # new section\n comment = False\n name = line[1:-1]\n if name[:4] == 'LUT ':\n section = []\n elif name in ignoresections:\n section = []\n comment = True\n else:\n section = {}\n sections.append(section)\n result[name] = section\n continue\n # add entry\n if comment:\n section.append(line)\n continue\n line = line.split('=', 1)\n if len(line) == 1:\n section[line[0].strip()] = None\n continue\n key, value = line\n if key[:4] == 'RGB ':\n section.extend(int(rgb) for rgb in value.split())\n else:\n section[key.strip()] = astype(value.strip())\n return result", "response": "Return metatata from FluoView image description as dict."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pilatus_description_metadata(description):\n result = {}\n if not description.startswith('# '):\n return result\n for c in '#:=,()':\n description = description.replace(c, ' ')\n for line in description.split('\\n'):\n if line[:2] != ' ':\n continue\n line = line.split()\n name = line[0]\n if line[0] not in TIFF.PILATUS_HEADER:\n try:\n result['DateTime'] = datetime.datetime.strptime(\n ' '.join(line), '%Y-%m-%dT%H %M %S.%f')\n except Exception:\n result[name] = ' '.join(line[1:])\n continue\n indices, dtype = TIFF.PILATUS_HEADER[line[0]]\n if isinstance(indices[0], slice):\n # assumes one slice\n values = line[indices[0]]\n else:\n values = [line[i] for i in indices]\n if dtype is float and values[0] == 'not':\n values = ['NaN']\n values = tuple(dtype(v) for v in values)\n if dtype == str:\n values = ' '.join(values)\n elif len(values) == 1:\n values = values[0]\n result[name] = values\n return result", "response": "Return metatata from Pilatus image description as dict."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning metatata from Aperio image description as dict.", "response": "def svs_description_metadata(description):\n \"\"\"Return metatata from Aperio image description as dict.\n\n The Aperio image description format is unspecified. Expect failures.\n\n >>> svs_description_metadata('Aperio Image Library v1.0')\n {'Aperio Image Library': 'v1.0'}\n\n \"\"\"\n if not description.startswith('Aperio Image Library '):\n raise ValueError('invalid Aperio image description')\n result = {}\n lines = description.split('\\n')\n key, value = lines[0].strip().rsplit(None, 1) # 'Aperio Image Library'\n result[key.strip()] = value.strip()\n if len(lines) == 1:\n return result\n items = lines[1].split('|')\n result[''] = items[0].strip() # TODO: parse this?\n for item in items[1:]:\n key, value = item.split(' = ')\n result[key.strip()] = astype(value.strip())\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning metadata from MetaMorph image description as list of dict.", "response": "def stk_description_metadata(description):\n \"\"\"Return metadata from MetaMorph image description as list of dict.\n\n The MetaMorph image description format is unspecified. Expect failures.\n\n \"\"\"\n description = description.strip()\n if not description:\n return []\n try:\n description = bytes2str(description)\n except UnicodeDecodeError as exc:\n log.warning('stk_description_metadata: %s: %s',\n exc.__class__.__name__, exc)\n return []\n result = []\n for plane in description.split('\\x00'):\n d = {}\n for line in plane.split('\\r\\n'):\n line = line.split(':', 1)\n if len(line) > 1:\n name, value = line\n d[name.strip()] = astype(value.strip())\n else:\n value = line[0].strip()\n if value:\n if '' in d:\n d[''].append(value)\n else:\n d[''] = [value]\n result.append(d)\n return result"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn metatata from MetaSeries image description as dict.", "response": "def metaseries_description_metadata(description):\n \"\"\"Return metatata from MetaSeries image description as dict.\"\"\"\n if not description.startswith('<MetaData>'):\n raise ValueError('invalid MetaSeries image description')\n\n from xml.etree import cElementTree as etree # delayed import\n root = etree.fromstring(description)\n types = {'float': float, 'int': int,\n 'bool': lambda x: asbool(x, 'on', 'off')}\n\n def parse(root, result):\n # recursive\n for child in root:\n attrib = child.attrib\n if not attrib:\n result[child.tag] = parse(child, {})\n continue\n if 'id' in attrib:\n i = attrib['id']\n t = attrib['type']\n v = attrib['value']\n if t in types:\n result[i] = types[t](v)\n else:\n result[i] = v\n return result\n\n adict = parse(root, {})\n if 'Description' in adict:\n adict['Description'] = adict['Description'].replace(' ', '\\n')\n return adict"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn metatata from ScanImage artist tag as dict.", "response": "def scanimage_artist_metadata(artist):\n \"\"\"Return metatata from ScanImage artist tag as dict.\"\"\"\n try:\n return json.loads(artist)\n except ValueError as exc:\n log.warning('scanimage_artist_metadata: %s: %s',\n exc.__class__.__name__, exc)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn OlympusSIS metadata from INI string.", "response": "def olympusini_metadata(inistr):\n \"\"\"Return OlympusSIS metadata from INI string.\n\n No documentation is available.\n\n \"\"\"\n def keyindex(key):\n # split key into name and index\n index = 0\n i = len(key.rstrip('0123456789'))\n if i < len(key):\n index = int(key[i:]) - 1\n key = key[:i]\n return key, index\n\n result = {}\n bands = []\n zpos = None\n tpos = None\n for line in inistr.splitlines():\n line = line.strip()\n if line == '' or line[0] == ';':\n continue\n if line[0] == '[' and line[-1] == ']':\n section_name = line[1:-1]\n result[section_name] = section = {}\n if section_name == 'Dimension':\n result['axes'] = axes = []\n result['shape'] = shape = []\n elif section_name == 'ASD':\n result[section_name] = []\n elif section_name == 'Z':\n if 'Dimension' in result:\n result[section_name]['ZPos'] = zpos = []\n elif section_name == 'Time':\n if 'Dimension' in result:\n result[section_name]['TimePos'] = tpos = []\n elif section_name == 'Band':\n nbands = result['Dimension']['Band']\n bands = [{'LUT': []} for i in range(nbands)]\n result[section_name] = bands\n iband = 0\n else:\n key, value = line.split('=')\n if value.strip() == '':\n value = None\n elif ',' in value:\n value = tuple(astype(v) for v in value.split(','))\n else:\n value = astype(value)\n\n if section_name == 'Dimension':\n section[key] = value\n axes.append(key)\n shape.append(value)\n elif section_name == 'ASD':\n if key == 'Count':\n result['ASD'] = [{}] * value\n else:\n key, index = keyindex(key)\n result['ASD'][index][key] = value\n elif section_name == 'Band':\n if key[:3] == 'LUT':\n lut = bands[iband]['LUT']\n value = struct.pack('<I', value)\n lut.append(\n [ord(value[0:1]), ord(value[1:2]), ord(value[2:3])])\n else:\n key, iband = keyindex(key)\n bands[iband][key] = value\n elif key[:4] == 'ZPos' and zpos is not None:\n zpos.append(value)\n elif key[:7] == 'TimePos' and tpos is not None:\n tpos.append(value)\n else:\n section[key] = value\n\n if 'axes' in result:\n sisaxes = {'Band': 'C'}\n axes = []\n shape = []\n for i, x in zip(result['shape'], result['axes']):\n if i > 1:\n axes.append(sisaxes.get(x, x[0].upper()))\n shape.append(i)\n result['axes'] = ''.join(axes)\n result['shape'] = tuple(shape)\n try:\n result['Z']['ZPos'] = numpy.array(\n result['Z']['ZPos'][:result['Dimension']['Z']], 'float64')\n except Exception:\n pass\n try:\n result['Time']['TimePos'] = numpy.array(\n result['Time']['TimePos'][:result['Dimension']['Time']], 'int32')\n except Exception:\n pass\n for band in bands:\n band['LUT'] = numpy.array(band['LUT'], 'uint8')\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndecoding tile segment bytes into 5D output array.", "response": "def tile_decode(tile, tileindex, tileshape, tiledshape,\n lsb2msb, decompress, unpack, unpredict, out):\n \"\"\"Decode tile segment bytes into 5D output array.\"\"\"\n _, imagedepth, imagelength, imagewidth, _ = out.shape\n tileddepth, tiledlength, tiledwidth = tiledshape\n tiledepth, tilelength, tilewidth, samples = tileshape\n tilesize = tiledepth * tilelength * tilewidth * samples\n\n pl = tileindex // (tiledwidth * tiledlength * tileddepth)\n td = (tileindex // (tiledwidth * tiledlength)) % tileddepth * tiledepth\n tl = (tileindex // tiledwidth) % tiledlength * tilelength\n tw = tileindex % tiledwidth * tilewidth\n\n if tile:\n if lsb2msb:\n tile = bitorder_decode(tile, out=tile)\n tile = decompress(tile)\n tile = unpack(tile)\n # decompression / unpacking might return too many bytes\n tile = tile[:tilesize]\n try:\n # complete tile according to TIFF specification\n tile.shape = tileshape\n except ValueError:\n # tile fills remaining space; found in some JPEG compressed slides\n s = (min(imagedepth - td, tiledepth),\n min(imagelength - tl, tilelength),\n min(imagewidth - tw, tilewidth),\n samples)\n try:\n tile.shape = s\n except ValueError:\n # incomplete tile; see gdal issue #1179\n log.warning('tile_decode: incomplete tile %s %s',\n tile.shape, tileshape)\n t = numpy.zeros(tilesize, tile.dtype)\n s = min(tile.size, tilesize)\n t[:s] = tile[:s]\n tile = t.reshape(tileshape)\n\n tile = unpredict(tile, axis=-2, out=tile)\n out[pl, td:td+tiledepth, tl:tl+tilelength, tw:tw+tilewidth] = (\n tile[:imagedepth-td, :imagelength-tl, :imagewidth-tw])\n else:\n out[pl, td:td+tiledepth, tl:tl+tilelength, tw:tw+tilewidth] = 0"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nunpacking a byte string containing packed RGB samples.", "response": "def unpack_rgb(data, dtype=None, bitspersample=None, rescale=True):\n \"\"\"Return array from byte string containing packed samples.\n\n Use to unpack RGB565 or RGB555 to RGB888 format.\n\n Parameters\n ----------\n data : byte str\n The data to be decoded. Samples in each pixel are stored consecutively.\n Pixels are aligned to 8, 16, or 32 bit boundaries.\n dtype : numpy.dtype\n The sample data type. The byteorder applies also to the data stream.\n bitspersample : tuple\n Number of bits for each sample in a pixel.\n rescale : bool\n Upscale samples to the number of bits in dtype.\n\n Returns\n -------\n numpy.ndarray\n Flattened array of unpacked samples of native dtype.\n\n Examples\n --------\n >>> data = struct.pack('BBBB', 0x21, 0x08, 0xff, 0xff)\n >>> print(unpack_rgb(data, '<B', (5, 6, 5), False))\n [ 1 1 1 31 63 31]\n >>> print(unpack_rgb(data, '<B', (5, 6, 5)))\n [ 8 4 8 255 255 255]\n >>> print(unpack_rgb(data, '<B', (5, 5, 5)))\n [ 16 8 8 255 255 255]\n\n \"\"\"\n if bitspersample is None:\n bitspersample = (5, 6, 5)\n if dtype is None:\n dtype = '<B'\n dtype = numpy.dtype(dtype)\n bits = int(numpy.sum(bitspersample))\n if not (bits <= 32 and all(i <= dtype.itemsize*8 for i in bitspersample)):\n raise ValueError('sample size not supported: %s' % str(bitspersample))\n dt = next(i for i in 'BHI' if numpy.dtype(i).itemsize*8 >= bits)\n data = numpy.frombuffer(data, dtype.byteorder+dt)\n result = numpy.empty((data.size, len(bitspersample)), dtype.char)\n for i, bps in enumerate(bitspersample):\n t = data >> int(numpy.sum(bitspersample[i+1:]))\n t &= int('0b'+'1'*bps, 2)\n if rescale:\n o = ((dtype.itemsize * 8) // bps + 1) * bps\n if o > data.dtype.itemsize * 8:\n t = t.astype('I')\n t *= (2**o - 1) // (2**bps - 1)\n t //= 2**(o - (dtype.itemsize * 8))\n result[:, i] = t\n return result.reshape(-1)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef bitorder_decode(data, out=None, _bitorder=[]):\n if not _bitorder:\n _bitorder.append(\n b'\\x00\\x80@\\xc0 \\xa0`\\xe0\\x10\\x90P\\xd00\\xb0p\\xf0\\x08\\x88H\\xc8('\n b'\\xa8h\\xe8\\x18\\x98X\\xd88\\xb8x\\xf8\\x04\\x84D\\xc4$\\xa4d\\xe4\\x14'\n b'\\x94T\\xd44\\xb4t\\xf4\\x0c\\x8cL\\xcc,\\xacl\\xec\\x1c\\x9c\\\\\\xdc<\\xbc|'\n b'\\xfc\\x02\\x82B\\xc2\"\\xa2b\\xe2\\x12\\x92R\\xd22\\xb2r\\xf2\\n\\x8aJ\\xca*'\n b'\\xaaj\\xea\\x1a\\x9aZ\\xda:\\xbaz\\xfa\\x06\\x86F\\xc6&\\xa6f\\xe6\\x16'\n b'\\x96V\\xd66\\xb6v\\xf6\\x0e\\x8eN\\xce.\\xaen\\xee\\x1e\\x9e^\\xde>\\xbe~'\n b'\\xfe\\x01\\x81A\\xc1!\\xa1a\\xe1\\x11\\x91Q\\xd11\\xb1q\\xf1\\t\\x89I\\xc9)'\n b'\\xa9i\\xe9\\x19\\x99Y\\xd99\\xb9y\\xf9\\x05\\x85E\\xc5%\\xa5e\\xe5\\x15'\n b'\\x95U\\xd55\\xb5u\\xf5\\r\\x8dM\\xcd-\\xadm\\xed\\x1d\\x9d]\\xdd=\\xbd}'\n b'\\xfd\\x03\\x83C\\xc3#\\xa3c\\xe3\\x13\\x93S\\xd33\\xb3s\\xf3\\x0b\\x8bK'\n b'\\xcb+\\xabk\\xeb\\x1b\\x9b[\\xdb;\\xbb{\\xfb\\x07\\x87G\\xc7\\'\\xa7g\\xe7'\n b'\\x17\\x97W\\xd77\\xb7w\\xf7\\x0f\\x8fO\\xcf/\\xafo\\xef\\x1f\\x9f_'\n b'\\xdf?\\xbf\\x7f\\xff')\n _bitorder.append(numpy.frombuffer(_bitorder[0], dtype='uint8'))\n try:\n view = data.view('uint8')\n numpy.take(_bitorder[1], view, out=view)\n return data\n except AttributeError:\n return data.translate(_bitorder[0])\n except ValueError:\n raise NotImplementedError('slices of arrays not supported')\n return None", "response": "Decode data into a byte string or numpy array."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndecompress byte string to array of integers.", "response": "def packints_decode(data, dtype, numbits, runlen=0, out=None):\n \"\"\"Decompress byte string to array of integers.\n\n This implementation only handles itemsizes 1, 8, 16, 32, and 64 bits.\n Install the imagecodecs package for decoding other integer sizes.\n\n Parameters\n ----------\n data : byte str\n Data to decompress.\n dtype : numpy.dtype or str\n A numpy boolean or integer type.\n numbits : int\n Number of bits per integer.\n runlen : int\n Number of consecutive integers, after which to start at next byte.\n\n Examples\n --------\n >>> packints_decode(b'a', 'B', 1)\n array([0, 1, 1, 0, 0, 0, 0, 1], dtype=uint8)\n\n \"\"\"\n if numbits == 1: # bitarray\n data = numpy.frombuffer(data, '|B')\n data = numpy.unpackbits(data)\n if runlen % 8:\n data = data.reshape(-1, runlen + (8 - runlen % 8))\n data = data[:, :runlen].reshape(-1)\n return data.astype(dtype)\n if numbits in (8, 16, 32, 64):\n return numpy.frombuffer(data, dtype)\n raise NotImplementedError('unpacking %s-bit integers to %s not supported'\n % (numbits, numpy.dtype(dtype)))"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the palette - colored image.", "response": "def apply_colormap(image, colormap, contig=True):\n \"\"\"Return palette-colored image.\n\n The image values are used to index the colormap on axis 1. The returned\n image is of shape image.shape+colormap.shape[0] and dtype colormap.dtype.\n\n Parameters\n ----------\n image : numpy.ndarray\n Indexes into the colormap.\n colormap : numpy.ndarray\n RGB lookup table aka palette of shape (3, 2**bits_per_sample).\n contig : bool\n If True, return a contiguous array.\n\n Examples\n --------\n >>> image = numpy.arange(256, dtype='uint8')\n >>> colormap = numpy.vstack([image, image, image]).astype('uint16') * 256\n >>> apply_colormap(image, colormap)[-1]\n array([65280, 65280, 65280], dtype=uint16)\n\n \"\"\"\n image = numpy.take(colormap, image, axis=1)\n image = numpy.rollaxis(image, 0, image.ndim)\n if contig:\n image = numpy.ascontiguousarray(image)\n return image"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a reoriented version of an image array.", "response": "def reorient(image, orientation):\n \"\"\"Return reoriented view of image array.\n\n Parameters\n ----------\n image : numpy.ndarray\n Non-squeezed output of asarray() functions.\n Axes -3 and -2 must be image length and width respectively.\n orientation : int or str\n One of TIFF.ORIENTATION names or values.\n\n \"\"\"\n orient = TIFF.ORIENTATION\n orientation = enumarg(orient, orientation)\n\n if orientation == orient.TOPLEFT:\n return image\n if orientation == orient.TOPRIGHT:\n return image[..., ::-1, :]\n if orientation == orient.BOTLEFT:\n return image[..., ::-1, :, :]\n if orientation == orient.BOTRIGHT:\n return image[..., ::-1, ::-1, :]\n if orientation == orient.LEFTTOP:\n return numpy.swapaxes(image, -3, -2)\n if orientation == orient.RIGHTTOP:\n return numpy.swapaxes(image, -3, -2)[..., ::-1, :]\n if orientation == orient.RIGHTBOT:\n return numpy.swapaxes(image, -3, -2)[..., ::-1, :, :]\n if orientation == orient.LEFTBOT:\n return numpy.swapaxes(image, -3, -2)[..., ::-1, ::-1, :]\n return image"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns read - only view into input array with elements repeated.", "response": "def repeat_nd(a, repeats):\n \"\"\"Return read-only view into input array with elements repeated.\n\n Zoom nD image by integer factors using nearest neighbor interpolation\n (box filter).\n\n Parameters\n ----------\n a : array_like\n Input array.\n repeats : sequence of int\n The number of repetitions to apply along each dimension of input array.\n\n Examples\n --------\n >>> repeat_nd([[1, 2], [3, 4]], (2, 2))\n array([[1, 1, 2, 2],\n [1, 1, 2, 2],\n [3, 3, 4, 4],\n [3, 3, 4, 4]])\n\n \"\"\"\n a = numpy.asarray(a)\n reshape = []\n shape = []\n strides = []\n for i, j, k in zip(a.strides, a.shape, repeats):\n shape.extend((j, k))\n strides.extend((i, 0))\n reshape.append(j * k)\n return numpy.lib.stride_tricks.as_strided(\n a, shape, strides, writeable=False).reshape(reshape)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef reshape_nd(data_or_shape, ndim):\n is_shape = isinstance(data_or_shape, tuple)\n shape = data_or_shape if is_shape else data_or_shape.shape\n if len(shape) >= ndim:\n return data_or_shape\n shape = (1,) * (ndim - len(shape)) + shape\n return shape if is_shape else data_or_shape.reshape(shape)", "response": "Return image array or shape with at least ndim dimensions."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef squeeze_axes(shape, axes, skip=None):\n if len(shape) != len(axes):\n raise ValueError('dimensions of axes and shape do not match')\n if skip is None:\n skip = 'XY'\n shape, axes = zip(*(i for i in zip(shape, axes)\n if i[0] > 1 or i[1] in skip))\n return tuple(shape), ''.join(axes)", "response": "Return shape and axes with single - dimensional entries removed."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning image with its axes permuted to match specified axes.", "response": "def transpose_axes(image, axes, asaxes=None):\n \"\"\"Return image with its axes permuted to match specified axes.\n\n A view is returned if possible.\n\n >>> transpose_axes(numpy.zeros((2, 3, 4, 5)), 'TYXC', asaxes='CTZYX').shape\n (5, 2, 1, 3, 4)\n\n \"\"\"\n for ax in axes:\n if ax not in asaxes:\n raise ValueError('unknown axis %s' % ax)\n # add missing axes to image\n if asaxes is None:\n asaxes = 'CTZYX'\n shape = image.shape\n for ax in reversed(asaxes):\n if ax not in axes:\n axes = ax + axes\n shape = (1,) + shape\n image = image.reshape(shape)\n # transpose axes\n image = image.transpose([axes.index(ax) for ax in asaxes])\n return image"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning axes matching new shape. By default unknown dimensions are labelled Q.", "response": "def reshape_axes(axes, shape, newshape, unknown=None):\n \"\"\"Return axes matching new shape.\n\n By default, unknown dimensions are labelled 'Q'.\n\n >>> reshape_axes('YXS', (219, 301, 1), (219, 301))\n 'YX'\n >>> reshape_axes('IYX', (12, 219, 301), (3, 4, 219, 1, 301, 1))\n 'QQYQXQ'\n\n \"\"\"\n shape = tuple(shape)\n newshape = tuple(newshape)\n if len(axes) != len(shape):\n raise ValueError('axes do not match shape')\n\n size = product(shape)\n newsize = product(newshape)\n if size != newsize:\n raise ValueError('cannot reshape %s to %s' % (shape, newshape))\n if not axes or not newshape:\n return ''\n\n lendiff = max(0, len(shape) - len(newshape))\n if lendiff:\n newshape = newshape + (1,) * lendiff\n\n i = len(shape)-1\n prodns = 1\n prods = 1\n result = []\n for ns in newshape[::-1]:\n prodns *= ns\n while i > 0 and shape[i] == 1 and ns != 1:\n i -= 1\n if ns == shape[i] and prodns == prods*shape[i]:\n prods *= shape[i]\n result.append(axes[i])\n i -= 1\n elif unknown:\n result.append(unknown)\n else:\n unknown = 'Q'\n result.append(unknown)\n\n return ''.join(reversed(result[lendiff:]))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nread data from sequence of TiffPage and stack them vertically.", "response": "def stack_pages(pages, out=None, maxworkers=None, **kwargs):\n \"\"\"Read data from sequence of TiffPage and stack them vertically.\n\n Additional parameters are passsed to the TiffPage.asarray function.\n\n \"\"\"\n npages = len(pages)\n if npages == 0:\n raise ValueError('no pages')\n\n if npages == 1:\n kwargs['maxworkers'] = maxworkers\n return pages[0].asarray(out=out, **kwargs)\n\n page0 = next(p for p in pages if p is not None).keyframe\n page0.asarray(validate=None) # ThreadPoolExecutor swallows exceptions\n shape = (npages,) + page0.shape\n dtype = page0.dtype\n out = create_output(out, shape, dtype)\n\n if maxworkers is None:\n if page0.compression > 1:\n if page0.is_tiled:\n maxworkers = 1\n kwargs['maxworkers'] = 0\n else:\n maxworkers = 0\n else:\n maxworkers = 1\n if maxworkers == 0:\n import multiprocessing # noqa: delay import\n maxworkers = multiprocessing.cpu_count() // 2\n if maxworkers > 1:\n kwargs['maxworkers'] = 1\n\n page0.parent.filehandle.lock = maxworkers > 1\n\n filecache = OpenFileCache(size=max(4, maxworkers),\n lock=page0.parent.filehandle.lock)\n\n def func(page, index, out=out, filecache=filecache, kwargs=kwargs):\n \"\"\"Read, decode, and copy page data.\"\"\"\n if page is not None:\n filecache.open(page.parent.filehandle)\n out[index] = page.asarray(lock=filecache.lock, reopen=False,\n validate=False, **kwargs)\n filecache.close(page.parent.filehandle)\n\n if maxworkers < 2:\n for i, page in enumerate(pages):\n func(page, i)\n else:\n # TODO: add exception handling\n with ThreadPoolExecutor(maxworkers) as executor:\n executor.map(func, pages, range(npages))\n\n filecache.clear()\n page0.parent.filehandle.lock = None\n\n return out"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning cleaned offsets and byte counts.", "response": "def clean_offsetscounts(offsets, counts):\n \"\"\"Return cleaned offsets and byte counts.\n\n Remove zero offsets and counts.\n Use to sanitize StripOffsets and StripByteCounts tag values.\n\n \"\"\"\n # TODO: cythonize this\n offsets = list(offsets)\n counts = list(counts)\n size = len(offsets)\n if size != len(counts):\n raise ValueError('StripOffsets and StripByteCounts mismatch')\n j = 0\n for i, (o, b) in enumerate(zip(offsets, counts)):\n if b > 0:\n if o > 0:\n if i > j:\n offsets[j] = o\n counts[j] = b\n j += 1\n continue\n raise ValueError('invalid offset')\n log.warning('clean_offsetscounts: empty bytecount')\n if size == len(offsets):\n return offsets, counts\n if j == 0:\n return [offsets[0]], [counts[0]]\n return offsets[:j], counts[:j]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef buffered_read(fh, lock, offsets, bytecounts, buffersize=None):\n if buffersize is None:\n buffersize = 2**26\n length = len(offsets)\n i = 0\n while i < length:\n data = []\n with lock:\n size = 0\n while size < buffersize and i < length:\n fh.seek(offsets[i])\n bytecount = bytecounts[i]\n data.append(fh.read(bytecount))\n # buffer = bytearray(bytecount)\n # n = fh.readinto(buffer)\n # data.append(buffer[:n])\n size += bytecount\n i += 1\n for segment in data:\n yield segment", "response": "Return iterator over segments read from file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncreate an array of numpy arrays of compatible shape and dtype.", "response": "def create_output(out, shape, dtype, mode='w+', suffix=None):\n \"\"\"Return numpy array where image data of shape and dtype can be copied.\n\n The 'out' parameter may have the following values or types:\n\n None\n An empty array of shape and dtype is created and returned.\n numpy.ndarray\n An existing writable array of compatible dtype and shape. A view of\n the same array is returned after verification.\n 'memmap' or 'memmap:tempdir'\n A memory-map to an array stored in a temporary binary file on disk\n is created and returned.\n str or open file\n The file name or file object used to create a memory-map to an array\n stored in a binary file on disk. The created memory-mapped array is\n returned.\n\n \"\"\"\n if out is None:\n return numpy.zeros(shape, dtype)\n if isinstance(out, str) and out[:6] == 'memmap':\n import tempfile # noqa: delay import\n tempdir = out[7:] if len(out) > 7 else None\n if suffix is None:\n suffix = '.memmap'\n with tempfile.NamedTemporaryFile(dir=tempdir, suffix=suffix) as fh:\n return numpy.memmap(fh, shape=shape, dtype=dtype, mode=mode)\n if isinstance(out, numpy.ndarray):\n if product(shape) != product(out.shape):\n raise ValueError('incompatible output shape')\n if not numpy.can_cast(dtype, out.dtype):\n raise ValueError('incompatible output dtype')\n return out.reshape(shape)\n if isinstance(out, pathlib.Path):\n out = str(out)\n return numpy.memmap(out, shape=shape, dtype=dtype, mode=mode)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning Python object from Matlab string representation.", "response": "def matlabstr2py(string):\n \"\"\"Return Python object from Matlab string representation.\n\n Return str, bool, int, float, list (Matlab arrays or cells), or\n dict (Matlab structures) types.\n\n Use to access ScanImage metadata.\n\n >>> matlabstr2py('1')\n 1\n >>> matlabstr2py(\"['x y z' true false; 1 2.0 -3e4; NaN Inf @class]\")\n [['x y z', True, False], [1, 2.0, -30000.0], [nan, inf, '@class']]\n >>> d = matlabstr2py(\"SI.hChannels.channelType = {'stripe' 'stripe'}\\\\n\"\n ... \"SI.hChannels.channelsActive = 2\")\n >>> d['SI.hChannels.channelType']\n ['stripe', 'stripe']\n\n \"\"\"\n # TODO: handle invalid input\n # TODO: review unboxing of multidimensional arrays\n\n def lex(s):\n # return sequence of tokens from matlab string representation\n tokens = ['[']\n while True:\n t, i = next_token(s)\n if t is None:\n break\n if t == ';':\n tokens.extend((']', '['))\n elif t == '[':\n tokens.extend(('[', '['))\n elif t == ']':\n tokens.extend((']', ']'))\n else:\n tokens.append(t)\n s = s[i:]\n tokens.append(']')\n return tokens\n\n def next_token(s):\n # return next token in matlab string\n length = len(s)\n if length == 0:\n return None, 0\n i = 0\n while i < length and s[i] == ' ':\n i += 1\n if i == length:\n return None, i\n if s[i] in '{[;]}':\n return s[i], i + 1\n if s[i] == \"'\":\n j = i + 1\n while j < length and s[j] != \"'\":\n j += 1\n return s[i: j+1], j + 1\n if s[i] == '<':\n j = i + 1\n while j < length and s[j] != '>':\n j += 1\n return s[i: j+1], j + 1\n j = i\n while j < length and not s[j] in ' {[;]}':\n j += 1\n return s[i:j], j\n\n def value(s, fail=False):\n # return Python value of token\n s = s.strip()\n if not s:\n return s\n if len(s) == 1:\n try:\n return int(s)\n except Exception:\n if fail:\n raise ValueError()\n return s\n if s[0] == \"'\":\n if fail and s[-1] != \"'\" or \"'\" in s[1:-1]:\n raise ValueError()\n return s[1:-1]\n if s[0] == '<':\n if fail and s[-1] != '>' or '<' in s[1:-1]:\n raise ValueError()\n return s\n if fail and any(i in s for i in \" ';[]{}\"):\n raise ValueError()\n if s[0] == '@':\n return s\n if s in ('true', 'True'):\n return True\n if s in ('false', 'False'):\n return False\n if s[:6] == 'zeros(':\n return numpy.zeros([int(i) for i in s[6:-1].split(',')]).tolist()\n if s[:5] == 'ones(':\n return numpy.ones([int(i) for i in s[5:-1].split(',')]).tolist()\n if '.' in s or 'e' in s:\n try:\n return float(s)\n except Exception:\n pass\n try:\n return int(s)\n except Exception:\n pass\n try:\n return float(s) # nan, inf\n except Exception:\n if fail:\n raise ValueError()\n return s\n\n def parse(s):\n # return Python value from string representation of Matlab value\n s = s.strip()\n try:\n return value(s, fail=True)\n except ValueError:\n pass\n result = add2 = []\n levels = [add2]\n for t in lex(s):\n if t in '[{':\n add2 = []\n levels.append(add2)\n elif t in ']}':\n x = levels.pop()\n if len(x) == 1 and isinstance(x[0], (list, str)):\n x = x[0]\n add2 = levels[-1]\n add2.append(x)\n else:\n add2.append(value(t))\n if len(result) == 1 and isinstance(result[0], (list, str)):\n result = result[0]\n return result\n\n if '\\r' in string or '\\n' in string:\n # structure\n d = {}\n for line in string.splitlines():\n line = line.strip()\n if not line or line[0] == '%':\n continue\n k, v = line.split('=', 1)\n k = k.strip()\n if any(c in k for c in \" ';[]{}<>\"):\n continue\n d[k] = parse(v)\n return d\n return parse(string)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef stripascii(string):\n # TODO: pythonize this\n i = len(string)\n while i:\n i -= 1\n if 8 < byte2int(string[i]) < 127:\n break\n else:\n i = -1\n return string[:i+1]", "response": "Return string truncated at last byte that is 7 - bit ASCII."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef asbool(value, true=(b'true', u'true'), false=(b'false', u'false')):\n value = value.strip().lower()\n if value in true: # might raise UnicodeWarning/BytesWarning\n return True\n if value in false:\n return False\n raise TypeError()", "response": "Return string as bool if possible else raise TypeError."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns argument as one of types if possible.", "response": "def astype(value, types=None):\n \"\"\"Return argument as one of types if possible.\n\n >>> astype('42')\n 42\n >>> astype('3.14')\n 3.14\n >>> astype('True')\n True\n >>> astype(b'Neee-Wom')\n 'Neee-Wom'\n\n \"\"\"\n if types is None:\n types = int, float, asbool, bytes2str\n for typ in types:\n try:\n return typ(value)\n except (ValueError, AttributeError, TypeError, UnicodeEncodeError):\n pass\n return value"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning file size as string from byte size.", "response": "def format_size(size, threshold=1536):\n \"\"\"Return file size as string from byte size.\n\n >>> format_size(1234)\n '1234 B'\n >>> format_size(12345678901)\n '11.50 GiB'\n\n \"\"\"\n if size < threshold:\n return \"%i B\" % size\n for unit in ('KiB', 'MiB', 'GiB', 'TiB', 'PiB'):\n size /= 1024.0\n if size < threshold:\n return \"%.2f %s\" % (size, unit)\n return 'ginormous'"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a list of strings in natural order.", "response": "def natural_sorted(iterable):\n \"\"\"Return human sorted list of strings.\n\n E.g. for sorting file names.\n\n >>> natural_sorted(['f1', 'f2', 'f10'])\n ['f1', 'f2', 'f10']\n\n \"\"\"\n def sortkey(x):\n return [(int(c) if c.isdigit() else c) for c in re.split(numbers, x)]\n\n numbers = re.compile(r'(\\d+)')\n return sorted(iterable, key=sortkey)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning datetime object from timestamp in Excel serial format.", "response": "def excel_datetime(timestamp, epoch=None):\n \"\"\"Return datetime object from timestamp in Excel serial format.\n\n Convert LSM time stamps.\n\n >>> excel_datetime(40237.029999999795)\n datetime.datetime(2010, 2, 28, 0, 43, 11, 999982)\n\n \"\"\"\n if epoch is None:\n epoch = datetime.datetime.fromordinal(693594)\n return epoch + datetime.timedelta(timestamp)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef julian_datetime(julianday, milisecond=0):\n if julianday <= 1721423:\n # no datetime before year 1\n return None\n\n a = julianday + 1\n if a > 2299160:\n alpha = math.trunc((a - 1867216.25) / 36524.25)\n a += 1 + alpha - alpha // 4\n b = a + (1524 if a > 1721423 else 1158)\n c = math.trunc((b - 122.1) / 365.25)\n d = math.trunc(365.25 * c)\n e = math.trunc((b - d) / 30.6001)\n\n day = b - d - math.trunc(30.6001 * e)\n month = e - (1 if e < 13.5 else 13)\n year = c - (4716 if month > 2.5 else 4715)\n\n hour, milisecond = divmod(milisecond, 1000 * 60 * 60)\n minute, milisecond = divmod(milisecond, 1000 * 60)\n second, milisecond = divmod(milisecond, 1000)\n\n return datetime.datetime(year, month, day,\n hour, minute, second, milisecond)", "response": "Convert Julian dates according to MetaMorph."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef byteorder_isnative(byteorder):\n if byteorder in ('=', sys.byteorder):\n return True\n keys = {'big': '>', 'little': '<'}\n return keys.get(byteorder, byteorder) == keys[sys.byteorder]", "response": "Return True if byteorder matches the system s byteorder."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning numpy. recarray as dict.", "response": "def recarray2dict(recarray):\n \"\"\"Return numpy.recarray as dict.\"\"\"\n # TODO: subarrays\n result = {}\n for descr, value in zip(recarray.dtype.descr, recarray):\n name, dtype = descr[:2]\n if dtype[1] == 'S':\n value = bytes2str(stripnull(value))\n elif value.ndim < 2:\n value = value.tolist()\n result[name] = value\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef xml2dict(xml, sanitize=True, prefix=None):\n from xml.etree import cElementTree as etree # delayed import\n\n at = tx = ''\n if prefix:\n at, tx = prefix\n\n def astype(value):\n # return value as int, float, bool, or str\n for t in (int, float, asbool):\n try:\n return t(value)\n except Exception:\n pass\n return value\n\n def etree2dict(t):\n # adapted from https://stackoverflow.com/a/10077069/453463\n key = t.tag\n if sanitize:\n key = key.rsplit('}', 1)[-1]\n d = {key: {} if t.attrib else None}\n children = list(t)\n if children:\n dd = collections.defaultdict(list)\n for dc in map(etree2dict, children):\n for k, v in dc.items():\n dd[k].append(astype(v))\n d = {key: {k: astype(v[0]) if len(v) == 1 else astype(v)\n for k, v in dd.items()}}\n if t.attrib:\n d[key].update((at + k, astype(v)) for k, v in t.attrib.items())\n if t.text:\n text = t.text.strip()\n if children or t.attrib:\n if text:\n d[key][tx + 'value'] = astype(text)\n else:\n d[key] = astype(text)\n return d\n\n return etree2dict(etree.fromstring(xml))", "response": "Return XML as dict."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef hexdump(bytestr, width=75, height=24, snipat=-2, modulo=2, ellipsis=None):\n size = len(bytestr)\n if size < 1 or width < 2 or height < 1:\n return ''\n if height == 1:\n addr = b''\n bytesperline = min(modulo * (((width - len(addr)) // 4) // modulo),\n size)\n if bytesperline < 1:\n return ''\n nlines = 1\n else:\n addr = b'%%0%ix: ' % len(b'%x' % size)\n bytesperline = min(modulo * (((width - len(addr % 1)) // 4) // modulo),\n size)\n if bytesperline < 1:\n return ''\n width = 3*bytesperline + len(addr % 1)\n nlines = (size - 1) // bytesperline + 1\n\n if snipat is None or snipat == 1:\n snipat = height\n elif 0 < abs(snipat) < 1:\n snipat = int(math.floor(height * snipat))\n if snipat < 0:\n snipat += height\n\n if height == 1 or nlines == 1:\n blocks = [(0, bytestr[:bytesperline])]\n addr = b''\n height = 1\n width = 3 * bytesperline\n elif height is None or nlines <= height:\n blocks = [(0, bytestr)]\n elif snipat <= 0:\n start = bytesperline * (nlines - height)\n blocks = [(start, bytestr[start:])] # (start, None)\n elif snipat >= height or height < 3:\n end = bytesperline * height\n blocks = [(0, bytestr[:end])] # (end, None)\n else:\n end1 = bytesperline * snipat\n end2 = bytesperline * (height - snipat - 1)\n blocks = [(0, bytestr[:end1]),\n (size-end1-end2, None),\n (size-end2, bytestr[size-end2:])]\n\n ellipsis = b'...' if ellipsis is None else str2bytes(ellipsis)\n result = []\n for start, bytestr in blocks:\n if bytestr is None:\n result.append(ellipsis) # 'skip %i bytes' % start)\n continue\n hexstr = binascii.hexlify(bytestr)\n strstr = re.sub(br'[^\\x20-\\x7f]', b'.', bytestr)\n for i in range(0, len(bytestr), bytesperline):\n h = hexstr[2*i:2*i+bytesperline*2]\n r = (addr % (i + start)) if height > 1 else addr\n r += b' '.join(h[i:i+2] for i in range(0, 2*bytesperline, 2))\n r += b' ' * (width - len(r))\n r += strstr[i:i+bytesperline]\n result.append(r)\n result = b'\\n'.join(result)\n if sys.version_info[0] == 3:\n result = result.decode('ascii')\n return result", "response": "Return hexdump representation of byte string."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef isprintable(string):\n string = string.strip()\n if not string:\n return True\n if sys.version_info[0] == 3:\n try:\n return string.isprintable()\n except Exception:\n pass\n try:\n return string.decode('utf-8').isprintable()\n except Exception:\n pass\n else:\n if string.isalnum():\n return True\n printable = ('0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRST'\n 'UVWXYZ!\"#$%&\\'()*+,-./:;<=>?@[\\\\]^_`{|}~ \\t\\n\\r\\x0b\\x0c')\n return all(c in printable for c in string)", "response": "Return True if all characters in string are printable."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef clean_whitespace(string, compact=False):\n for a, b in (('\\r\\n', '\\n'), ('\\r', '\\n'), ('\\n\\n', '\\n'),\n ('\\t', ' '), (' ', ' ')):\n string = string.replace(a, b)\n if compact:\n for a, b in (('\\n', ' '), ('[ ', '['),\n (' ', ' '), (' ', ' '), (' ', ' ')):\n string = string.replace(a, b)\n return string.strip()", "response": "Return string with compressed whitespace."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns pretty formatted XML.", "response": "def pformat_xml(xml):\n \"\"\"Return pretty formatted XML.\"\"\"\n try:\n from lxml import etree # delayed import\n if not isinstance(xml, bytes):\n xml = xml.encode('utf-8')\n xml = etree.parse(io.BytesIO(xml))\n xml = etree.tostring(xml, pretty_print=True, xml_declaration=True,\n encoding=xml.docinfo.encoding)\n xml = bytes2str(xml)\n except Exception:\n if isinstance(xml, bytes):\n xml = bytes2str(xml)\n xml = xml.replace('><', '>\\n<')\n return xml.replace(' ', ' ').replace('\\t', ' ')"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef pformat(arg, width=79, height=24, compact=True):\n if height is None or height < 1:\n height = 1024\n if width is None or width < 1:\n width = 256\n\n npopt = numpy.get_printoptions()\n numpy.set_printoptions(threshold=100, linewidth=width)\n\n if isinstance(arg, basestring):\n if arg[:5].lower() in ('<?xml', b'<?xml'):\n if isinstance(arg, bytes):\n arg = bytes2str(arg)\n if height == 1:\n arg = arg[:4*width]\n else:\n arg = pformat_xml(arg)\n elif isinstance(arg, bytes):\n if isprintable(arg):\n arg = bytes2str(arg)\n arg = clean_whitespace(arg)\n else:\n numpy.set_printoptions(**npopt)\n return hexdump(arg, width=width, height=height, modulo=1)\n arg = arg.rstrip()\n elif isinstance(arg, numpy.record):\n arg = arg.pprint()\n else:\n import pprint # delayed import\n compact = {} if sys.version_info[0] == 2 else dict(compact=compact)\n arg = pprint.pformat(arg, width=width, **compact)\n\n numpy.set_printoptions(**npopt)\n\n if height == 1:\n arg = clean_whitespace(arg, compact=True)\n return arg[:width]\n\n argl = list(arg.splitlines())\n if len(argl) > height:\n arg = '\\n'.join(argl[:height//2] + ['...'] + argl[-height//2:])\n return arg", "response": "Return pretty formatted representation of object as string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning string cut to specified length.", "response": "def snipstr(string, width=79, snipat=None, ellipsis='...'):\n \"\"\"Return string cut to specified length.\n\n >>> snipstr('abcdefghijklmnop', 8)\n 'abc...op'\n\n \"\"\"\n if snipat is None:\n snipat = 0.5\n if ellipsis is None:\n if isinstance(string, bytes):\n ellipsis = b'...'\n else:\n ellipsis = u'\\u2026' # does not print on win-py3.5\n esize = len(ellipsis)\n\n splitlines = string.splitlines()\n # TODO: finish and test multiline snip\n\n result = []\n for line in splitlines:\n if line is None:\n result.append(ellipsis)\n continue\n linelen = len(line)\n if linelen <= width:\n result.append(string)\n continue\n\n split = snipat\n if split is None or split == 1:\n split = linelen\n elif 0 < abs(split) < 1:\n split = int(math.floor(linelen * split))\n if split < 0:\n split += linelen\n if split < 0:\n split = 0\n\n if esize == 0 or width < esize + 1:\n if split <= 0:\n result.append(string[-width:])\n else:\n result.append(string[:width])\n elif split <= 0:\n result.append(ellipsis + string[esize-width:])\n elif split >= linelen or width < esize + 4:\n result.append(string[:width-esize] + ellipsis)\n else:\n splitlen = linelen - width + esize\n end1 = split - splitlen // 2\n end2 = end1 + splitlen\n result.append(string[:end1] + ellipsis + string[end2:])\n\n if isinstance(string, bytes):\n return b'\\n'.join(result)\n return '\\n'.join(result)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef enumarg(enum, arg):\n try:\n return enum(arg)\n except Exception:\n try:\n return enum[arg.upper()]\n except Exception:\n raise ValueError('invalid argument %s' % arg)", "response": "Return enum member from its name or value."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef parse_kwargs(kwargs, *keys, **keyvalues):\n result = {}\n for key in keys:\n if key in kwargs:\n result[key] = kwargs[key]\n del kwargs[key]\n for key, value in keyvalues.items():\n if key in kwargs:\n result[key] = kwargs[key]\n del kwargs[key]\n else:\n result[key] = value\n return result", "response": "Parse kwargs and return a dict with keys from keys and values from keyvals."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nupdate dict with keys and values if keys do not already exist.", "response": "def update_kwargs(kwargs, **keyvalues):\n \"\"\"Update dict with keys and values if keys do not already exist.\n\n >>> kwargs = {'one': 1, }\n >>> update_kwargs(kwargs, one=None, two=2)\n >>> kwargs == {'one': 1, 'two': 2}\n True\n\n \"\"\"\n for key, value in keyvalues.items():\n if key not in kwargs:\n kwargs[key] = value"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef validate_jhove(filename, jhove=None, ignore=None):\n import subprocess # noqa: delayed import\n if ignore is None:\n ignore = ['More than 50 IFDs']\n if jhove is None:\n jhove = 'jhove'\n out = subprocess.check_output([jhove, filename, '-m', 'TIFF-hul'])\n if b'ErrorMessage: ' in out:\n for line in out.splitlines():\n line = line.strip()\n if line.startswith(b'ErrorMessage: '):\n error = line[14:].decode('utf8')\n for i in ignore:\n if i in error:\n break\n else:\n raise ValueError(error)\n break", "response": "Validate TIFF file using JHOVE."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef lsm2bin(lsmfile, binfile=None, tile=None, verbose=True):\n verbose = print_ if verbose else nullfunc\n\n if tile is None:\n tile = (256, 256)\n\n if binfile is None:\n binfile = lsmfile\n elif binfile.lower() == 'none':\n binfile = None\n if binfile:\n binfile += '_(z%ic%iy%ix%i)_m%%ip%%it%%03iy%%ix%%i.bin'\n\n verbose('\\nOpening LSM file... ', end='', flush=True)\n start_time = time.time()\n\n with TiffFile(lsmfile) as lsm:\n if not lsm.is_lsm:\n verbose('\\n', lsm, flush=True)\n raise ValueError('not a LSM file')\n series = lsm.series[0] # first series contains the image data\n shape = series.shape\n axes = series.axes\n dtype = series.dtype\n size = product(shape) * dtype.itemsize\n\n verbose('%.3f s' % (time.time() - start_time))\n # verbose(lsm, flush=True)\n verbose('Image\\n axes: %s\\n shape: %s\\n dtype: %s\\n size: %s'\n % (axes, shape, dtype, format_size(size)), flush=True)\n if not series.axes.endswith('TZCYX'):\n raise ValueError('not a *TZCYX LSM file')\n\n verbose('Copying image from LSM to BIN files', end='', flush=True)\n start_time = time.time()\n tiles = shape[-2] // tile[-2], shape[-1] // tile[-1]\n if binfile:\n binfile = binfile % (shape[-4], shape[-3], tile[0], tile[1])\n shape = (1,) * (7-len(shape)) + shape\n # cache for ZCYX stacks and output files\n data = numpy.empty(shape[3:], dtype=dtype)\n out = numpy.empty((shape[-4], shape[-3], tile[0], tile[1]),\n dtype=dtype)\n # iterate over Tiff pages containing data\n pages = iter(series.pages)\n for m in range(shape[0]): # mosaic axis\n for p in range(shape[1]): # position axis\n for t in range(shape[2]): # time axis\n for z in range(shape[3]): # z slices\n data[z] = next(pages).asarray()\n for y in range(tiles[0]): # tile y\n for x in range(tiles[1]): # tile x\n out[:] = data[...,\n y*tile[0]:(y+1)*tile[0],\n x*tile[1]:(x+1)*tile[1]]\n if binfile:\n out.tofile(binfile % (m, p, t, y, x))\n verbose('.', end='', flush=True)\n verbose(' %.3f s' % (time.time() - start_time))", "response": "Convert a LSM file to a series of BIN files."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nplot n - dimensional images using matplotlib. pyplot.", "response": "def imshow(data, photometric=None, planarconfig=None, bitspersample=None,\n interpolation=None, cmap=None, vmin=None, vmax=None,\n figure=None, title=None, dpi=96, subplot=None, maxdim=None,\n **kwargs):\n \"\"\"Plot n-dimensional images using matplotlib.pyplot.\n\n Return figure, subplot and plot axis.\n Requires pyplot already imported C{from matplotlib import pyplot}.\n\n Parameters\n ----------\n data : nd array\n The image data.\n photometric : {'MINISWHITE', 'MINISBLACK', 'RGB', or 'PALETTE'}\n The color space of the image data.\n planarconfig : {'CONTIG' or 'SEPARATE'}\n Defines how components of each pixel are stored.\n bitspersample : int\n Number of bits per channel in integer RGB images.\n interpolation : str\n The image interpolation method used in matplotlib.imshow. By default,\n 'nearest' will be used for image dimensions <= 512, else 'bilinear'.\n cmap : str or matplotlib.colors.Colormap\n The colormap maps non-RGBA scalar data to colors.\n vmin, vmax : scalar\n Data range covered by the colormap. By default, the complete\n range of the data is covered.\n figure : matplotlib.figure.Figure\n Matplotlib figure to use for plotting.\n title : str\n Window and subplot title.\n subplot : int\n A matplotlib.pyplot.subplot axis.\n maxdim : int\n Maximum image width and length.\n kwargs : dict\n Additional arguments for matplotlib.pyplot.imshow.\n\n \"\"\"\n # TODO: rewrite detection of isrgb, iscontig\n # TODO: use planarconfig\n if photometric is None:\n photometric = 'RGB'\n if maxdim is None:\n maxdim = 2**16\n isrgb = photometric in ('RGB', 'YCBCR') # 'PALETTE', 'YCBCR'\n\n if data.dtype == 'float16':\n data = data.astype('float32')\n\n if data.dtype.kind == 'b':\n isrgb = False\n\n if isrgb and not (data.shape[-1] in (3, 4) or (\n data.ndim > 2 and data.shape[-3] in (3, 4))):\n isrgb = False\n photometric = 'MINISBLACK'\n\n data = data.squeeze()\n if photometric in ('MINISWHITE', 'MINISBLACK', None):\n data = reshape_nd(data, 2)\n else:\n data = reshape_nd(data, 3)\n\n dims = data.ndim\n if dims < 2:\n raise ValueError('not an image')\n if dims == 2:\n dims = 0\n isrgb = False\n else:\n if isrgb and data.shape[-3] in (3, 4):\n data = numpy.swapaxes(data, -3, -2)\n data = numpy.swapaxes(data, -2, -1)\n elif not isrgb and (data.shape[-1] < data.shape[-2] // 8 and\n data.shape[-1] < data.shape[-3] // 8 and\n data.shape[-1] < 5):\n data = numpy.swapaxes(data, -3, -1)\n data = numpy.swapaxes(data, -2, -1)\n isrgb = isrgb and data.shape[-1] in (3, 4)\n dims -= 3 if isrgb else 2\n\n if interpolation is None:\n threshold = 512\n elif isinstance(interpolation, int):\n threshold = interpolation\n else:\n threshold = 0\n\n if isrgb:\n data = data[..., :maxdim, :maxdim, :maxdim]\n if threshold:\n if (data.shape[-2] > threshold or data.shape[-3] > threshold):\n interpolation = 'bilinear'\n else:\n interpolation = 'nearest'\n else:\n data = data[..., :maxdim, :maxdim]\n if threshold:\n if (data.shape[-1] > threshold or data.shape[-2] > threshold):\n interpolation = 'bilinear'\n else:\n interpolation = 'nearest'\n\n if photometric == 'PALETTE' and isrgb:\n datamax = data.max()\n if datamax > 255:\n data = data >> 8 # possible precision loss\n data = data.astype('B')\n elif data.dtype.kind in 'ui':\n if not (isrgb and data.dtype.itemsize <= 1) or bitspersample is None:\n try:\n bitspersample = int(math.ceil(math.log(data.max(), 2)))\n except Exception:\n bitspersample = data.dtype.itemsize * 8\n elif not isinstance(bitspersample, inttypes):\n # bitspersample can be tuple, e.g. (5, 6, 5)\n bitspersample = data.dtype.itemsize * 8\n datamax = 2**bitspersample\n if isrgb:\n if bitspersample < 8:\n data = data << (8 - bitspersample)\n elif bitspersample > 8:\n data = data >> (bitspersample - 8) # precision loss\n data = data.astype('B')\n elif data.dtype.kind == 'f':\n datamax = data.max()\n if isrgb and datamax > 1.0:\n if data.dtype.char == 'd':\n data = data.astype('f')\n data /= datamax\n else:\n data = data / datamax\n elif data.dtype.kind == 'b':\n datamax = 1\n elif data.dtype.kind == 'c':\n data = numpy.absolute(data)\n datamax = data.max()\n\n if isrgb:\n vmin = 0\n else:\n if vmax is None:\n vmax = datamax\n if vmin is None:\n if data.dtype.kind == 'i':\n dtmin = numpy.iinfo(data.dtype).min\n vmin = numpy.min(data)\n if vmin == dtmin:\n vmin = numpy.min(data[data > dtmin])\n elif data.dtype.kind == 'f':\n dtmin = numpy.finfo(data.dtype).min\n vmin = numpy.min(data)\n if vmin == dtmin:\n vmin = numpy.min(data[data > dtmin])\n else:\n vmin = 0\n\n pyplot = sys.modules['matplotlib.pyplot']\n\n if figure is None:\n pyplot.rc('font', family='sans-serif', weight='normal', size=8)\n figure = pyplot.figure(dpi=dpi, figsize=(10.3, 6.3), frameon=True,\n facecolor='1.0', edgecolor='w')\n try:\n figure.canvas.manager.window.title(title)\n except Exception:\n pass\n size = len(title.splitlines()) if title else 1\n pyplot.subplots_adjust(bottom=0.03*(dims+2), top=0.98-size*0.03,\n left=0.1, right=0.95, hspace=0.05, wspace=0.0)\n if subplot is None:\n subplot = 111\n subplot = pyplot.subplot(subplot)\n subplot.set_facecolor((0, 0, 0))\n\n if title:\n try:\n title = unicode(title, 'Windows-1252')\n except TypeError:\n pass\n pyplot.title(title, size=11)\n\n if cmap is None:\n if data.dtype.char == '?':\n cmap = 'gray'\n elif data.dtype.kind in 'buf' or vmin == 0:\n cmap = 'viridis'\n else:\n cmap = 'coolwarm'\n if photometric == 'MINISWHITE':\n cmap += '_r'\n\n image = pyplot.imshow(numpy.atleast_2d(data[(0,) * dims].squeeze()),\n vmin=vmin, vmax=vmax, cmap=cmap,\n interpolation=interpolation, **kwargs)\n\n if not isrgb:\n pyplot.colorbar() # panchor=(0.55, 0.5), fraction=0.05\n\n def format_coord(x, y):\n # callback function to format coordinate display in toolbar\n x = int(x + 0.5)\n y = int(y + 0.5)\n try:\n if dims:\n return '%s @ %s [%4i, %4i]' % (\n curaxdat[1][y, x], current, y, x)\n return '%s @ [%4i, %4i]' % (data[y, x], y, x)\n except IndexError:\n return ''\n\n def none(event):\n return ''\n\n subplot.format_coord = format_coord\n image.get_cursor_data = none\n image.format_cursor_data = none\n\n if dims:\n current = list((0,) * dims)\n curaxdat = [0, data[tuple(current)].squeeze()]\n sliders = [pyplot.Slider(\n pyplot.axes([0.125, 0.03*(axis+1), 0.725, 0.025]),\n 'Dimension %i' % axis, 0, data.shape[axis]-1, 0, facecolor='0.5',\n valfmt='%%.0f [%i]' % data.shape[axis]) for axis in range(dims)]\n for slider in sliders:\n slider.drawon = False\n\n def set_image(current, sliders=sliders, data=data):\n # change image and redraw canvas\n curaxdat[1] = data[tuple(current)].squeeze()\n image.set_data(curaxdat[1])\n for ctrl, index in zip(sliders, current):\n ctrl.eventson = False\n ctrl.set_val(index)\n ctrl.eventson = True\n figure.canvas.draw()\n\n def on_changed(index, axis, data=data, current=current):\n # callback function for slider change event\n index = int(round(index))\n curaxdat[0] = axis\n if index == current[axis]:\n return\n if index >= data.shape[axis]:\n index = 0\n elif index < 0:\n index = data.shape[axis] - 1\n current[axis] = index\n set_image(current)\n\n def on_keypressed(event, data=data, current=current):\n # callback function for key press event\n key = event.key\n axis = curaxdat[0]\n if str(key) in '0123456789':\n on_changed(key, axis)\n elif key == 'right':\n on_changed(current[axis] + 1, axis)\n elif key == 'left':\n on_changed(current[axis] - 1, axis)\n elif key == 'up':\n curaxdat[0] = 0 if axis == len(data.shape)-1 else axis + 1\n elif key == 'down':\n curaxdat[0] = len(data.shape)-1 if axis == 0 else axis - 1\n elif key == 'end':\n on_changed(data.shape[axis] - 1, axis)\n elif key == 'home':\n on_changed(0, axis)\n\n figure.canvas.mpl_connect('key_press_event', on_keypressed)\n for axis, ctrl in enumerate(sliders):\n ctrl.on_changed(lambda k, a=axis: on_changed(k, a))\n\n return figure, subplot, image"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef askopenfilename(**kwargs):\n try:\n from Tkinter import Tk\n import tkFileDialog as filedialog\n except ImportError:\n from tkinter import Tk, filedialog\n root = Tk()\n root.withdraw()\n root.update()\n filenames = filedialog.askopenfilename(**kwargs)\n root.destroy()\n return filenames", "response": "Return file name s from Tkinter s file open dialog."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nwriting numpy array and tags to TIFF file.", "response": "def save(self, data=None, shape=None, dtype=None, returnoffset=False,\n photometric=None, planarconfig=None, extrasamples=None, tile=None,\n contiguous=True, align=16, truncate=False, compress=0,\n rowsperstrip=None, predictor=False, colormap=None,\n description=None, datetime=None, resolution=None, subfiletype=0,\n software='tifffile.py', metadata={}, ijmetadata=None,\n extratags=()):\n \"\"\"Write numpy array and tags to TIFF file.\n\n The data shape's last dimensions are assumed to be image depth,\n height (length), width, and samples.\n If a colormap is provided, the data's dtype must be uint8 or uint16\n and the data values are indices into the last dimension of the\n colormap.\n If 'shape' and 'dtype' are specified, an empty array is saved.\n This option cannot be used with compression or multiple tiles.\n Image data are written uncompressed in one strip per plane by default.\n Dimensions larger than 2 to 4 (depending on photometric mode, planar\n configuration, and SGI mode) are flattened and saved as separate pages.\n The SampleFormat and BitsPerSample tags are derived from the data type.\n\n Parameters\n ----------\n data : numpy.ndarray or None\n Input image array.\n shape : tuple or None\n Shape of the empty array to save. Used only if 'data' is None.\n dtype : numpy.dtype or None\n Data-type of the empty array to save. Used only if 'data' is None.\n returnoffset : bool\n If True and the image data in the file is memory-mappable, return\n the offset and number of bytes of the image data in the file.\n photometric : {'MINISBLACK', 'MINISWHITE', 'RGB', 'PALETTE', 'CFA'}\n The color space of the image data.\n By default, this setting is inferred from the data shape and the\n value of colormap.\n For CFA images, DNG tags must be specified in 'extratags'.\n planarconfig : {'CONTIG', 'SEPARATE'}\n Specifies if samples are stored interleaved or in separate planes.\n By default, this setting is inferred from the data shape.\n If this parameter is set, extra samples are used to store grayscale\n images.\n 'CONTIG': last dimension contains samples.\n 'SEPARATE': third last dimension contains samples.\n extrasamples : tuple of {'UNSPECIFIED', 'ASSOCALPHA', 'UNASSALPHA'}\n Defines the interpretation of extra components in pixels.\n 'UNSPECIFIED': no transparency information (default).\n 'ASSOCALPHA': single, true transparency with pre-multiplied color.\n 'UNASSALPHA': independent transparency masks.\n tile : tuple of int\n The shape (depth, length, width) of image tiles to write.\n If None (default), image data are written in strips.\n The tile length and width must be a multiple of 16.\n If the tile depth is provided, the SGI ImageDepth and TileDepth\n tags are used to save volume data.\n Unless a single tile is used, tiles cannot be used to write\n contiguous files.\n Few software can read the SGI format, e.g. MeVisLab.\n contiguous : bool\n If True (default) and the data and parameters are compatible with\n previous ones, if any, the image data are stored contiguously after\n the previous one. In that case, 'photometric', 'planarconfig',\n 'rowsperstrip', are ignored. Metadata such as 'description',\n 'metadata', 'datetime', and 'extratags' are written to the first\n page of a contiguous series only.\n align : int\n Byte boundary on which to align the image data in the file.\n Default 16. Use mmap.ALLOCATIONGRANULARITY for memory-mapped data.\n Following contiguous writes are not aligned.\n truncate : bool\n If True, only write the first page including shape metadata if\n possible (uncompressed, contiguous, not tiled).\n Other TIFF readers will only be able to read part of the data.\n compress : int or str or (str, int)\n If 0 (default), data are written uncompressed.\n If 0-9, the level of ADOBE_DEFLATE compression.\n If a str, one of TIFF.COMPRESSION, e.g. 'LZMA' or 'ZSTD'.\n If a tuple, first item is one of TIFF.COMPRESSION and second item\n is compression level.\n Compression cannot be used to write contiguous files.\n rowsperstrip : int\n The number of rows per strip. By default strips will be ~64 KB\n if compression is enabled, else rowsperstrip is set to the image\n length. Bilevel images are always stored in one strip per plane.\n predictor : bool\n If True, apply horizontal differencing or floating-point predictor\n before compression.\n colormap : numpy.ndarray\n RGB color values for the corresponding data value.\n Must be of shape (3, 2**(data.itemsize*8)) and dtype uint16.\n description : str\n The subject of the image. Must be 7-bit ASCII. Cannot be used with\n the ImageJ format. Saved with the first page only.\n datetime : datetime, str, or bool\n Date and time of image creation in '%Y:%m:%d %H:%M:%S' format or\n datetime object. Else if True, the current date and time is used.\n Saved with the first page only.\n resolution : (float, float[, str]) or ((int, int), (int, int)[, str])\n X and Y resolutions in pixels per resolution unit as float or\n rational numbers. A third, optional parameter specifies the\n resolution unit, which must be None (default for ImageJ),\n 'INCH' (default), or 'CENTIMETER'.\n subfiletype : int\n Bitfield to indicate the kind of data. Set bit 0 if the image\n is a reduced-resolution version of another image. Set bit 1 if\n the image is part of a multi-page image. Set bit 2 if the image\n is transparency mask for another image (photometric must be\n MASK, SamplesPerPixel and BitsPerSample must be 1).\n software : str\n Name of the software used to create the file. Must be 7-bit ASCII.\n Saved with the first page only.\n metadata : dict\n Additional metadata to be saved along with shape information\n in JSON or ImageJ formats in an ImageDescription tag.\n If None, do not write a second ImageDescription tag.\n Strings must be 7-bit ASCII. Saved with the first page only.\n ijmetadata : dict\n Additional metadata to be saved in application specific\n IJMetadata and IJMetadataByteCounts tags. Refer to the\n imagej_metadata_tag function for valid keys and values.\n Saved with the first page only.\n extratags : sequence of tuples\n Additional tags as [(code, dtype, count, value, writeonce)].\n\n code : int\n The TIFF tag Id.\n dtype : str\n Data type of items in 'value' in Python struct format.\n One of B, s, H, I, 2I, b, h, i, 2i, f, d, Q, or q.\n count : int\n Number of data values. Not used for string or byte string\n values.\n value : sequence\n 'Count' values compatible with 'dtype'.\n Byte strings must contain count values of dtype packed as\n binary data.\n writeonce : bool\n If True, the tag is written to the first page only.\n\n \"\"\"\n # TODO: refactor this function\n fh = self._fh\n byteorder = self._byteorder\n\n if data is None:\n if compress:\n raise ValueError('cannot save compressed empty file')\n datashape = shape\n datadtype = numpy.dtype(dtype).newbyteorder(byteorder)\n datadtypechar = datadtype.char\n else:\n data = numpy.asarray(data, byteorder+data.dtype.char, 'C')\n if data.size == 0:\n raise ValueError('cannot save empty array')\n datashape = data.shape\n datadtype = data.dtype\n datadtypechar = data.dtype.char\n\n returnoffset = returnoffset and datadtype.isnative\n bilevel = datadtypechar == '?'\n if bilevel:\n index = -1 if datashape[-1] > 1 else -2\n datasize = product(datashape[:index])\n if datashape[index] % 8:\n datasize *= datashape[index] // 8 + 1\n else:\n datasize *= datashape[index] // 8\n else:\n datasize = product(datashape) * datadtype.itemsize\n\n # just append contiguous data if possible\n self._truncate = bool(truncate)\n if self._datashape:\n if (not contiguous\n or self._datashape[1:] != datashape\n or self._datadtype != datadtype\n or (compress and self._tags)\n or tile\n or not numpy.array_equal(colormap, self._colormap)):\n # incompatible shape, dtype, compression mode, or colormap\n self._write_remaining_pages()\n self._write_image_description()\n self._truncate = False\n self._descriptionoffset = 0\n self._descriptionlenoffset = 0\n self._datashape = None\n self._colormap = None\n if self._imagej:\n raise ValueError(\n 'ImageJ does not support non-contiguous data')\n else:\n # consecutive mode\n self._datashape = (self._datashape[0] + 1,) + datashape\n if not compress:\n # write contiguous data, write IFDs/tags later\n offset = fh.tell()\n if data is None:\n fh.write_empty(datasize)\n else:\n fh.write_array(data)\n if returnoffset:\n return offset, datasize\n return None\n\n input_shape = datashape\n tagnoformat = self._tagnoformat\n valueformat = self._valueformat\n offsetformat = self._offsetformat\n offsetsize = self._offsetsize\n tagsize = self._tagsize\n\n MINISBLACK = TIFF.PHOTOMETRIC.MINISBLACK\n MINISWHITE = TIFF.PHOTOMETRIC.MINISWHITE\n RGB = TIFF.PHOTOMETRIC.RGB\n CFA = TIFF.PHOTOMETRIC.CFA\n PALETTE = TIFF.PHOTOMETRIC.PALETTE\n CONTIG = TIFF.PLANARCONFIG.CONTIG\n SEPARATE = TIFF.PLANARCONFIG.SEPARATE\n\n # parse input\n if photometric is not None:\n photometric = enumarg(TIFF.PHOTOMETRIC, photometric)\n if planarconfig:\n planarconfig = enumarg(TIFF.PLANARCONFIG, planarconfig)\n if extrasamples is None:\n extrasamples_ = None\n else:\n extrasamples_ = tuple(enumarg(TIFF.EXTRASAMPLE, es)\n for es in sequence(extrasamples))\n if not compress:\n compress = False\n compresstag = 1\n # TODO: support predictors without compression\n predictor = False\n predictortag = 1\n else:\n if isinstance(compress, (tuple, list)):\n compress, compresslevel = compress\n elif isinstance(compress, int):\n compress, compresslevel = 'ADOBE_DEFLATE', int(compress)\n if not 0 <= compresslevel <= 9:\n raise ValueError('invalid compression level %s' % compress)\n else:\n compresslevel = None\n compress = compress.upper()\n compresstag = enumarg(TIFF.COMPRESSION, compress)\n\n if predictor:\n if datadtype.kind in 'iu':\n predictortag = 2\n predictor = TIFF.PREDICTORS[2]\n elif datadtype.kind == 'f':\n predictortag = 3\n predictor = TIFF.PREDICTORS[3]\n else:\n raise ValueError('cannot apply predictor to %s' % datadtype)\n\n # prepare ImageJ format\n if self._imagej:\n # if predictor or compress:\n # warnings.warn(\n # 'ImageJ cannot handle predictors or compression')\n if description:\n warnings.warn('not writing description to ImageJ file')\n description = None\n volume = False\n if datadtypechar not in 'BHhf':\n raise ValueError(\n 'ImageJ does not support data type %s' % datadtypechar)\n ijrgb = photometric == RGB if photometric else None\n if datadtypechar not in 'B':\n ijrgb = False\n ijshape = imagej_shape(datashape, ijrgb)\n if ijshape[-1] in (3, 4):\n photometric = RGB\n if datadtypechar not in 'B':\n raise ValueError('ImageJ does not support data type %s '\n 'for RGB' % datadtypechar)\n elif photometric is None:\n photometric = MINISBLACK\n planarconfig = None\n if planarconfig == SEPARATE:\n raise ValueError('ImageJ does not support planar images')\n planarconfig = CONTIG if ijrgb else None\n\n # define compress function\n if compress:\n compressor = TIFF.COMPESSORS[compresstag]\n if predictor:\n def compress(data, level=compresslevel):\n data = predictor(data, axis=-2)\n return compressor(data, level)\n else:\n def compress(data, level=compresslevel):\n return compressor(data, level)\n\n # verify colormap and indices\n if colormap is not None:\n if datadtypechar not in 'BH':\n raise ValueError('invalid data dtype for palette mode')\n colormap = numpy.asarray(colormap, dtype=byteorder+'H')\n if colormap.shape != (3, 2**(datadtype.itemsize * 8)):\n raise ValueError('invalid color map shape')\n self._colormap = colormap\n\n # verify tile shape\n if tile:\n tile = tuple(int(i) for i in tile[:3])\n volume = len(tile) == 3\n if (len(tile) < 2 or tile[-1] % 16 or tile[-2] % 16 or\n any(i < 1 for i in tile)):\n raise ValueError('invalid tile shape')\n else:\n tile = ()\n volume = False\n\n # normalize data shape to 5D or 6D, depending on volume:\n # (pages, planar_samples, [depth,] height, width, contig_samples)\n datashape = reshape_nd(datashape, 3 if photometric == RGB else 2)\n shape = datashape\n ndim = len(datashape)\n\n samplesperpixel = 1\n extrasamples = 0\n if volume and ndim < 3:\n volume = False\n if colormap is not None:\n photometric = PALETTE\n planarconfig = None\n if photometric is None:\n photometric = MINISBLACK\n if bilevel:\n photometric = MINISWHITE\n elif planarconfig == CONTIG:\n if ndim > 2 and shape[-1] in (3, 4):\n photometric = RGB\n elif planarconfig == SEPARATE:\n if volume and ndim > 3 and shape[-4] in (3, 4):\n photometric = RGB\n elif ndim > 2 and shape[-3] in (3, 4):\n photometric = RGB\n elif ndim > 2 and shape[-1] in (3, 4):\n photometric = RGB\n elif self._imagej:\n photometric = MINISBLACK\n elif volume and ndim > 3 and shape[-4] in (3, 4):\n photometric = RGB\n elif ndim > 2 and shape[-3] in (3, 4):\n photometric = RGB\n if planarconfig and len(shape) <= (3 if volume else 2):\n planarconfig = None\n if photometric not in (0, 1, 3, 4):\n photometric = MINISBLACK\n if photometric == RGB:\n if len(shape) < 3:\n raise ValueError('not a RGB(A) image')\n if len(shape) < 4:\n volume = False\n if planarconfig is None:\n if shape[-1] in (3, 4):\n planarconfig = CONTIG\n elif shape[-4 if volume else -3] in (3, 4):\n planarconfig = SEPARATE\n elif shape[-1] > shape[-4 if volume else -3]:\n planarconfig = SEPARATE\n else:\n planarconfig = CONTIG\n if planarconfig == CONTIG:\n datashape = (-1, 1) + shape[(-4 if volume else -3):]\n samplesperpixel = datashape[-1]\n else:\n datashape = (-1,) + shape[(-4 if volume else -3):] + (1,)\n samplesperpixel = datashape[1]\n if samplesperpixel > 3:\n extrasamples = samplesperpixel - 3\n elif photometric == CFA:\n if len(shape) != 2:\n raise ValueError('invalid CFA image')\n volume = False\n planarconfig = None\n datashape = (-1, 1) + shape[-2:] + (1,)\n if 50706 not in (et[0] for et in extratags):\n raise ValueError('must specify DNG tags for CFA image')\n elif planarconfig and len(shape) > (3 if volume else 2):\n if planarconfig == CONTIG:\n datashape = (-1, 1) + shape[(-4 if volume else -3):]\n samplesperpixel = datashape[-1]\n else:\n datashape = (-1,) + shape[(-4 if volume else -3):] + (1,)\n samplesperpixel = datashape[1]\n extrasamples = samplesperpixel - 1\n else:\n planarconfig = None\n while len(shape) > 2 and shape[-1] == 1:\n shape = shape[:-1] # remove trailing 1s\n if len(shape) < 3:\n volume = False\n if extrasamples_ is None:\n datashape = (-1, 1) + shape[(-3 if volume else -2):] + (1,)\n else:\n datashape = (-1, 1) + shape[(-4 if volume else -3):]\n samplesperpixel = datashape[-1]\n extrasamples = samplesperpixel - 1\n\n if subfiletype & 0b100:\n # FILETYPE_MASK\n if not (bilevel and samplesperpixel == 1 and\n photometric in (0, 1, 4)):\n raise ValueError('invalid SubfileType MASK')\n photometric = TIFF.PHOTOMETRIC.MASK\n\n # normalize shape to 6D\n assert len(datashape) in (5, 6)\n if len(datashape) == 5:\n datashape = datashape[:2] + (1,) + datashape[2:]\n if datashape[0] == -1:\n s0 = product(input_shape) // product(datashape[1:])\n datashape = (s0,) + datashape[1:]\n shape = datashape\n if data is not None:\n data = data.reshape(shape)\n\n if tile and not volume:\n tile = (1, tile[-2], tile[-1])\n\n if photometric == PALETTE:\n if (samplesperpixel != 1 or extrasamples or\n shape[1] != 1 or shape[-1] != 1):\n raise ValueError('invalid data shape for palette mode')\n\n if photometric == RGB and samplesperpixel == 2:\n raise ValueError('not a RGB image (samplesperpixel=2)')\n\n if bilevel:\n if compresstag not in (1, 32773):\n raise ValueError('cannot compress bilevel image')\n if tile:\n raise ValueError('cannot save tiled bilevel image')\n if photometric not in (0, 1, 4):\n raise ValueError('cannot save bilevel image as %s' %\n str(photometric))\n datashape = list(datashape)\n if datashape[-2] % 8:\n datashape[-2] = datashape[-2] // 8 + 1\n else:\n datashape[-2] = datashape[-2] // 8\n datashape = tuple(datashape)\n assert datasize == product(datashape)\n if data is not None:\n data = numpy.packbits(data, axis=-2)\n assert datashape[-2] == data.shape[-2]\n\n bytestr = bytes if sys.version[0] == '2' else (\n lambda x: bytes(x, 'ascii') if isinstance(x, str) else x)\n tags = [] # list of (code, ifdentry, ifdvalue, writeonce)\n\n strip_or_tile = 'Tile' if tile else 'Strip'\n tagbytecounts = TIFF.TAG_NAMES[strip_or_tile + 'ByteCounts']\n tagoffsets = TIFF.TAG_NAMES[strip_or_tile + 'Offsets']\n self._tagoffsets = tagoffsets\n\n def pack(fmt, *val):\n return struct.pack(byteorder+fmt, *val)\n\n def addtag(code, dtype, count, value, writeonce=False):\n # Compute ifdentry & ifdvalue bytes from code, dtype, count, value\n # Append (code, ifdentry, ifdvalue, writeonce) to tags list\n code = int(TIFF.TAG_NAMES.get(code, code))\n try:\n tifftype = TIFF.DATA_DTYPES[dtype]\n except KeyError:\n raise ValueError('unknown dtype %s' % dtype)\n rawcount = count\n\n if dtype == 's':\n # strings\n value = bytestr(value) + b'\\0'\n count = rawcount = len(value)\n rawcount = value.find(b'\\0\\0')\n if rawcount < 0:\n rawcount = count\n else:\n rawcount += 1 # length of string without buffer\n value = (value,)\n elif isinstance(value, bytes):\n # packed binary data\n dtsize = struct.calcsize(dtype)\n if len(value) % dtsize:\n raise ValueError('invalid packed binary data')\n count = len(value) // dtsize\n if len(dtype) > 1:\n count *= int(dtype[:-1])\n dtype = dtype[-1]\n ifdentry = [pack('HH', code, tifftype),\n pack(offsetformat, rawcount)]\n ifdvalue = None\n if struct.calcsize(dtype) * count <= offsetsize:\n # value(s) can be written directly\n if isinstance(value, bytes):\n ifdentry.append(pack(valueformat, value))\n elif count == 1:\n if isinstance(value, (tuple, list, numpy.ndarray)):\n value = value[0]\n ifdentry.append(pack(valueformat, pack(dtype, value)))\n else:\n ifdentry.append(pack(valueformat,\n pack(str(count)+dtype, *value)))\n else:\n # use offset to value(s)\n ifdentry.append(pack(offsetformat, 0))\n if isinstance(value, bytes):\n ifdvalue = value\n elif isinstance(value, numpy.ndarray):\n assert value.size == count\n assert value.dtype.char == dtype\n ifdvalue = value.tostring()\n elif isinstance(value, (tuple, list)):\n ifdvalue = pack(str(count)+dtype, *value)\n else:\n ifdvalue = pack(dtype, value)\n tags.append((code, b''.join(ifdentry), ifdvalue, writeonce))\n\n def rational(arg, max_denominator=1000000):\n \"\"\"\"Return nominator and denominator from float or two integers.\"\"\"\n from fractions import Fraction # delayed import\n try:\n f = Fraction.from_float(arg)\n except TypeError:\n f = Fraction(arg[0], arg[1])\n f = f.limit_denominator(max_denominator)\n return f.numerator, f.denominator\n\n if description:\n # user provided description\n addtag('ImageDescription', 's', 0, description, writeonce=True)\n\n # write shape and metadata to ImageDescription\n self._metadata = {} if not metadata else metadata.copy()\n if self._imagej:\n description = imagej_description(\n input_shape, shape[-1] in (3, 4), self._colormap is not None,\n **self._metadata)\n elif metadata or metadata == {}:\n if self._truncate:\n self._metadata.update(truncated=True)\n description = json_description(input_shape, **self._metadata)\n # elif metadata is None and self._truncate:\n # raise ValueError('cannot truncate without writing metadata')\n else:\n description = None\n if description:\n # add 64 bytes buffer\n # the image description might be updated later with the final shape\n description = str2bytes(description, 'ascii')\n description += b'\\0' * 64\n self._descriptionlen = len(description)\n addtag('ImageDescription', 's', 0, description, writeonce=True)\n\n if software:\n addtag('Software', 's', 0, software, writeonce=True)\n if datetime:\n if isinstance(datetime, str):\n if len(datetime) != 19 or datetime[16] != ':':\n raise ValueError('invalid datetime string')\n else:\n try:\n datetime = datetime.strftime('%Y:%m:%d %H:%M:%S')\n except AttributeError:\n datetime = self._now().strftime('%Y:%m:%d %H:%M:%S')\n addtag('DateTime', 's', 0, datetime, writeonce=True)\n addtag('Compression', 'H', 1, compresstag)\n if predictor:\n addtag('Predictor', 'H', 1, predictortag)\n addtag('ImageWidth', 'I', 1, shape[-2])\n addtag('ImageLength', 'I', 1, shape[-3])\n if tile:\n addtag('TileWidth', 'I', 1, tile[-1])\n addtag('TileLength', 'I', 1, tile[-2])\n if tile[0] > 1:\n addtag('ImageDepth', 'I', 1, shape[-4])\n addtag('TileDepth', 'I', 1, tile[0])\n addtag('NewSubfileType', 'I', 1, subfiletype)\n if not bilevel:\n sampleformat = {'u': 1, 'i': 2, 'f': 3, 'c': 6}[datadtype.kind]\n addtag('SampleFormat', 'H', samplesperpixel,\n (sampleformat,) * samplesperpixel)\n addtag('PhotometricInterpretation', 'H', 1, photometric.value)\n if colormap is not None:\n addtag('ColorMap', 'H', colormap.size, colormap)\n addtag('SamplesPerPixel', 'H', 1, samplesperpixel)\n if bilevel:\n pass\n elif planarconfig and samplesperpixel > 1:\n addtag('PlanarConfiguration', 'H', 1, planarconfig.value)\n addtag('BitsPerSample', 'H', samplesperpixel,\n (datadtype.itemsize * 8,) * samplesperpixel)\n else:\n addtag('BitsPerSample', 'H', 1, datadtype.itemsize * 8)\n if extrasamples:\n if extrasamples_ is not None:\n if extrasamples != len(extrasamples_):\n raise ValueError('wrong number of extrasamples specified')\n addtag('ExtraSamples', 'H', extrasamples, extrasamples_)\n elif photometric == RGB and extrasamples == 1:\n # Unassociated alpha channel\n addtag('ExtraSamples', 'H', 1, 2)\n else:\n # Unspecified alpha channel\n addtag('ExtraSamples', 'H', extrasamples, (0,) * extrasamples)\n if resolution is not None:\n addtag('XResolution', '2I', 1, rational(resolution[0]))\n addtag('YResolution', '2I', 1, rational(resolution[1]))\n if len(resolution) > 2:\n unit = resolution[2]\n unit = 1 if unit is None else enumarg(TIFF.RESUNIT, unit)\n elif self._imagej:\n unit = 1\n else:\n unit = 2\n addtag('ResolutionUnit', 'H', 1, unit)\n elif not self._imagej:\n addtag('XResolution', '2I', 1, (1, 1))\n addtag('YResolution', '2I', 1, (1, 1))\n addtag('ResolutionUnit', 'H', 1, 1)\n if ijmetadata:\n for t in imagej_metadata_tag(ijmetadata, byteorder):\n addtag(*t)\n\n contiguous = not compress\n if tile:\n # one chunk per tile per plane\n tiles = ((shape[2] + tile[0] - 1) // tile[0],\n (shape[3] + tile[1] - 1) // tile[1],\n (shape[4] + tile[2] - 1) // tile[2])\n numtiles = product(tiles) * shape[1]\n databytecounts = [\n product(tile) * shape[-1] * datadtype.itemsize] * numtiles\n addtag(tagbytecounts, offsetformat, numtiles, databytecounts)\n addtag(tagoffsets, offsetformat, numtiles, [0] * numtiles)\n contiguous = contiguous and product(tiles) == 1\n if not contiguous:\n # allocate tile buffer\n chunk = numpy.empty(tile + (shape[-1],), dtype=datadtype)\n elif contiguous and (bilevel or rowsperstrip is None):\n # one strip per plane\n if bilevel:\n databytecounts = [product(datashape[2:])] * shape[1]\n else:\n databytecounts = [\n product(datashape[2:]) * datadtype.itemsize] * shape[1]\n addtag(tagbytecounts, offsetformat, shape[1], databytecounts)\n addtag(tagoffsets, offsetformat, shape[1], [0] * shape[1])\n addtag('RowsPerStrip', 'I', 1, shape[-3])\n else:\n # use rowsperstrip\n rowsize = product(shape[-2:]) * datadtype.itemsize\n if rowsperstrip is None:\n # compress ~64 KB chunks by default\n rowsperstrip = 65536 // rowsize if compress else shape[-3]\n if rowsperstrip < 1:\n rowsperstrip = 1\n elif rowsperstrip > shape[-3]:\n rowsperstrip = shape[-3]\n addtag('RowsPerStrip', 'I', 1, rowsperstrip)\n\n numstrips1 = (shape[-3] + rowsperstrip - 1) // rowsperstrip\n numstrips = numstrips1 * shape[1]\n if compress:\n databytecounts = [0] * numstrips\n else:\n # TODO: save bilevel data with rowsperstrip\n stripsize = rowsperstrip * rowsize\n databytecounts = [stripsize] * numstrips\n stripsize -= rowsize * (numstrips1 * rowsperstrip - shape[-3])\n for i in range(numstrips1-1, numstrips, numstrips1):\n databytecounts[i] = stripsize\n addtag(tagbytecounts, offsetformat, numstrips, databytecounts)\n addtag(tagoffsets, offsetformat, numstrips, [0] * numstrips)\n\n if data is None and not contiguous:\n raise ValueError('cannot write non-contiguous empty file')\n\n # add extra tags from user\n for t in extratags:\n addtag(*t)\n\n # TODO: check TIFFReadDirectoryCheckOrder warning in files containing\n # multiple tags of same code\n # the entries in an IFD must be sorted in ascending order by tag code\n tags = sorted(tags, key=lambda x: x[0])\n\n fhpos = fh.tell()\n if not (self._bigtiff or self._imagej) and fhpos + datasize > 2**32-1:\n raise ValueError('data too large for standard TIFF file')\n\n # if not compressed or multi-tiled, write the first IFD and then\n # all data contiguously; else, write all IFDs and data interleaved\n for pageindex in range(1 if contiguous else shape[0]):\n\n ifdpos = fhpos\n if ifdpos % 2:\n # location of IFD must begin on a word boundary\n fh.write(b'\\0')\n ifdpos += 1\n\n # update pointer at ifdoffset\n fh.seek(self._ifdoffset)\n fh.write(pack(offsetformat, ifdpos))\n fh.seek(ifdpos)\n\n # create IFD in memory\n if pageindex < 2:\n ifd = io.BytesIO()\n ifd.write(pack(tagnoformat, len(tags)))\n tagoffset = ifd.tell()\n ifd.write(b''.join(t[1] for t in tags))\n ifdoffset = ifd.tell()\n ifd.write(pack(offsetformat, 0)) # offset to next IFD\n # write tag values and patch offsets in ifdentries\n for tagindex, tag in enumerate(tags):\n offset = tagoffset + tagindex * tagsize + offsetsize + 4\n code = tag[0]\n value = tag[2]\n if value:\n pos = ifd.tell()\n if pos % 2:\n # tag value is expected to begin on word boundary\n ifd.write(b'\\0')\n pos += 1\n ifd.seek(offset)\n ifd.write(pack(offsetformat, ifdpos + pos))\n ifd.seek(pos)\n ifd.write(value)\n if code == tagoffsets:\n dataoffsetsoffset = offset, pos\n elif code == tagbytecounts:\n databytecountsoffset = offset, pos\n elif code == 270 and value.endswith(b'\\0\\0\\0\\0'):\n # image description buffer\n self._descriptionoffset = ifdpos + pos\n self._descriptionlenoffset = (\n ifdpos + tagoffset + tagindex*tagsize + 4)\n elif code == tagoffsets:\n dataoffsetsoffset = offset, None\n elif code == tagbytecounts:\n databytecountsoffset = offset, None\n ifdsize = ifd.tell()\n if ifdsize % 2:\n ifd.write(b'\\0')\n ifdsize += 1\n\n # write IFD later when strip/tile bytecounts and offsets are known\n fh.seek(ifdsize, 1)\n\n # write image data\n dataoffset = fh.tell()\n skip = align - dataoffset % align\n fh.seek(skip, 1)\n dataoffset += skip\n if contiguous:\n if data is None:\n fh.write_empty(datasize)\n else:\n fh.write_array(data)\n elif tile:\n if data is None:\n fh.write_empty(numtiles * databytecounts[0])\n else:\n stripindex = 0\n for plane in data[pageindex]:\n for tz in range(tiles[0]):\n for ty in range(tiles[1]):\n for tx in range(tiles[2]):\n c0 = min(tile[0], shape[2] - tz*tile[0])\n c1 = min(tile[1], shape[3] - ty*tile[1])\n c2 = min(tile[2], shape[4] - tx*tile[2])\n chunk[c0:, c1:, c2:] = 0\n chunk[:c0, :c1, :c2] = plane[\n tz*tile[0]:tz*tile[0]+c0,\n ty*tile[1]:ty*tile[1]+c1,\n tx*tile[2]:tx*tile[2]+c2]\n if compress:\n t = compress(chunk)\n fh.write(t)\n databytecounts[stripindex] = len(t)\n stripindex += 1\n else:\n fh.write_array(chunk)\n # fh.flush()\n elif compress:\n # write one strip per rowsperstrip\n assert data.shape[2] == 1 # not handling depth\n numstrips = (shape[-3] + rowsperstrip - 1) // rowsperstrip\n stripindex = 0\n for plane in data[pageindex]:\n for i in range(numstrips):\n strip = plane[0, i*rowsperstrip: (i+1)*rowsperstrip]\n strip = compress(strip)\n fh.write(strip)\n databytecounts[stripindex] = len(strip)\n stripindex += 1\n else:\n fh.write_array(data[pageindex])\n\n # update strip/tile offsets\n offset, pos = dataoffsetsoffset\n ifd.seek(offset)\n if pos:\n ifd.write(pack(offsetformat, ifdpos + pos))\n ifd.seek(pos)\n offset = dataoffset\n for size in databytecounts:\n ifd.write(pack(offsetformat, offset))\n offset += size\n else:\n ifd.write(pack(offsetformat, dataoffset))\n\n if compress:\n # update strip/tile bytecounts\n offset, pos = databytecountsoffset\n ifd.seek(offset)\n if pos:\n ifd.write(pack(offsetformat, ifdpos + pos))\n ifd.seek(pos)\n for size in databytecounts:\n ifd.write(pack(offsetformat, size))\n else:\n ifd.write(pack(offsetformat, databytecounts[0]))\n\n fhpos = fh.tell()\n fh.seek(ifdpos)\n fh.write(iogetbuffer(ifd))\n fh.flush()\n fh.seek(fhpos)\n\n self._ifdoffset = ifdpos + ifdoffset\n\n # remove tags that should be written only once\n if pageindex == 0:\n tags = [tag for tag in tags if not tag[-1]]\n\n self._shape = shape\n self._datashape = (1,) + input_shape\n self._datadtype = datadtype\n self._dataoffset = dataoffset\n self._databytecounts = databytecounts\n\n if contiguous:\n # write remaining IFDs/tags later\n self._tags = tags\n # return offset and size of image data\n if returnoffset:\n return dataoffset, sum(databytecounts)\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nwrites out the remaining IFDs and tags to file.", "response": "def _write_remaining_pages(self):\n \"\"\"Write outstanding IFDs and tags to file.\"\"\"\n if not self._tags or self._truncate:\n return\n\n pageno = self._shape[0] * self._datashape[0] - 1\n if pageno < 1:\n self._tags = None\n self._datadtype = None\n self._dataoffset = None\n self._databytecounts = None\n return\n\n fh = self._fh\n fhpos = fh.tell()\n if fhpos % 2:\n fh.write(b'\\0')\n fhpos += 1\n\n pack = struct.pack\n offsetformat = self._byteorder + self._offsetformat\n offsetsize = self._offsetsize\n tagnoformat = self._byteorder + self._tagnoformat\n tagsize = self._tagsize\n dataoffset = self._dataoffset\n pagedatasize = sum(self._databytecounts)\n\n # construct template IFD in memory\n # need to patch offsets to next IFD and data before writing to file\n ifd = io.BytesIO()\n ifd.write(pack(tagnoformat, len(self._tags)))\n tagoffset = ifd.tell()\n ifd.write(b''.join(t[1] for t in self._tags))\n ifdoffset = ifd.tell()\n ifd.write(pack(offsetformat, 0)) # offset to next IFD\n # tag values\n for tagindex, tag in enumerate(self._tags):\n offset = tagoffset + tagindex * tagsize + offsetsize + 4\n code = tag[0]\n value = tag[2]\n if value:\n pos = ifd.tell()\n if pos % 2:\n # tag value is expected to begin on word boundary\n ifd.write(b'\\0')\n pos += 1\n ifd.seek(offset)\n try:\n ifd.write(pack(offsetformat, fhpos + pos))\n except Exception: # struct.error\n if self._imagej:\n warnings.warn('truncating ImageJ file')\n self._truncate = True\n return\n raise ValueError('data too large for non-BigTIFF file')\n ifd.seek(pos)\n ifd.write(value)\n if code == self._tagoffsets:\n # save strip/tile offsets for later updates\n dataoffsetsoffset = offset, pos\n elif code == self._tagoffsets:\n dataoffsetsoffset = offset, None\n\n ifdsize = ifd.tell()\n if ifdsize % 2:\n ifd.write(b'\\0')\n ifdsize += 1\n\n # check if all IFDs fit in file\n if not self._bigtiff and fhpos + ifdsize * pageno > 2**32 - 32:\n if self._imagej:\n warnings.warn('truncating ImageJ file')\n self._truncate = True\n return\n raise ValueError('data too large for non-BigTIFF file')\n\n # assemble IFD chain in memory from IFD template\n ifds = io.BytesIO(bytes(ifdsize * pageno))\n ifdpos = fhpos\n for _ in range(pageno):\n # update strip/tile offsets in IFD\n dataoffset += pagedatasize # offset to image data\n offset, pos = dataoffsetsoffset\n ifd.seek(offset)\n if pos:\n ifd.write(pack(offsetformat, ifdpos + pos))\n ifd.seek(pos)\n offset = dataoffset\n for size in self._databytecounts:\n ifd.write(pack(offsetformat, offset))\n offset += size\n else:\n ifd.write(pack(offsetformat, dataoffset))\n # update pointer at ifdoffset to point to next IFD in file\n ifdpos += ifdsize\n ifd.seek(ifdoffset)\n ifd.write(pack(offsetformat, ifdpos))\n # write IFD entry\n ifds.write(iogetbuffer(ifd))\n\n # terminate IFD chain\n ifdoffset += ifdsize * (pageno - 1)\n ifds.seek(ifdoffset)\n ifds.write(pack(offsetformat, 0))\n # write IFD chain to file\n fh.write(iogetbuffer(ifds))\n # update file to point to new IFD chain\n pos = fh.tell()\n fh.seek(self._ifdoffset)\n fh.write(pack(offsetformat, fhpos))\n fh.flush()\n fh.seek(pos)\n\n self._ifdoffset = fhpos + ifdoffset\n self._tags = None\n self._datadtype = None\n self._dataoffset = None\n self._databytecounts = None"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _write_image_description(self):\n if (not self._datashape or self._datashape[0] == 1 or\n self._descriptionoffset <= 0):\n return\n\n colormapped = self._colormap is not None\n if self._imagej:\n isrgb = self._shape[-1] in (3, 4)\n description = imagej_description(\n self._datashape, isrgb, colormapped, **self._metadata)\n else:\n description = json_description(self._datashape, **self._metadata)\n\n # rewrite description and its length to file\n description = description.encode('utf-8')\n description = description[:self._descriptionlen-1]\n pos = self._fh.tell()\n self._fh.seek(self._descriptionoffset)\n self._fh.write(description)\n self._fh.seek(self._descriptionlenoffset)\n self._fh.write(struct.pack(self._byteorder+self._offsetformat,\n len(description)+1))\n self._fh.seek(pos)\n\n self._descriptionoffset = 0\n self._descriptionlenoffset = 0\n self._descriptionlen = 0", "response": "Write metadata to ImageDescription tag."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nwrites remaining pages and close file handle.", "response": "def close(self):\n \"\"\"Write remaining pages and close file handle.\"\"\"\n if not self._truncate:\n self._write_remaining_pages()\n self._write_image_description()\n self._fh.close()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef close(self):\n for tif in self._files.values():\n tif.filehandle.close()\n self._files = {}", "response": "Close open file handle s."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning image data from selected TIFF pages as numpy array.", "response": "def asarray(self, key=None, series=None, out=None, validate=True,\n maxworkers=None):\n \"\"\"Return image data from selected TIFF page(s) as numpy array.\n\n By default, the data from the first series is returned.\n\n Parameters\n ----------\n key : int, slice, or sequence of indices\n Defines which pages to return as array.\n If None (default), data from a series (default 0) is returned.\n If not None, data from the specified pages in the whole file\n (if 'series' is None) or a specified series are returned as a\n stacked array.\n Requesting an array from multiple pages that are not compatible\n wrt. shape, dtype, compression etc is undefined, i.e. may crash\n or return incorrect values.\n series : int or TiffPageSeries\n Defines which series of pages to return as array.\n out : numpy.ndarray, str, or file-like object\n Buffer where image data will be saved.\n If None (default), a new array will be created.\n If numpy.ndarray, a writable array of compatible dtype and shape.\n If 'memmap', directly memory-map the image data in the TIFF file\n if possible; else create a memory-mapped array in a temporary file.\n If str or open file, the file name or file object used to\n create a memory-map to an array stored in a binary file on disk.\n validate : bool\n If True (default), validate various tags.\n Passed to TiffPage.asarray().\n maxworkers : int or None\n Maximum number of threads to concurrently get data from pages\n or tiles. If None (default), mutli-threading is enabled if data\n are compressed. If 0, up to half the CPU cores are used.\n If 1, mutli-threading is disabled.\n Reading data from file is limited to a single thread.\n Using multiple threads can significantly speed up this function\n if the bottleneck is decoding compressed data, e.g. in case of\n large LZW compressed LSM files or JPEG compressed tiled slides.\n If the bottleneck is I/O or pure Python code, using multiple\n threads might be detrimental.\n\n Returns\n -------\n numpy.ndarray\n Image data from the specified pages.\n See the TiffPage.asarray function for operations that are\n applied (or not) to the raw data stored in the file.\n\n \"\"\"\n if not self.pages:\n return numpy.array([])\n if key is None and series is None:\n series = 0\n if series is None:\n pages = self.pages\n else:\n try:\n series = self.series[series]\n except (KeyError, TypeError):\n pass\n pages = series.pages\n\n if key is None:\n pass\n elif series is None:\n pages = self.pages._getlist(key)\n elif isinstance(key, inttypes):\n pages = [pages[key]]\n elif isinstance(key, slice):\n pages = pages[key]\n elif isinstance(key, Iterable):\n pages = [pages[k] for k in key]\n else:\n raise TypeError('key must be an int, slice, or sequence')\n\n if not pages:\n raise ValueError('no pages selected')\n\n if key is None and series and series.offset:\n typecode = self.byteorder + series.dtype.char\n if pages[0].is_memmappable and (isinstance(out, str) and\n out == 'memmap'):\n # direct mapping\n result = self.filehandle.memmap_array(\n typecode, series.shape, series.offset)\n else:\n # read into output\n if out is not None:\n out = create_output(out, series.shape, series.dtype)\n self.filehandle.seek(series.offset)\n result = self.filehandle.read_array(\n typecode, product(series.shape), out=out)\n elif len(pages) == 1:\n result = pages[0].asarray(out=out, validate=validate,\n maxworkers=maxworkers)\n else:\n result = stack_pages(pages, out=out, maxworkers=maxworkers)\n\n if result is None:\n return None\n\n if key is None:\n try:\n result.shape = series.shape\n except ValueError:\n try:\n log.warning('TiffFile.asarray: failed to reshape %s to %s',\n result.shape, series.shape)\n # try series of expected shapes\n result.shape = (-1,) + series.shape\n except ValueError:\n # revert to generic shape\n result.shape = (-1,) + pages[0].shape\n elif len(pages) == 1:\n result.shape = pages[0].shape\n else:\n result.shape = (-1,) + pages[0].shape\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef series(self):\n if not self.pages:\n return []\n\n useframes = self.pages.useframes\n keyframe = self.pages.keyframe.index\n series = []\n for name in ('lsm', 'ome', 'imagej', 'shaped', 'fluoview', 'sis',\n 'uniform', 'mdgel'):\n if getattr(self, 'is_' + name, False):\n series = getattr(self, '_series_' + name)()\n break\n self.pages.useframes = useframes\n self.pages.keyframe = keyframe\n if not series:\n series = self._series_generic()\n\n # remove empty series, e.g. in MD Gel files\n series = [s for s in series if product(s.shape) > 0]\n\n for i, s in enumerate(series):\n s.index = i\n return series", "response": "Return related pages as TiffPageSeries."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns image series in file.", "response": "def _series_generic(self):\n \"\"\"Return image series in file.\n\n A series is a sequence of TiffPages with the same hash.\n\n \"\"\"\n pages = self.pages\n pages._clear(False)\n pages.useframes = False\n if pages.cache:\n pages._load()\n\n result = []\n keys = []\n series = {}\n for page in pages:\n if not page.shape or product(page.shape) == 0:\n continue\n key = page.hash\n if key in series:\n series[key].append(page)\n else:\n keys.append(key)\n series[key] = [page]\n\n for key in keys:\n pages = series[key]\n page = pages[0]\n shape = page.shape\n axes = page.axes\n if len(pages) > 1:\n shape = (len(pages),) + shape\n axes = 'I' + axes\n result.append(TiffPageSeries(pages, shape, page.dtype, axes,\n kind='Generic'))\n\n self.is_uniform = len(result) == 1\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns all images in file as single series.", "response": "def _series_uniform(self):\n \"\"\"Return all images in file as single series.\"\"\"\n page = self.pages[0]\n shape = page.shape\n axes = page.axes\n dtype = page.dtype\n validate = not (page.is_scanimage or page.is_nih)\n pages = self.pages._getlist(validate=validate)\n lenpages = len(pages)\n if lenpages > 1:\n shape = (lenpages,) + shape\n axes = 'I' + axes\n if page.is_scanimage:\n kind = 'ScanImage'\n elif page.is_nih:\n kind = 'NIHImage'\n else:\n kind = 'Uniform'\n return [TiffPageSeries(pages, shape, dtype, axes, kind=kind)]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _series_shaped(self):\n pages = self.pages\n pages.useframes = True\n lenpages = len(pages)\n\n def append_series(series, pages, axes, shape, reshape, name,\n truncated):\n page = pages[0]\n if not axes:\n shape = page.shape\n axes = page.axes\n if len(pages) > 1:\n shape = (len(pages),) + shape\n axes = 'Q' + axes\n size = product(shape)\n resize = product(reshape)\n if page.is_contiguous and resize > size and resize % size == 0:\n if truncated is None:\n truncated = True\n axes = 'Q' + axes\n shape = (resize // size,) + shape\n try:\n axes = reshape_axes(axes, shape, reshape)\n shape = reshape\n except ValueError as exc:\n log.warning('Shaped series: %s: %s',\n exc.__class__.__name__, exc)\n series.append(\n TiffPageSeries(pages, shape, page.dtype, axes, name=name,\n kind='Shaped', truncated=truncated))\n\n keyframe = axes = shape = reshape = name = None\n series = []\n index = 0\n while True:\n if index >= lenpages:\n break\n # new keyframe; start of new series\n pages.keyframe = index\n keyframe = pages.keyframe\n if not keyframe.is_shaped:\n log.warning(\n 'Shaped series: invalid metadata or corrupted file')\n return None\n # read metadata\n axes = None\n shape = None\n metadata = json_description_metadata(keyframe.is_shaped)\n name = metadata.get('name', '')\n reshape = metadata['shape']\n truncated = metadata.get('truncated', None)\n if 'axes' in metadata:\n axes = metadata['axes']\n if len(axes) == len(reshape):\n shape = reshape\n else:\n axes = ''\n log.warning('Shaped series: axes do not match shape')\n # skip pages if possible\n spages = [keyframe]\n size = product(reshape)\n npages, mod = divmod(size, product(keyframe.shape))\n if mod:\n log.warning(\n 'Shaped series: series shape does not match page shape')\n return None\n if 1 < npages <= lenpages - index:\n size *= keyframe._dtype.itemsize\n if truncated:\n npages = 1\n elif (keyframe.is_final and\n keyframe.offset + size < pages[index+1].offset):\n truncated = False\n else:\n # need to read all pages for series\n truncated = False\n for j in range(index+1, index+npages):\n page = pages[j]\n page.keyframe = keyframe\n spages.append(page)\n append_series(series, spages, axes, shape, reshape, name,\n truncated)\n index += npages\n\n self.is_uniform = len(series) == 1\n\n return series", "response": "Return image series in shaped file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _series_imagej(self):\n # ImageJ's dimension order is always TZCYXS\n # TODO: fix loading of color, composite, or palette images\n pages = self.pages\n pages.useframes = True\n pages.keyframe = 0\n page = pages[0]\n ij = self.imagej_metadata\n\n def is_hyperstack():\n # ImageJ hyperstack store all image metadata in the first page and\n # image data are stored contiguously before the second page, if any\n if not page.is_final:\n return False\n images = ij.get('images', 0)\n if images <= 1:\n return False\n offset, count = page.is_contiguous\n if (count != product(page.shape) * page.bitspersample // 8\n or offset + count*images > self.filehandle.size):\n raise ValueError()\n # check that next page is stored after data\n if len(pages) > 1 and offset + count*images > pages[1].offset:\n return False\n return True\n\n try:\n hyperstack = is_hyperstack()\n except ValueError:\n log.warning('ImageJ series: invalid metadata or corrupted file')\n return None\n if hyperstack:\n # no need to read other pages\n pages = [page]\n else:\n pages = pages[:]\n\n shape = []\n axes = []\n if 'frames' in ij:\n shape.append(ij['frames'])\n axes.append('T')\n if 'slices' in ij:\n shape.append(ij['slices'])\n axes.append('Z')\n if 'channels' in ij and not (page.photometric == 2 and not\n ij.get('hyperstack', False)):\n shape.append(ij['channels'])\n axes.append('C')\n remain = ij.get('images', len(pages))//(product(shape) if shape else 1)\n if remain > 1:\n shape.append(remain)\n axes.append('I')\n if page.axes[0] == 'I':\n # contiguous multiple images\n shape.extend(page.shape[1:])\n axes.extend(page.axes[1:])\n elif page.axes[:2] == 'SI':\n # color-mapped contiguous multiple images\n shape = page.shape[0:1] + tuple(shape) + page.shape[2:]\n axes = list(page.axes[0]) + axes + list(page.axes[2:])\n else:\n shape.extend(page.shape)\n axes.extend(page.axes)\n\n truncated = (\n hyperstack and len(self.pages) == 1 and\n page.is_contiguous[1] != product(shape) * page.bitspersample // 8)\n\n self.is_uniform = True\n\n return [TiffPageSeries(pages, shape, page.dtype, axes, kind='ImageJ',\n truncated=truncated)]", "response": "Return image series in ImageJ file."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns image series in FluoView file.", "response": "def _series_fluoview(self):\n \"\"\"Return image series in FluoView file.\"\"\"\n pages = self.pages._getlist(validate=False)\n\n mm = self.fluoview_metadata\n mmhd = list(reversed(mm['Dimensions']))\n axes = ''.join(TIFF.MM_DIMENSIONS.get(i[0].upper(), 'Q')\n for i in mmhd if i[1] > 1)\n shape = tuple(int(i[1]) for i in mmhd if i[1] > 1)\n self.is_uniform = True\n return [TiffPageSeries(pages, shape, pages[0].dtype, axes,\n name=mm['ImageName'], kind='FluoView')]"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns image series in MD Gel file.", "response": "def _series_mdgel(self):\n \"\"\"Return image series in MD Gel file.\"\"\"\n # only a single page, scaled according to metadata in second page\n self.pages.useframes = False\n self.pages.keyframe = 0\n md = self.mdgel_metadata\n if md['FileTag'] in (2, 128):\n dtype = numpy.dtype('float32')\n scale = md['ScalePixel']\n scale = scale[0] / scale[1] # rational\n if md['FileTag'] == 2:\n # squary root data format\n def transform(a):\n return a.astype('float32')**2 * scale\n else:\n def transform(a):\n return a.astype('float32') * scale\n else:\n transform = None\n page = self.pages[0]\n self.is_uniform = False\n return [TiffPageSeries([page], page.shape, dtype, page.axes,\n transform=transform, kind='MDGel')]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _series_sis(self):\n pages = self.pages._getlist(validate=False)\n page = pages[0]\n lenpages = len(pages)\n md = self.sis_metadata\n if 'shape' in md and 'axes' in md:\n shape = md['shape'] + page.shape\n axes = md['axes'] + page.axes\n elif lenpages == 1:\n shape = page.shape\n axes = page.axes\n else:\n shape = (lenpages,) + page.shape\n axes = 'I' + page.axes\n self.is_uniform = True\n return [TiffPageSeries(pages, shape, page.dtype, axes, kind='SIS')]", "response": "Return image series in Olympus SIS file."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _series_ome(self):\n from xml.etree import cElementTree as etree # delayed import\n omexml = self.pages[0].description\n try:\n root = etree.fromstring(omexml)\n except etree.ParseError as exc:\n # TODO: test badly encoded OME-XML\n log.warning('OME series: %s: %s', exc.__class__.__name__, exc)\n try:\n # might work on Python 2\n omexml = omexml.decode('utf-8', 'ignore').encode('utf-8')\n root = etree.fromstring(omexml)\n except Exception:\n return None\n\n self.pages.cache = True\n self.pages.useframes = True\n self.pages.keyframe = 0\n self.pages._load(keyframe=None)\n\n uuid = root.attrib.get('UUID', None)\n self._files = {uuid: self}\n dirname = self._fh.dirname\n modulo = {}\n series = []\n for element in root:\n if element.tag.endswith('BinaryOnly'):\n # TODO: load OME-XML from master or companion file\n log.warning('OME series: not an ome-tiff master file')\n break\n if element.tag.endswith('StructuredAnnotations'):\n for annot in element:\n if not annot.attrib.get('Namespace',\n '').endswith('modulo'):\n continue\n for value in annot:\n for modul in value:\n for along in modul:\n if not along.tag[:-1].endswith('Along'):\n continue\n axis = along.tag[-1]\n newaxis = along.attrib.get('Type', 'other')\n newaxis = TIFF.AXES_LABELS[newaxis]\n if 'Start' in along.attrib:\n step = float(along.attrib.get('Step', 1))\n start = float(along.attrib['Start'])\n stop = float(along.attrib['End']) + step\n labels = numpy.arange(start, stop, step)\n else:\n labels = [label.text for label in along\n if label.tag.endswith('Label')]\n modulo[axis] = (newaxis, labels)\n\n if not element.tag.endswith('Image'):\n continue\n\n attr = element.attrib\n name = attr.get('Name', None)\n\n for pixels in element:\n if not pixels.tag.endswith('Pixels'):\n continue\n attr = pixels.attrib\n # dtype = attr.get('PixelType', None)\n axes = ''.join(reversed(attr['DimensionOrder']))\n shape = idxshape = list(int(attr['Size'+ax]) for ax in axes)\n size = product(shape[:-2])\n ifds = None\n spp = 1 # samples per pixel\n for data in pixels:\n if data.tag.endswith('Channel'):\n attr = data.attrib\n if ifds is None:\n spp = int(attr.get('SamplesPerPixel', spp))\n ifds = [None] * (size // spp)\n if spp > 1:\n # correct channel dimension for spp\n idxshape = list((shape[i] // spp if ax == 'C'\n else shape[i])\n for i, ax in enumerate(axes))\n elif int(attr.get('SamplesPerPixel', 1)) != spp:\n raise ValueError(\n 'cannot handle differing SamplesPerPixel')\n continue\n if ifds is None:\n ifds = [None] * (size // spp)\n if not data.tag.endswith('TiffData'):\n continue\n attr = data.attrib\n ifd = int(attr.get('IFD', 0))\n num = int(attr.get('NumPlanes', 1 if 'IFD' in attr else 0))\n num = int(attr.get('PlaneCount', num))\n idx = [int(attr.get('First'+ax, 0)) for ax in axes[:-2]]\n try:\n idx = numpy.ravel_multi_index(idx, idxshape[:-2])\n except ValueError:\n # ImageJ produces invalid ome-xml when cropping\n log.warning('OME series: invalid TiffData index')\n continue\n for uuid in data:\n if not uuid.tag.endswith('UUID'):\n continue\n if uuid.text not in self._files:\n if not self._multifile:\n # abort reading multifile OME series\n # and fall back to generic series\n return []\n fname = uuid.attrib['FileName']\n try:\n tif = TiffFile(os.path.join(dirname, fname))\n tif.pages.cache = True\n tif.pages.useframes = True\n tif.pages.keyframe = 0\n tif.pages._load(keyframe=None)\n except (IOError, FileNotFoundError, ValueError):\n log.warning(\"OME series: failed to read '%s'\",\n fname)\n break\n self._files[uuid.text] = tif\n tif.close()\n pages = self._files[uuid.text].pages\n try:\n for i in range(num if num else len(pages)):\n ifds[idx + i] = pages[ifd + i]\n except IndexError:\n log.warning('OME series: index out of range')\n # only process first UUID\n break\n else:\n pages = self.pages\n try:\n for i in range(num if num else\n min(len(pages), len(ifds))):\n ifds[idx + i] = pages[ifd + i]\n except IndexError:\n log.warning('OME series: index out of range')\n\n if all(i is None for i in ifds):\n # skip images without data\n continue\n\n # find a keyframe\n keyframe = None\n for i in ifds:\n # try find a TiffPage\n if i and i == i.keyframe:\n keyframe = i\n break\n if keyframe is None:\n # reload a TiffPage from file\n for i, keyframe in enumerate(ifds):\n if keyframe:\n keyframe.parent.pages.keyframe = keyframe.index\n keyframe = keyframe.parent.pages[keyframe.index]\n ifds[i] = keyframe\n break\n\n # move channel axis to match PlanarConfiguration storage\n # TODO: is this a bug or a inconsistency in the OME spec?\n if spp > 1:\n if keyframe.planarconfig == 1 and axes[-1] != 'C':\n i = axes.index('C')\n axes = axes[:i] + axes[i+1:] + axes[i:i+1]\n shape = shape[:i] + shape[i+1:] + shape[i:i+1]\n\n # FIXME: this implementation assumes the last dimensions are\n # stored in TIFF pages. Apparently that is not always the case.\n # For now, verify that shapes of keyframe and series match\n # If not, skip series.\n if keyframe.shape != tuple(shape[-len(keyframe.shape):]):\n log.warning('OME series: incompatible page shape %s; '\n 'expected %s', keyframe.shape,\n tuple(shape[-len(keyframe.shape):]))\n del ifds\n continue\n\n # set a keyframe on all IFDs\n for i in ifds:\n if i is not None:\n i.keyframe = keyframe\n\n series.append(\n TiffPageSeries(ifds, shape, keyframe.dtype, axes,\n parent=self, name=name, kind='OME'))\n del ifds\n\n for serie in series:\n shape = list(serie.shape)\n for axis, (newaxis, labels) in modulo.items():\n i = serie.axes.index(axis)\n size = len(labels)\n if shape[i] == size:\n serie.axes = serie.axes.replace(axis, newaxis, 1)\n else:\n shape[i] //= size\n shape.insert(i+1, size)\n serie.axes = serie.axes.replace(axis, axis+newaxis, 1)\n serie.shape = tuple(shape)\n\n # squeeze dimensions\n for serie in series:\n serie.shape, serie.axes = squeeze_axes(serie.shape, serie.axes)\n self.is_uniform = len(series) == 1\n return series", "response": "Return image series in OME - TIFF file ( s )."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn main and thumbnail series in LSM file.", "response": "def _series_lsm(self):\n \"\"\"Return main and thumbnail series in LSM file.\"\"\"\n lsmi = self.lsm_metadata\n axes = TIFF.CZ_LSMINFO_SCANTYPE[lsmi['ScanType']]\n if self.pages[0].photometric == 2: # RGB; more than one channel\n axes = axes.replace('C', '').replace('XY', 'XYC')\n if lsmi.get('DimensionP', 0) > 1:\n axes += 'P'\n if lsmi.get('DimensionM', 0) > 1:\n axes += 'M'\n axes = axes[::-1]\n shape = tuple(int(lsmi[TIFF.CZ_LSMINFO_DIMENSIONS[i]]) for i in axes)\n name = lsmi.get('Name', '')\n pages = self.pages._getlist(slice(0, None, 2), validate=False)\n dtype = pages[0].dtype\n series = [TiffPageSeries(pages, shape, dtype, axes, name=name,\n kind='LSM')]\n\n if self.pages[1].is_reduced:\n pages = self.pages._getlist(slice(1, None, 2), validate=False)\n dtype = pages[0].dtype\n cp = 1\n i = 0\n while cp < len(pages) and i < len(shape)-2:\n cp *= shape[i]\n i += 1\n shape = shape[:i] + pages[0].shape\n axes = axes[:i] + 'CYX'\n series.append(TiffPageSeries(pages, shape, dtype, axes, name=name,\n kind='LSMreduced'))\n\n self.is_uniform = False\n return series"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _lsm_load_pages(self):\n # cache all pages to preserve corrected values\n pages = self.pages\n pages.cache = True\n pages.useframes = True\n # use first and second page as keyframes\n pages.keyframe = 1\n pages.keyframe = 0\n # load remaining pages as frames\n pages._load(keyframe=None)\n # fix offsets and bytecounts first\n self._lsm_fix_strip_offsets()\n self._lsm_fix_strip_bytecounts()\n # assign keyframes for data and thumbnail series\n keyframe = pages[0]\n for page in pages[::2]:\n page.keyframe = keyframe\n keyframe = pages[1]\n for page in pages[1::2]:\n page.keyframe = keyframe", "response": "Load and fix all pages from LSM file."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _lsm_fix_strip_offsets(self):\n if self.filehandle.size < 2**32:\n return\n\n pages = self.pages\n npages = len(pages)\n series = self.series[0]\n axes = series.axes\n\n # find positions\n positions = 1\n for i in 0, 1:\n if series.axes[i] in 'PM':\n positions *= series.shape[i]\n\n # make time axis first\n if positions > 1:\n ntimes = 0\n for i in 1, 2:\n if axes[i] == 'T':\n ntimes = series.shape[i]\n break\n if ntimes:\n div, mod = divmod(npages, 2*positions*ntimes)\n assert mod == 0\n shape = (positions, ntimes, div, 2)\n indices = numpy.arange(product(shape)).reshape(shape)\n indices = numpy.moveaxis(indices, 1, 0)\n else:\n indices = numpy.arange(npages).reshape(-1, 2)\n\n # images of reduced page might be stored first\n if pages[0]._offsetscounts[0][0] > pages[1]._offsetscounts[0][0]:\n indices = indices[..., ::-1]\n\n # unwrap offsets\n wrap = 0\n previousoffset = 0\n for i in indices.flat:\n page = pages[int(i)]\n dataoffsets = []\n for currentoffset in page._offsetscounts[0]:\n if currentoffset < previousoffset:\n wrap += 2**32\n dataoffsets.append(currentoffset + wrap)\n previousoffset = currentoffset\n page._offsetscounts = dataoffsets, page._offsetscounts[1]", "response": "Unwrap strip offsets for LSM files greater than 4 GB."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfix the bytecounts of compressed data in LSM files.", "response": "def _lsm_fix_strip_bytecounts(self):\n \"\"\"Set databytecounts to size of compressed data.\n\n The StripByteCounts tag in LSM files contains the number of bytes\n for the uncompressed data.\n\n \"\"\"\n pages = self.pages\n if pages[0].compression == 1:\n return\n # sort pages by first strip offset\n pages = sorted(pages, key=lambda p: p._offsetscounts[0][0])\n npages = len(pages) - 1\n for i, page in enumerate(pages):\n if page.index % 2:\n continue\n offsets, bytecounts = page._offsetscounts\n if i < npages:\n lastoffset = pages[i+1]._offsetscounts[0][0]\n else:\n # LZW compressed strips might be longer than uncompressed\n lastoffset = min(offsets[-1] + 2*bytecounts[-1], self._fh.size)\n for j in range(len(bytecounts) - 1):\n bytecounts[j] = offsets[j+1] - offsets[j]\n bytecounts[-1] = lastoffset - offsets[-1]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncheck if the file has MD Gel format.", "response": "def is_mdgel(self):\n \"\"\"File has MD Gel format.\"\"\"\n # TODO: this likely reads the second page from file\n try:\n ismdgel = self.pages[0].is_mdgel or self.pages[1].is_mdgel\n if ismdgel:\n self.is_uniform = False\n return ismdgel\n except IndexError:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn if file contains a uniform series of pages.", "response": "def is_uniform(self):\n \"\"\"Return if file contains a uniform series of pages.\"\"\"\n # the hashes of IFDs 0, 7, and -1 are the same\n pages = self.pages\n page = pages[0]\n if page.is_scanimage or page.is_nih:\n return True\n try:\n useframes = pages.useframes\n pages.useframes = False\n h = page.hash\n for i in (1, 7, -1):\n if pages[i].aspage().hash != h:\n return False\n except IndexError:\n return False\n finally:\n pages.useframes = useframes\n return True"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef is_appendable(self):\n # TODO: check other formats\n return not (self.is_lsm or self.is_stk or self.is_imagej or\n self.is_fluoview or self.is_micromanager)", "response": "Return if pages can be appended to file without corrupting."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef shaped_metadata(self):\n if not self.is_shaped:\n return None\n return tuple(json_description_metadata(s.pages[0].is_shaped)\n for s in self.series if s.kind.lower() == 'shaped')", "response": "Return tifffile metadata from JSON descriptions as dicts."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef stk_metadata(self):\n if not self.is_stk:\n return None\n page = self.pages[0]\n tags = page.tags\n result = {}\n result['NumberPlanes'] = tags['UIC2tag'].count\n if page.description:\n result['PlaneDescriptions'] = page.description.split('\\0')\n # result['plane_descriptions'] = stk_description_metadata(\n # page.image_description)\n if 'UIC1tag' in tags:\n result.update(tags['UIC1tag'].value)\n if 'UIC3tag' in tags:\n result.update(tags['UIC3tag'].value) # wavelengths\n if 'UIC4tag' in tags:\n result.update(tags['UIC4tag'].value) # override uic1 tags\n uic2tag = tags['UIC2tag'].value\n result['ZDistance'] = uic2tag['ZDistance']\n result['TimeCreated'] = uic2tag['TimeCreated']\n result['TimeModified'] = uic2tag['TimeModified']\n try:\n result['DatetimeCreated'] = numpy.array(\n [julian_datetime(*dt) for dt in\n zip(uic2tag['DateCreated'], uic2tag['TimeCreated'])],\n dtype='datetime64[ns]')\n result['DatetimeModified'] = numpy.array(\n [julian_datetime(*dt) for dt in\n zip(uic2tag['DateModified'], uic2tag['TimeModified'])],\n dtype='datetime64[ns]')\n except ValueError as exc:\n log.warning('STK metadata: %s: %s', exc.__class__.__name__, exc)\n return result", "response": "Return STK metadata from UIC tags as dict."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning consolidated ImageJ metadata as dict.", "response": "def imagej_metadata(self):\n \"\"\"Return consolidated ImageJ metadata as dict.\"\"\"\n if not self.is_imagej:\n return None\n page = self.pages[0]\n result = imagej_description_metadata(page.is_imagej)\n if 'IJMetadata' in page.tags:\n try:\n result.update(page.tags['IJMetadata'].value)\n except Exception:\n pass\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fluoview_metadata(self):\n if not self.is_fluoview:\n return None\n result = {}\n page = self.pages[0]\n result.update(page.tags['MM_Header'].value)\n # TODO: read stamps from all pages\n result['Stamp'] = page.tags['MM_Stamp'].value\n # skip parsing image description; not reliable\n # try:\n # t = fluoview_description_metadata(page.image_description)\n # if t is not None:\n # result['ImageDescription'] = t\n # except Exception as exc:\n # log.warning('FluoView metadata: '\n # 'failed to parse image description (%s)', str(exc))\n return result", "response": "Return consolidated FluoView metadata as dict."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning FEI metadata from SFEG or HELIOS tags as dict.", "response": "def fei_metadata(self):\n \"\"\"Return FEI metadata from SFEG or HELIOS tags as dict.\"\"\"\n if not self.is_fei:\n return None\n tags = self.pages[0].tags\n if 'FEI_SFEG' in tags:\n return tags['FEI_SFEG'].value\n if 'FEI_HELIOS' in tags:\n return tags['FEI_HELIOS'].value\n return None"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef sis_metadata(self):\n if not self.is_sis:\n return None\n tags = self.pages[0].tags\n result = {}\n try:\n result.update(tags['OlympusINI'].value)\n except Exception:\n pass\n try:\n result.update(tags['OlympusSIS'].value)\n except Exception:\n pass\n return result", "response": "Return Olympus SIS metadata from SIS and INI tags as dict."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef mdgel_metadata(self):\n for page in self.pages[:2]:\n if 'MDFileTag' in page.tags:\n tags = page.tags\n break\n else:\n return None\n result = {}\n for code in range(33445, 33453):\n name = TIFF.TAGS[code]\n if name not in tags:\n continue\n result[name[2:]] = tags[name].value\n return result", "response": "Return consolidated metadata from MD GEL tags as dict."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning consolidated MicroManager metadata as dict.", "response": "def micromanager_metadata(self):\n \"\"\"Return consolidated MicroManager metadata as dict.\"\"\"\n if not self.is_micromanager:\n return None\n # from file header\n result = read_micromanager_metadata(self._fh)\n # from tag\n result.update(self.pages[0].tags['MicroManagerMetadata'].value)\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn ScanImage non - varying frame and ROI metadata as dict.", "response": "def scanimage_metadata(self):\n \"\"\"Return ScanImage non-varying frame and ROI metadata as dict.\"\"\"\n if not self.is_scanimage:\n return None\n result = {}\n try:\n framedata, roidata = read_scanimage_metadata(self._fh)\n result['FrameData'] = framedata\n result.update(roidata)\n except ValueError:\n pass\n # TODO: scanimage_artist_metadata\n try:\n result['Description'] = scanimage_description_metadata(\n self.pages[0].description)\n except Exception as exc:\n log.warning('ScanImage metadata: %s: %s',\n exc.__class__.__name__, exc)\n return result"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nenabling or disable caching of pages or frames. Clear cache if False.", "response": "def cache(self, value):\n \"\"\"Enable or disable caching of pages/frames. Clear cache if False.\"\"\"\n value = bool(value)\n if self._cache and not value:\n self._clear()\n self._cache = value"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef keyframe(self, index):\n index = int(index)\n if index < 0:\n index %= len(self)\n if self._keyframe.index == index:\n return\n if index == 0:\n self._keyframe = self.pages[0]\n return\n if self._indexed or index < len(self.pages):\n page = self.pages[index]\n if isinstance(page, TiffPage):\n self._keyframe = page\n return\n if isinstance(page, TiffFrame):\n # remove existing TiffFrame\n self.pages[index] = page.offset\n # load TiffPage from file\n tiffpage = self._tiffpage\n self._tiffpage = TiffPage\n try:\n self._keyframe = self._getitem(index)\n finally:\n self._tiffpage = tiffpage\n # always cache keyframes\n self.pages[index] = self._keyframe", "response": "Set current keyframe. Load TiffPage from file if necessary."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreading all remaining pages from file.", "response": "def _load(self, keyframe=True):\n \"\"\"Read all remaining pages from file.\"\"\"\n if self._cached:\n return\n pages = self.pages\n if not pages:\n return\n if not self._indexed:\n self._seek(-1)\n if not self._cache:\n return\n fh = self.parent.filehandle\n if keyframe is not None:\n keyframe = self._keyframe\n for i, page in enumerate(pages):\n if isinstance(page, inttypes):\n fh.seek(page)\n page = self._tiffpage(self.parent, index=i, keyframe=keyframe)\n pages[i] = page\n self._cached = True"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _clear(self, fully=True):\n pages = self.pages\n if not pages:\n return\n self._keyframe = pages[0]\n if fully:\n # delete all but first TiffPage/TiffFrame\n for i, page in enumerate(pages[1:]):\n if not isinstance(page, inttypes) and page.offset is not None:\n pages[i+1] = page.offset\n elif TiffFrame is not TiffPage:\n # delete only TiffFrames\n for i, page in enumerate(pages):\n if isinstance(page, TiffFrame) and page.offset is not None:\n pages[i] = page.offset\n self._cached = False", "response": "Delete all but first page from cache. Set keyframe to first page."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nseek file to offset of page specified by index.", "response": "def _seek(self, index, maxpages=None):\n \"\"\"Seek file to offset of page specified by index.\"\"\"\n pages = self.pages\n lenpages = len(pages)\n if lenpages == 0:\n raise IndexError('index out of range')\n\n fh = self.parent.filehandle\n if fh.closed:\n raise ValueError('seek of closed file')\n\n if self._indexed or 0 <= index < lenpages:\n page = pages[index]\n offset = page if isinstance(page, inttypes) else page.offset\n fh.seek(offset)\n return\n\n tiff = self.parent.tiff\n offsetformat = tiff.ifdoffsetformat\n offsetsize = tiff.ifdoffsetsize\n tagnoformat = tiff.tagnoformat\n tagnosize = tiff.tagnosize\n tagsize = tiff.tagsize\n unpack = struct.unpack\n\n page = pages[-1]\n offset = page if isinstance(page, inttypes) else page.offset\n\n if maxpages is None:\n maxpages = 2**22\n while lenpages < maxpages:\n # read offsets to pages from file until index is reached\n fh.seek(offset)\n # skip tags\n try:\n tagno = unpack(tagnoformat, fh.read(tagnosize))[0]\n if tagno > 4096:\n raise TiffFileError(\n 'suspicious number of tags: %i' % tagno)\n except Exception:\n log.warning('TiffPages: corrupted tag list of page %i @ %i',\n lenpages, offset)\n del pages[-1]\n lenpages -= 1\n self._indexed = True\n break\n self._nextpageoffset = offset + tagnosize + tagno * tagsize\n fh.seek(self._nextpageoffset)\n\n # read offset to next page\n offset = unpack(offsetformat, fh.read(offsetsize))[0]\n if offset == 0:\n self._indexed = True\n break\n if offset >= fh.size:\n log.warning('TiffPages: invalid page offset (%i)', offset)\n self._indexed = True\n break\n\n pages.append(offset)\n lenpages += 1\n if 0 <= index < lenpages:\n break\n\n # detect some circular references\n if lenpages == 100:\n for p in pages[:-1]:\n if offset == (p if isinstance(p, inttypes) else p.offset):\n raise TiffFileError('invalid circular IFD reference')\n\n if index >= lenpages:\n raise IndexError('index out of range')\n\n page = pages[index]\n fh.seek(page if isinstance(page, inttypes) else page.offset)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _getlist(self, key=None, useframes=True, validate=True):\n getitem = self._getitem\n _useframes = self.useframes\n\n if key is None:\n key = iter(range(len(self)))\n elif isinstance(key, Iterable):\n key = iter(key)\n elif isinstance(key, slice):\n start, stop, _ = key.indices(2**31-1)\n if not self._indexed and max(stop, start) > len(self.pages):\n self._seek(-1)\n key = iter(range(*key.indices(len(self.pages))))\n elif isinstance(key, inttypes):\n # return single TiffPage\n self.useframes = False\n if key == 0:\n return [self.pages[key]]\n try:\n return [getitem(key)]\n finally:\n self.useframes = _useframes\n else:\n raise TypeError('key must be an integer, slice, or iterable')\n\n # use first page as keyframe\n keyframe = self._keyframe\n self.keyframe = next(key)\n if validate:\n validate = self._keyframe.hash\n if useframes:\n self.useframes = True\n try:\n pages = [getitem(i, validate) for i in key]\n pages.insert(0, self._keyframe)\n finally:\n # restore state\n self._keyframe = keyframe\n if useframes:\n self.useframes = _useframes\n\n return pages", "response": "Return specified pages as list of TiffPages or TiffFrames."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _getitem(self, key, validate=False):\n key = int(key)\n pages = self.pages\n\n if key < 0:\n key %= len(self)\n elif self._indexed and key >= len(pages):\n raise IndexError('index out of range')\n\n if key < len(pages):\n page = pages[key]\n if self._cache:\n if not isinstance(page, inttypes):\n if validate and validate != page.hash:\n raise RuntimeError('page hash mismatch')\n return page\n elif isinstance(page, (TiffPage, self._tiffpage)):\n if validate and validate != page.hash:\n raise RuntimeError('page hash mismatch')\n return page\n\n self._seek(key)\n page = self._tiffpage(self.parent, index=key, keyframe=self._keyframe)\n if validate and validate != page.hash:\n raise RuntimeError('page hash mismatch')\n if self._cache:\n pages[key] = page\n return page", "response": "Return specified page from cache or file."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreads image data from file and return as numpy array.", "response": "def asarray(self, out=None, squeeze=True, lock=None, reopen=True,\n maxsize=None, maxworkers=None, validate=True):\n \"\"\"Read image data from file and return as numpy array.\n\n Raise ValueError if format is unsupported.\n\n Parameters\n ----------\n out : numpy.ndarray, str, or file-like object\n Buffer where image data will be saved.\n If None (default), a new array will be created.\n If numpy.ndarray, a writable array of compatible dtype and shape.\n If 'memmap', directly memory-map the image data in the TIFF file\n if possible; else create a memory-mapped array in a temporary file.\n If str or open file, the file name or file object used to\n create a memory-map to an array stored in a binary file on disk.\n squeeze : bool\n If True (default), all length-1 dimensions (except X and Y) are\n squeezed out from the array.\n If False, the shape of the returned array might be different from\n the page.shape.\n lock : {RLock, NullContext}\n A reentrant lock used to syncronize reads from file.\n If None (default), the lock of the parent's filehandle is used.\n reopen : bool\n If True (default) and the parent file handle is closed, the file\n is temporarily re-opened and closed if no exception occurs.\n maxsize: int\n Maximum size of data before a ValueError is raised.\n Can be used to catch DOS. Default: 16 TB.\n maxworkers : int or None\n Maximum number of threads to concurrently decode tile data.\n If None (default), up to half the CPU cores are used for\n compressed tiles.\n See remarks in TiffFile.asarray.\n validate : bool\n If True (default), validate various parameters.\n If None, only validate parameters and return None.\n\n Returns\n -------\n numpy.ndarray\n Numpy array of decompressed, depredicted, and unpacked image data\n read from Strip/Tile Offsets/ByteCounts, formatted according to\n shape and dtype metadata found in tags and parameters.\n Photometric conversion, pre-multiplied alpha, orientation, and\n colorimetry corrections are not applied. Specifically, CMYK images\n are not converted to RGB, MinIsWhite images are not inverted,\n and color palettes are not applied.\n\n \"\"\"\n # properties from TiffPage or TiffFrame\n fh = self.parent.filehandle\n byteorder = self.parent.tiff.byteorder\n offsets, bytecounts = self._offsetscounts\n self_ = self\n self = self.keyframe # self or keyframe\n\n if not self._shape or product(self._shape) == 0:\n return None\n\n tags = self.tags\n\n if validate or validate is None:\n if maxsize is None:\n maxsize = 2**44\n if maxsize and product(self._shape) > maxsize:\n raise ValueError('data are too large %s' % str(self._shape))\n if self.dtype is None:\n raise ValueError('data type not supported: %s%i' % (\n self.sampleformat, self.bitspersample))\n if self.compression not in TIFF.DECOMPESSORS:\n raise ValueError(\n 'cannot decompress %s' % self.compression.name)\n if 'SampleFormat' in tags:\n tag = tags['SampleFormat']\n if tag.count != 1 and any((i-tag.value[0] for i in tag.value)):\n raise ValueError(\n 'sample formats do not match %s' % tag.value)\n if self.is_subsampled and (self.compression not in (6, 7) or\n self.planarconfig == 2):\n raise NotImplementedError('chroma subsampling not supported')\n if validate is None:\n return None\n\n lock = fh.lock if lock is None else lock\n with lock:\n closed = fh.closed\n if closed:\n if reopen:\n fh.open()\n else:\n raise IOError('file handle is closed')\n\n dtype = self._dtype\n shape = self._shape\n imagewidth = self.imagewidth\n imagelength = self.imagelength\n imagedepth = self.imagedepth\n bitspersample = self.bitspersample\n typecode = byteorder + dtype.char\n lsb2msb = self.fillorder == 2\n istiled = self.is_tiled\n\n if istiled:\n tilewidth = self.tilewidth\n tilelength = self.tilelength\n tiledepth = self.tiledepth\n tw = (imagewidth + tilewidth - 1) // tilewidth\n tl = (imagelength + tilelength - 1) // tilelength\n td = (imagedepth + tiledepth - 1) // tiledepth\n tiledshape = (td, tl, tw)\n tileshape = (tiledepth, tilelength, tilewidth, shape[-1])\n runlen = tilewidth\n else:\n runlen = imagewidth\n\n if self.planarconfig == 1:\n runlen *= self.samplesperpixel\n\n if isinstance(out, str) and out == 'memmap' and self.is_memmappable:\n # direct memory map array in file\n with lock:\n result = fh.memmap_array(typecode, shape, offset=offsets[0])\n elif self.is_contiguous:\n # read contiguous bytes to array\n if out is not None:\n out = create_output(out, shape, dtype)\n with lock:\n fh.seek(offsets[0])\n result = fh.read_array(typecode, product(shape), out=out)\n if lsb2msb:\n bitorder_decode(result, out=result)\n else:\n # decompress, unpack,... individual strips or tiles\n result = create_output(out, shape, dtype)\n\n decompress = TIFF.DECOMPESSORS[self.compression]\n\n if self.compression in (6, 7): # COMPRESSION.JPEG\n colorspace = None\n outcolorspace = None\n jpegtables = None\n if lsb2msb:\n log.warning('TiffPage.asarray: disabling LSB2MSB for JPEG')\n lsb2msb = False\n if 'JPEGTables' in tags:\n # load JPEGTables from TiffFrame\n jpegtables = self_._gettags({347}, lock=lock)[0][1].value\n # TODO: obtain table from OJPEG tags\n # elif ('JPEGInterchangeFormat' in tags and\n # 'JPEGInterchangeFormatLength' in tags and\n # tags['JPEGInterchangeFormat'].value != offsets[0]):\n # fh.seek(tags['JPEGInterchangeFormat'].value)\n # fh.read(tags['JPEGInterchangeFormatLength'].value)\n if 'ExtraSamples' in tags:\n pass\n elif self.photometric == 6:\n # YCBCR -> RGB\n outcolorspace = 'RGB'\n elif self.photometric == 2:\n if self.planarconfig == 2:\n # TODO: decode JPEG to planar RGB\n raise NotImplementedError(\n 'cannot decode JPEG to planar RGB')\n colorspace = outcolorspace = 'RGB'\n else:\n outcolorspace = TIFF.PHOTOMETRIC(self.photometric).name\n if istiled:\n heightwidth = tilelength, tilewidth\n else:\n heightwidth = imagelength, imagewidth\n\n def decompress(data, bitspersample=bitspersample,\n jpegtables=jpegtables, colorspace=colorspace,\n outcolorspace=outcolorspace, shape=heightwidth,\n out=None, _decompress=decompress):\n return _decompress(data, bitspersample, jpegtables,\n colorspace, outcolorspace, shape, out)\n\n def unpack(data):\n return data.reshape(-1)\n\n elif bitspersample in (8, 16, 32, 64, 128):\n if (bitspersample * runlen) % 8:\n raise ValueError('data and sample size mismatch')\n if self.predictor == 3: # PREDICTOR.FLOATINGPOINT\n # the floating-point horizontal differencing decoder\n # needs the raw byte order\n typecode = dtype.char\n\n def unpack(data, typecode=typecode, out=None):\n try:\n # read only numpy array\n return numpy.frombuffer(data, typecode)\n except ValueError:\n # strips may be missing EOI\n # log.warning('TiffPage.asarray: ...')\n bps = bitspersample // 8\n xlen = (len(data) // bps) * bps\n return numpy.frombuffer(data[:xlen], typecode)\n\n elif isinstance(bitspersample, tuple):\n\n def unpack(data, out=None):\n return unpack_rgb(data, typecode, bitspersample)\n\n else:\n\n def unpack(data, out=None):\n return packints_decode(data, typecode, bitspersample,\n runlen)\n\n # TODO: store decode function for future use\n # TODO: unify tile and strip decoding\n if istiled:\n unpredict = TIFF.UNPREDICTORS[self.predictor]\n\n def decode(tile, tileindex):\n return tile_decode(tile, tileindex, tileshape, tiledshape,\n lsb2msb, decompress, unpack, unpredict,\n result[0])\n\n tileiter = buffered_read(fh, lock, offsets, bytecounts)\n if maxworkers is None:\n maxworkers = 0 if self.compression > 1 else 1\n if maxworkers == 0:\n import multiprocessing # noqa: delay import\n maxworkers = multiprocessing.cpu_count() // 2\n if maxworkers < 2:\n for i, tile in enumerate(tileiter):\n decode(tile, i)\n else:\n # decode first tile un-threaded to catch exceptions\n decode(next(tileiter), 0)\n with ThreadPoolExecutor(maxworkers) as executor:\n executor.map(decode, tileiter, range(1, len(offsets)))\n\n else:\n stripsize = self.rowsperstrip * self.imagewidth\n if self.planarconfig == 1:\n stripsize *= self.samplesperpixel\n outsize = stripsize * self.dtype.itemsize\n result = result.reshape(-1)\n index = 0\n for strip in buffered_read(fh, lock, offsets, bytecounts):\n if lsb2msb:\n strip = bitorder_decode(strip, out=strip)\n strip = decompress(strip, out=outsize)\n strip = unpack(strip)\n size = min(result.size, strip.size, stripsize,\n result.size - index)\n result[index:index+size] = strip[:size]\n del strip\n index += size\n\n result.shape = self._shape\n\n if self.predictor != 1 and not (istiled and not self.is_contiguous):\n unpredict = TIFF.UNPREDICTORS[self.predictor]\n result = unpredict(result, axis=-2, out=result)\n\n if squeeze:\n try:\n result.shape = self.shape\n except ValueError:\n log.warning('TiffPage.asarray: failed to reshape %s to %s',\n result.shape, self.shape)\n\n if closed:\n # TODO: file should remain open if an exception occurred above\n fh.close()\n return result"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef asrgb(self, uint8=False, alpha=None, colormap=None,\n dmin=None, dmax=None, **kwargs):\n \"\"\"Return image data as RGB(A).\n\n Work in progress.\n\n \"\"\"\n data = self.asarray(**kwargs)\n self = self.keyframe # self or keyframe\n photometric = self.photometric\n PHOTOMETRIC = TIFF.PHOTOMETRIC\n\n if photometric == PHOTOMETRIC.PALETTE:\n colormap = self.colormap\n if (colormap.shape[1] < 2**self.bitspersample or\n self.dtype.char not in 'BH'):\n raise ValueError('cannot apply colormap')\n if uint8:\n if colormap.max() > 255:\n colormap >>= 8\n colormap = colormap.astype('uint8')\n if 'S' in self.axes:\n data = data[..., 0] if self.planarconfig == 1 else data[0]\n data = apply_colormap(data, colormap)\n\n elif photometric == PHOTOMETRIC.RGB:\n if 'ExtraSamples' in self.tags:\n if alpha is None:\n alpha = TIFF.EXTRASAMPLE\n extrasamples = self.extrasamples\n if self.tags['ExtraSamples'].count == 1:\n extrasamples = (extrasamples,)\n for i, exs in enumerate(extrasamples):\n if exs in alpha:\n if self.planarconfig == 1:\n data = data[..., [0, 1, 2, 3+i]]\n else:\n data = data[:, [0, 1, 2, 3+i]]\n break\n else:\n if self.planarconfig == 1:\n data = data[..., :3]\n else:\n data = data[:, :3]\n # TODO: convert to uint8?\n\n elif photometric == PHOTOMETRIC.MINISBLACK:\n raise NotImplementedError()\n elif photometric == PHOTOMETRIC.MINISWHITE:\n raise NotImplementedError()\n elif photometric == PHOTOMETRIC.SEPARATED:\n raise NotImplementedError()\n else:\n raise NotImplementedError()\n return data", "response": "Return image data as RGB."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns list of ( code TiffTag", "response": "def _gettags(self, codes=None, lock=None):\n \"\"\"Return list of (code, TiffTag).\"\"\"\n tags = []\n for tag in self.tags.values():\n code = tag.code\n if not codes or code in codes:\n tags.append((code, tag))\n return tags"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef hash(self):\n return hash(\n self._shape + (\n self.tilewidth, self.tilelength, self.tiledepth,\n self.bitspersample, self.fillorder, self.predictor,\n self.extrasamples, self.photometric, self.compression,\n self.planarconfig))", "response": "Return checksum to identify pages in same series."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning simplified offsets and bytecounts.", "response": "def _offsetscounts(self):\n \"\"\"Return simplified offsets and bytecounts.\"\"\"\n if self.is_contiguous:\n offset, bytecount = self.is_contiguous\n return [offset], [bytecount]\n if self.is_tiled:\n return self.dataoffsets, self.databytecounts\n return clean_offsetscounts(self.dataoffsets, self.databytecounts)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef is_contiguous(self):\n if (self.compression != 1\n or self.bitspersample not in (8, 16, 32, 64)):\n return None\n if 'TileWidth' in self.tags:\n if (self.imagewidth != self.tilewidth or\n self.imagelength % self.tilelength or\n self.tilewidth % 16 or self.tilelength % 16):\n return None\n if ('ImageDepth' in self.tags and 'TileDepth' in self.tags and\n (self.imagelength != self.tilelength or\n self.imagedepth % self.tiledepth)):\n return None\n\n offsets = self.dataoffsets\n bytecounts = self.databytecounts\n if len(offsets) == 1:\n return offsets[0], bytecounts[0]\n if self.is_stk or all((offsets[i] + bytecounts[i] == offsets[i+1] or\n bytecounts[i+1] == 0) # no data/ignore offset\n for i in range(len(offsets)-1)):\n return offsets[0], sum(bytecounts)\n return None", "response": "Return offset and size of contiguous data else None."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn True if page s image data are stored in final form.", "response": "def is_final(self):\n \"\"\"Return if page's image data are stored in final form.\n\n Excludes byte-swapping.\n\n \"\"\"\n return (self.is_contiguous and self.fillorder == 1 and\n self.predictor == 1 and not self.is_subsampled)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning if page s image data in file can be memory - mapped.", "response": "def is_memmappable(self):\n \"\"\"Return if page's image data in file can be memory-mapped.\"\"\"\n return (self.parent.filehandle.is_file and self.is_final and\n # (self.bitspersample == 8 or self.parent.isnative) and\n self.is_contiguous[0] % self.dtype.itemsize == 0)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns set of flags.", "response": "def flags(self):\n \"\"\"Return set of flags.\"\"\"\n return set((name.lower() for name in sorted(TIFF.FILE_FLAGS)\n if getattr(self, 'is_' + name)))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef andor_tags(self):\n if not self.is_andor:\n return None\n tags = self.tags\n result = {'Id': tags['AndorId'].value}\n for tag in list(self.tags.values()):\n code = tag.code\n if not 4864 < code < 5031:\n continue\n value = tag.value\n name = tag.name[5:] if len(tag.name) > 5 else tag.name\n result[name] = value\n del tags[tag.name]\n return result", "response": "Return consolidated metadata from Andor tags as dict."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef epics_tags(self):\n if not self.is_epics:\n return None\n result = {}\n tags = self.tags\n for tag in list(self.tags.values()):\n code = tag.code\n if not 65000 <= code < 65500:\n continue\n value = tag.value\n if code == 65000:\n result['timeStamp'] = datetime.datetime.fromtimestamp(\n float(value))\n elif code == 65001:\n result['uniqueID'] = int(value)\n elif code == 65002:\n result['epicsTSSec'] = int(value)\n elif code == 65003:\n result['epicsTSNsec'] = int(value)\n else:\n key, value = value.split(':', 1)\n result[key] = astype(value)\n del tags[tag.name]\n return result", "response": "Return consolidated metadata from EPICS areaDetector tags as dict."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns consolidated metadata from Hamamatsu NDPI as dict.", "response": "def ndpi_tags(self):\n \"\"\"Return consolidated metadata from Hamamatsu NDPI as dict.\"\"\"\n if not self.is_ndpi:\n return None\n tags = self.tags\n result = {}\n for name in ('Make', 'Model', 'Software'):\n result[name] = tags[name].value\n for code, name in TIFF.NDPI_TAGS.items():\n code = str(code)\n if code in tags:\n result[name] = tags[code].value\n # del tags[code]\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning consolidated metadata from GeoTIFF tags as dict.", "response": "def geotiff_tags(self):\n \"\"\"Return consolidated metadata from GeoTIFF tags as dict.\"\"\"\n if not self.is_geotiff:\n return None\n tags = self.tags\n\n gkd = tags['GeoKeyDirectoryTag'].value\n if gkd[0] != 1:\n log.warning('GeoTIFF tags: invalid GeoKeyDirectoryTag')\n return {}\n\n result = {\n 'KeyDirectoryVersion': gkd[0],\n 'KeyRevision': gkd[1],\n 'KeyRevisionMinor': gkd[2],\n # 'NumberOfKeys': gkd[3],\n }\n # deltags = ['GeoKeyDirectoryTag']\n geokeys = TIFF.GEO_KEYS\n geocodes = TIFF.GEO_CODES\n for index in range(gkd[3]):\n keyid, tagid, count, offset = gkd[4 + index * 4: index * 4 + 8]\n keyid = geokeys.get(keyid, keyid)\n if tagid == 0:\n value = offset\n else:\n tagname = TIFF.TAGS[tagid]\n # deltags.append(tagname)\n value = tags[tagname].value[offset: offset + count]\n if tagid == 34737 and count > 1 and value[-1] == '|':\n value = value[:-1]\n value = value if count > 1 else value[0]\n if keyid in geocodes:\n try:\n value = geocodes[keyid](value)\n except Exception:\n pass\n result[keyid] = value\n\n if 'IntergraphMatrixTag' in tags:\n value = tags['IntergraphMatrixTag'].value\n value = numpy.array(value)\n if len(value) == 16:\n value = value.reshape((4, 4)).tolist()\n result['IntergraphMatrix'] = value\n if 'ModelPixelScaleTag' in tags:\n value = numpy.array(tags['ModelPixelScaleTag'].value).tolist()\n result['ModelPixelScale'] = value\n if 'ModelTiepointTag' in tags:\n value = tags['ModelTiepointTag'].value\n value = numpy.array(value).reshape((-1, 6)).squeeze().tolist()\n result['ModelTiepoint'] = value\n if 'ModelTransformationTag' in tags:\n value = tags['ModelTransformationTag'].value\n value = numpy.array(value).reshape((4, 4)).tolist()\n result['ModelTransformation'] = value\n # if 'ModelPixelScaleTag' in tags and 'ModelTiepointTag' in tags:\n # sx, sy, sz = tags['ModelPixelScaleTag'].value\n # tiepoints = tags['ModelTiepointTag'].value\n # transforms = []\n # for tp in range(0, len(tiepoints), 6):\n # i, j, k, x, y, z = tiepoints[tp:tp+6]\n # transforms.append([\n # [sx, 0.0, 0.0, x - i * sx],\n # [0.0, -sy, 0.0, y + j * sy],\n # [0.0, 0.0, sz, z - k * sz],\n # [0.0, 0.0, 0.0, 1.0]])\n # if len(tiepoints) == 6:\n # transforms = transforms[0]\n # result['ModelTransformation'] = transforms\n\n if 'RPCCoefficientTag' in tags:\n rpcc = tags['RPCCoefficientTag'].value\n result['RPCCoefficient'] = {\n 'ERR_BIAS': rpcc[0],\n 'ERR_RAND': rpcc[1],\n 'LINE_OFF': rpcc[2],\n 'SAMP_OFF': rpcc[3],\n 'LAT_OFF': rpcc[4],\n 'LONG_OFF': rpcc[5],\n 'HEIGHT_OFF': rpcc[6],\n 'LINE_SCALE': rpcc[7],\n 'SAMP_SCALE': rpcc[8],\n 'LAT_SCALE': rpcc[9],\n 'LONG_SCALE': rpcc[10],\n 'HEIGHT_SCALE': rpcc[11],\n 'LINE_NUM_COEFF': rpcc[12:33],\n 'LINE_DEN_COEFF ': rpcc[33:53],\n 'SAMP_NUM_COEFF': rpcc[53:73],\n 'SAMP_DEN_COEFF': rpcc[73:]}\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn ImageJ description if exists else None.", "response": "def is_imagej(self):\n \"\"\"Return ImageJ description if exists, else None.\"\"\"\n for description in (self.description, self.description1):\n if not description:\n return None\n if description[:7] == 'ImageJ=':\n return description\n return None"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn description containing array shape if exists else None.", "response": "def is_shaped(self):\n \"\"\"Return description containing array shape if exists, else None.\"\"\"\n for description in (self.description, self.description1):\n if not description:\n return None\n if description[:1] == '{' and '\"shape\":' in description:\n return description\n if description[:6] == 'shape=':\n return description\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn True if the page contains MDS MetaSeries metadata in ImageDescription tag.", "response": "def is_metaseries(self):\n \"\"\"Page contains MDS MetaSeries metadata in ImageDescription tag.\"\"\"\n if self.index > 1 or self.software != 'MetaSeries':\n return False\n d = self.description\n return d.startswith('<MetaData>') and d.endswith('</MetaData>')"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn True if the page contains OME - XML in ImageDescription tag.", "response": "def is_ome(self):\n \"\"\"Page contains OME-XML in ImageDescription tag.\"\"\"\n if self.index > 1 or not self.description:\n return False\n d = self.description\n return d[:14] == '<?xml version=' and d[-6:] == '</OME>'"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef is_scn(self):\n if self.index > 1 or not self.description:\n return False\n d = self.description\n return d[:14] == '<?xml version=' and d[-6:] == '</scn>'", "response": "Return True if the page contains Leica SCN XML in ImageDescription tag."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn list of ( code TiffTag ) from file.", "response": "def _gettags(self, codes=None, lock=None):\n \"\"\"Return list of (code, TiffTag) from file.\"\"\"\n fh = self.parent.filehandle\n tiff = self.parent.tiff\n unpack = struct.unpack\n lock = NullContext() if lock is None else lock\n tags = []\n\n with lock:\n fh.seek(self.offset)\n try:\n tagno = unpack(tiff.tagnoformat, fh.read(tiff.tagnosize))[0]\n if tagno > 4096:\n raise TiffFileError('suspicious number of tags')\n except Exception:\n raise TiffFileError(\n 'corrupted page list at offset %i' % self.offset)\n\n tagoffset = self.offset + tiff.tagnosize # fh.tell()\n tagsize = tiff.tagsize\n tagindex = -tagsize\n codeformat = tiff.tagformat1[:2]\n tagbytes = fh.read(tagsize * tagno)\n\n for _ in range(tagno):\n tagindex += tagsize\n code = unpack(codeformat, tagbytes[tagindex:tagindex+2])[0]\n if codes and code not in codes:\n continue\n try:\n tag = TiffTag(self.parent,\n tagbytes[tagindex:tagindex+tagsize],\n tagoffset+tagindex)\n except TiffFileError as exc:\n log.warning('%s: %s', exc.__class__.__name__, exc)\n continue\n tags.append((code, tag))\n\n return tags"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef aspage(self):\n if self.offset is None:\n raise ValueError('cannot return virtual frame as page.')\n self.parent.filehandle.seek(self.offset)\n return TiffPage(self.parent, index=self.index)", "response": "Return a TiffPage from file."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef asarray(self, *args, **kwargs):\n # TODO: fix TypeError on Python 2\n # \"TypeError: unbound method asarray() must be called with TiffPage\n # instance as first argument (got TiffFrame instance instead)\"\n if self._keyframe is None:\n raise RuntimeError('keyframe not set')\n kwargs['validate'] = False\n return TiffPage.asarray(self, *args, **kwargs)", "response": "Read image data from file and return as numpy array."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef asrgb(self, *args, **kwargs):\n if self._keyframe is None:\n raise RuntimeError('keyframe not set')\n kwargs['validate'] = False\n return TiffPage.asrgb(self, *args, **kwargs)", "response": "Read image data from file and return RGB image as numpy array."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning offset and size of contiguous data else None.", "response": "def is_contiguous(self):\n \"\"\"Return offset and size of contiguous data, else None.\"\"\"\n if self._keyframe is None:\n raise RuntimeError('keyframe not set')\n if self._keyframe.is_contiguous:\n return self._offsetscounts[0][0], self._keyframe.is_contiguous[1]\n return None"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef name(self):\n try:\n return TIFF.TAGS[self.code]\n except KeyError:\n return str(self.code)", "response": "Return name of tag from TIFF. TAGS registry."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _fix_lsm_bitspersample(self, parent):\n if self.code != 258 or self.count != 2:\n return\n # TODO: test this case; need example file\n log.warning('TiffTag %i: correcting LSM bitspersample tag', self.code)\n value = struct.pack('<HH', *self.value)\n self.valueoffset = struct.unpack('<I', value)[0]\n parent.filehandle.seek(self.valueoffset)\n self.value = struct.unpack('<HH', parent.filehandle.read(4))", "response": "Correct the LSM bitspersample tag."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning image data from series of TIFF pages as numpy array.", "response": "def asarray(self, out=None):\n \"\"\"Return image data from series of TIFF pages as numpy array.\"\"\"\n if self.parent:\n result = self.parent.asarray(series=self, out=out)\n if self.transform is not None:\n result = self.transform(result)\n return result\n return None"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn offset to series data in file if any.", "response": "def offset(self):\n \"\"\"Return offset to series data in file, if any.\"\"\"\n if not self._pages:\n return None\n\n pos = 0\n for page in self._pages:\n if page is None:\n return None\n if not page.is_final:\n return None\n if not pos:\n pos = page.is_contiguous[0] + page.is_contiguous[1]\n continue\n if pos != page.is_contiguous[0]:\n return None\n pos += page.is_contiguous[1]\n\n page = self._pages[0]\n offset = page.is_contiguous[0]\n if (page.is_imagej or page.is_shaped) and len(self._pages) == 1:\n # truncated files\n return offset\n if pos == offset + product(self.shape) * self.dtype.itemsize:\n return offset\n return None"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _getitem(self, key):\n key = int(key)\n if key < 0:\n key %= self._len\n if len(self._pages) == 1 and 0 < key < self._len:\n index = self._pages[0].index\n return self.parent.pages._getitem(index + key)\n return self._pages[key]", "response": "Return specified page of series from cache or file."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreading image data from files and return as numpy array.", "response": "def asarray(self, file=None, out=None, **kwargs):\n \"\"\"Read image data from files and return as numpy array.\n\n The kwargs parameters are passed to the imread function.\n\n Raise IndexError or ValueError if image shapes do not match.\n\n \"\"\"\n if file is not None:\n if isinstance(file, int):\n return self.imread(self.files[file], **kwargs)\n return self.imread(file, **kwargs)\n\n im = self.imread(self.files[0], **kwargs)\n shape = self.shape + im.shape\n result = create_output(out, shape, dtype=im.dtype)\n result = result.reshape(-1, *im.shape)\n for index, fname in zip(self._indices, self.files):\n index = [i-j for i, j in zip(index, self._startindex)]\n index = numpy.ravel_multi_index(index, self.shape)\n im = self.imread(fname, **kwargs)\n result[index] = im\n result.shape = shape\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nparses the file names and return a tuple of axes and shape.", "response": "def _parse(self):\n \"\"\"Get axes and shape from file names.\"\"\"\n if not self.pattern:\n raise TiffSequence.ParseError('invalid pattern')\n pattern = re.compile(self.pattern, re.IGNORECASE | re.VERBOSE)\n matches = pattern.findall(os.path.split(self.files[0])[-1])\n if not matches:\n raise TiffSequence.ParseError('pattern does not match file names')\n matches = matches[-1]\n if len(matches) % 2:\n raise TiffSequence.ParseError(\n 'pattern does not match axis name and index')\n axes = ''.join(m for m in matches[::2] if m)\n if not axes:\n raise TiffSequence.ParseError('pattern does not match file names')\n\n indices = []\n for fname in self.files:\n fname = os.path.split(fname)[-1]\n matches = pattern.findall(fname)[-1]\n if axes != ''.join(m for m in matches[::2] if m):\n raise ValueError('axes do not match within image sequence')\n indices.append([int(m) for m in matches[1::2] if m])\n shape = tuple(numpy.max(indices, axis=0))\n startindex = tuple(numpy.min(indices, axis=0))\n shape = tuple(i-j+1 for i, j in zip(shape, startindex))\n if product(shape) != len(self.files):\n log.warning(\n 'TiffSequence: files are missing. Missing data are zeroed')\n\n self.axes = axes.upper()\n self.shape = shape\n self._indices = indices\n self._startindex = startindex"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef open(self):\n if self._fh:\n return # file is open\n\n if isinstance(self._file, pathlib.Path):\n self._file = str(self._file)\n if isinstance(self._file, basestring):\n # file name\n self._file = os.path.realpath(self._file)\n self._dir, self._name = os.path.split(self._file)\n self._fh = open(self._file, self._mode)\n self._close = True\n if self._offset is None:\n self._offset = 0\n elif isinstance(self._file, FileHandle):\n # FileHandle\n self._fh = self._file._fh\n if self._offset is None:\n self._offset = 0\n self._offset += self._file._offset\n self._close = False\n if not self._name:\n if self._offset:\n name, ext = os.path.splitext(self._file._name)\n self._name = '%s@%i%s' % (name, self._offset, ext)\n else:\n self._name = self._file._name\n if self._mode and self._mode != self._file._mode:\n raise ValueError('FileHandle has wrong mode')\n self._mode = self._file._mode\n self._dir = self._file._dir\n elif hasattr(self._file, 'seek'):\n # binary stream: open file, BytesIO\n try:\n self._file.tell()\n except Exception:\n raise ValueError('binary stream is not seekable')\n self._fh = self._file\n if self._offset is None:\n self._offset = self._file.tell()\n self._close = False\n if not self._name:\n try:\n self._dir, self._name = os.path.split(self._fh.name)\n except AttributeError:\n self._name = 'Unnamed binary stream'\n try:\n self._mode = self._fh.mode\n except AttributeError:\n pass\n else:\n raise ValueError('The first parameter must be a file name, '\n 'seekable binary stream, or FileHandle')\n\n if self._offset:\n self._fh.seek(self._offset)\n\n if self._size is None:\n pos = self._fh.tell()\n self._fh.seek(self._offset, 2)\n self._size = self._fh.tell()\n self._fh.seek(pos)\n\n try:\n self._fh.fileno()\n self.is_file = True\n except Exception:\n self.is_file = False", "response": "Open or re - open file."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef read(self, size=-1):\n if size < 0 and self._offset:\n size = self._size\n return self._fh.read(size)", "response": "Read size bytes from file."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef memmap_array(self, dtype, shape, offset=0, mode='r', order='C'):\n if not self.is_file:\n raise ValueError('Cannot memory-map file without fileno')\n return numpy.memmap(self._fh, dtype=dtype, mode=mode,\n offset=self._offset + offset,\n shape=shape, order=order)", "response": "Return numpy. memmap of data stored in file."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef read_array(self, dtype, count=-1, out=None):\n fh = self._fh\n dtype = numpy.dtype(dtype)\n\n if count < 0:\n size = self._size if out is None else out.nbytes\n count = size // dtype.itemsize\n else:\n size = count * dtype.itemsize\n\n result = numpy.empty(count, dtype) if out is None else out\n\n if result.nbytes != size:\n raise ValueError('size mismatch')\n\n n = fh.readinto(result)\n if n != size:\n raise ValueError('failed to read %i bytes' % size)\n\n if not result.dtype.isnative:\n if not dtype.isnative:\n result.byteswap(True)\n result = result.newbyteorder()\n elif result.dtype.isnative != dtype.isnative:\n result.byteswap(True)\n\n if out is not None:\n if hasattr(out, 'flush'):\n out.flush()\n\n return result", "response": "Return numpy array from file in native byte order."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns numpy record from file.", "response": "def read_record(self, dtype, shape=1, byteorder=None):\n \"\"\"Return numpy record from file.\"\"\"\n rec = numpy.rec\n try:\n record = rec.fromfile(self._fh, dtype, shape, byteorder=byteorder)\n except Exception:\n dtype = numpy.dtype(dtype)\n if shape is None:\n shape = self._size // dtype.itemsize\n size = product(sequence(shape)) * dtype.itemsize\n data = self._fh.read(size)\n record = rec.fromstring(data, dtype, shape, byteorder=byteorder)\n return record[0] if shape == 1 else record"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nappend size bytes to file.", "response": "def write_empty(self, size):\n \"\"\"Append size bytes to file. Position must be at end of file.\"\"\"\n if size < 1:\n return\n self._fh.seek(size-1, 1)\n self._fh.write(b'\\x00')"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef write_array(self, data):\n try:\n data.tofile(self._fh)\n except Exception:\n # BytesIO\n self._fh.write(data.tostring())", "response": "Write numpy array to binary file."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting file s current position.", "response": "def seek(self, offset, whence=0):\n \"\"\"Set file's current position.\"\"\"\n if self._offset:\n if whence == 0:\n self._fh.seek(self._offset + offset, whence)\n return\n if whence == 2 and self._size > 0:\n self._fh.seek(self._offset + self._size + offset, 0)\n return\n self._fh.seek(offset, whence)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nre-open file if necessary.", "response": "def open(self, filehandle):\n \"\"\"Re-open file if necessary.\"\"\"\n with self.lock:\n if filehandle in self.files:\n self.files[filehandle] += 1\n elif filehandle.closed:\n filehandle.open()\n self.files[filehandle] = 1\n self.past.append(filehandle)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef close(self, filehandle):\n with self.lock:\n if filehandle in self.files:\n self.files[filehandle] -= 1\n # trim the file cache\n index = 0\n size = len(self.past)\n while size > self.size and index < size:\n filehandle = self.past[index]\n if self.files[filehandle] == 0:\n filehandle.close()\n del self.files[filehandle]\n del self.past[index]\n size -= 1\n else:\n index += 1", "response": "Close openend file if no longer used."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncloses all opened files if not in use.", "response": "def clear(self):\n \"\"\"Close all opened files if not in use.\"\"\"\n with self.lock:\n for filehandle, refcount in list(self.files.items()):\n if refcount == 0:\n filehandle.close()\n del self.files[filehandle]\n del self.past[self.past.index(filehandle)]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef grab_literal(template, l_del):\n\n global _CURRENT_LINE\n\n try:\n # Look for the next tag and move the template to it\n literal, template = template.split(l_del, 1)\n _CURRENT_LINE += literal.count('\\n')\n return (literal, template)\n\n # There are no more tags in the template?\n except ValueError:\n # Then the rest of the template is a literal\n return (template, '')", "response": "Parse a literal from the template"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndoing a preliminary check to see if a tag could be a standalone", "response": "def l_sa_check(template, literal, is_standalone):\n \"\"\"Do a preliminary check to see if a tag could be a standalone\"\"\"\n\n # If there is a newline, or the previous tag was a standalone\n if literal.find('\\n') != -1 or is_standalone:\n padding = literal.split('\\n')[-1]\n\n # If all the characters since the last newline are spaces\n if padding.isspace() or padding == '':\n # Then the next tag could be a standalone\n return True\n else:\n # Otherwise it can't be\n return False"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndoing a final checkto see if a tag could be a standalone", "response": "def r_sa_check(template, tag_type, is_standalone):\n \"\"\"Do a final checkto see if a tag could be a standalone\"\"\"\n\n # Check right side if we might be a standalone\n if is_standalone and tag_type not in ['variable', 'no escape']:\n on_newline = template.split('\\n', 1)\n\n # If the stuff to the right of us are spaces we're a standalone\n if on_newline[0].isspace() or not on_newline[0]:\n return True\n else:\n return False\n\n # If we're a tag can't be a standalone\n else:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef parse_tag(template, l_del, r_del):\n global _CURRENT_LINE\n global _LAST_TAG_LINE\n\n tag_types = {\n '!': 'comment',\n '#': 'section',\n '^': 'inverted section',\n '/': 'end',\n '>': 'partial',\n '=': 'set delimiter?',\n '{': 'no escape?',\n '&': 'no escape'\n }\n\n # Get the tag\n try:\n tag, template = template.split(r_del, 1)\n except ValueError:\n raise ChevronError('unclosed tag '\n 'at line {0}'.format(_CURRENT_LINE))\n\n # Find the type meaning of the first character\n tag_type = tag_types.get(tag[0], 'variable')\n\n # If the type is not a variable\n if tag_type != 'variable':\n # Then that first character is not needed\n tag = tag[1:]\n\n # If we might be a set delimiter tag\n if tag_type == 'set delimiter?':\n # Double check to make sure we are\n if tag.endswith('='):\n tag_type = 'set delimiter'\n # Remove the equal sign\n tag = tag[:-1]\n\n # Otherwise we should complain\n else:\n raise ChevronError('unclosed set delimiter tag\\n'\n 'at line {0}'.format(_CURRENT_LINE))\n\n # If we might be a no html escape tag\n elif tag_type == 'no escape?':\n # And we have a third curly brace\n # (And are using curly braces as delimiters)\n if l_del == '{{' and r_del == '}}' and template.startswith('}'):\n # Then we are a no html escape tag\n template = template[1:]\n tag_type = 'no escape'\n\n # Strip the whitespace off the key and return\n return ((tag_type, tag.strip()), template)", "response": "Parse a tag from a template"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef tokenize(template, def_ldel='{{', def_rdel='}}'):\n\n global _CURRENT_LINE, _LAST_TAG_LINE\n _CURRENT_LINE = 1\n _LAST_TAG_LINE = None\n # If the template is a file-like object then read it\n try:\n template = template.read()\n except AttributeError:\n pass\n\n is_standalone = True\n open_sections = []\n l_del = def_ldel\n r_del = def_rdel\n\n while template:\n literal, template = grab_literal(template, l_del)\n\n # If the template is completed\n if not template:\n # Then yield the literal and leave\n yield ('literal', literal)\n break\n\n # Do the first check to see if we could be a standalone\n is_standalone = l_sa_check(template, literal, is_standalone)\n\n # Parse the tag\n tag, template = parse_tag(template, l_del, r_del)\n tag_type, tag_key = tag\n\n # Special tag logic\n\n # If we are a set delimiter tag\n if tag_type == 'set delimiter':\n # Then get and set the delimiters\n dels = tag_key.strip().split(' ')\n l_del, r_del = dels[0], dels[-1]\n\n # If we are a section tag\n elif tag_type in ['section', 'inverted section']:\n # Then open a new section\n open_sections.append(tag_key)\n _LAST_TAG_LINE = _CURRENT_LINE\n\n # If we are an end tag\n elif tag_type == 'end':\n # Then check to see if the last opened section\n # is the same as us\n try:\n last_section = open_sections.pop()\n except IndexError:\n raise ChevronError('Trying to close tag \"{0}\"\\n'\n 'Looks like it was not opened.\\n'\n 'line {1}'\n .format(tag_key, _CURRENT_LINE + 1))\n if tag_key != last_section:\n # Otherwise we need to complain\n raise ChevronError('Trying to close tag \"{0}\"\\n'\n 'last open tag is \"{1}\"\\n'\n 'line {2}'\n .format(tag_key, last_section,\n _CURRENT_LINE + 1))\n\n # Do the second check to see if we're a standalone\n is_standalone = r_sa_check(template, tag_type, is_standalone)\n\n # Which if we are\n if is_standalone:\n # Remove the stuff before the newline\n template = template.split('\\n', 1)[-1]\n\n # Partials need to keep the spaces on their left\n if tag_type != 'partial':\n # But other tags don't\n literal = literal.rstrip(' ')\n\n # Start yielding\n # Ignore literals that are empty\n if literal != '':\n yield ('literal', literal)\n\n # Ignore comments and set delimiters\n if tag_type not in ['comment', 'set delimiter?']:\n yield (tag_type, tag_key)\n\n # If there are any open sections when we're done\n if open_sections:\n # Then we need to complain\n raise ChevronError('Unexpected EOF\\n'\n 'the tag \"{0}\" was never closed\\n'\n 'was opened at line {1}'\n .format(open_sections[-1], _LAST_TAG_LINE))", "response": "Tokenizes a mustache template in a generator fashion."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nrendering mustache templates using json files", "response": "def cli_main():\n \"\"\"Render mustache templates using json files\"\"\"\n import argparse\n import os\n\n def is_file_or_pipe(arg):\n if not os.path.exists(arg) or os.path.isdir(arg):\n parser.error('The file {0} does not exist!'.format(arg))\n else:\n return arg\n\n def is_dir(arg):\n if not os.path.isdir(arg):\n parser.error('The directory {0} does not exist!'.format(arg))\n else:\n return arg\n\n parser = argparse.ArgumentParser(description=__doc__)\n\n parser.add_argument('-v', '--version', action='version',\n version=version)\n\n parser.add_argument('template', help='The mustache file',\n type=is_file_or_pipe)\n\n parser.add_argument('-d', '--data', dest='data',\n help='The json data file',\n type=is_file_or_pipe, default={})\n\n parser.add_argument('-p', '--path', dest='partials_path',\n help='The directory where your partials reside',\n type=is_dir, default='.')\n\n parser.add_argument('-e', '--ext', dest='partials_ext',\n help='The extension for your mustache\\\n partials, \\'mustache\\' by default',\n default='mustache')\n\n parser.add_argument('-l', '--left-delimiter', dest='def_ldel',\n help='The default left delimiter, \"{{\" by default.',\n default='{{')\n\n parser.add_argument('-r', '--right-delimiter', dest='def_rdel',\n help='The default right delimiter, \"}}\" by default.',\n default='}}')\n\n args = vars(parser.parse_args())\n\n try:\n sys.stdout.write(main(**args))\n sys.stdout.flush()\n except SyntaxError as e:\n print('Chevron: syntax error')\n print(' ' + '\\n '.join(e.args[0].split('\\n')))\n exit(1)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _html_escape(string):\n\n html_codes = {\n '\"': '"',\n '<': '<',\n '>': '>',\n }\n\n # & must be handled first\n string = string.replace('&', '&')\n for char in html_codes:\n string = string.replace(char, html_codes[char])\n return string", "response": "HTML escape all of these & < >"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _get_key(key, scopes):\n\n # If the key is a dot\n if key == '.':\n # Then just return the current scope\n return scopes[0]\n\n # Loop through the scopes\n for scope in scopes:\n try:\n # For every dot seperated key\n for child in key.split('.'):\n # Move into the scope\n try:\n # Try subscripting (Normal dictionaries)\n scope = scope[child]\n except (TypeError, AttributeError):\n try:\n # Try the dictionary (Complex types)\n scope = scope.__dict__[child]\n except (TypeError, AttributeError):\n # Try as a list\n scope = scope[int(child)]\n\n # Return an empty string if falsy, with two exceptions\n # 0 should return 0, and False should return False\n # While using is for this check is undefined it works and is fast\n if scope is 0: # noqa: F632\n return 0\n if scope is False:\n return False\n\n try:\n # This allows for custom falsy data types\n # https://github.com/noahmorrison/chevron/issues/35\n if scope._CHEVRON_return_scope_when_falsy:\n return scope\n except AttributeError:\n return scope or ''\n except (AttributeError, KeyError, IndexError, ValueError):\n # We couldn't find the key in the current scope\n # We'll try again on the next pass\n pass\n\n # We couldn't find the key in any of the scopes\n return ''", "response": "Get a key from the current scope"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nloads a partial from the dictionary", "response": "def _get_partial(name, partials_dict, partials_path, partials_ext):\n \"\"\"Load a partial\"\"\"\n try:\n # Maybe the partial is in the dictionary\n return partials_dict[name]\n except KeyError:\n # Nope...\n try:\n # Maybe it's in the file system\n path_ext = ('.' + partials_ext if partials_ext else '')\n path = partials_path + '/' + name + path_ext\n with io.open(path, 'r', encoding='utf-8') as partial:\n return partial.read()\n\n except IOError:\n # Alright I give up on you\n return ''"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrendering a mustache template.", "response": "def render(template='', data={}, partials_path='.', partials_ext='mustache',\n partials_dict={}, padding='', def_ldel='{{', def_rdel='}}',\n scopes=None):\n \"\"\"Render a mustache template.\n\n Renders a mustache template with a data scope and partial capability.\n Given the file structure...\n \u2577\n \u251c\u2500\u257c main.py\n \u251c\u2500\u257c main.ms\n \u2514\u2500\u252e partials\n \u2514\u2500\u2500 part.ms\n\n then main.py would make the following call:\n\n render(open('main.ms', 'r'), {...}, 'partials', 'ms')\n\n\n Arguments:\n\n template -- A file-like object or a string containing the template\n\n data -- A python dictionary with your data scope\n\n partials_path -- The path to where your partials are stored\n (defaults to '.')\n\n partials_ext -- The extension that you want the parser to look for\n (defaults to 'mustache')\n\n partials_dict -- A python dictionary which will be search for partials\n before the filesystem is. {'include': 'foo'} is the same\n as a file called include.mustache\n (defaults to {})\n\n padding -- This is for padding partials, and shouldn't be used\n (but can be if you really want to)\n\n def_ldel -- The default left delimiter\n (\"{{\" by default, as in spec compliant mustache)\n\n def_rdel -- The default right delimiter\n (\"}}\" by default, as in spec compliant mustache)\n\n scopes -- The list of scopes that get_key will look through\n\n\n Returns:\n\n A string containing the rendered template.\n \"\"\"\n\n # If the template is a seqeuence but not derived from a string\n if isinstance(template, Sequence) and \\\n not isinstance(template, string_type):\n # Then we don't need to tokenize it\n # But it does need to be a generator\n tokens = (token for token in template)\n else:\n if template in g_token_cache:\n tokens = (token for token in g_token_cache[template])\n else:\n # Otherwise make a generator\n tokens = tokenize(template, def_ldel, def_rdel)\n\n output = unicode('', 'utf-8')\n\n if scopes is None:\n scopes = [data]\n\n # Run through the tokens\n for tag, key in tokens:\n # Set the current scope\n current_scope = scopes[0]\n\n # If we're an end tag\n if tag == 'end':\n # Pop out of the latest scope\n del scopes[0]\n\n # If the current scope is falsy and not the only scope\n elif not current_scope and len(scopes) != 1:\n if tag in ['section', 'inverted section']:\n # Set the most recent scope to a falsy value\n # (I heard False is a good one)\n scopes.insert(0, False)\n\n # If we're a literal tag\n elif tag == 'literal':\n # Add padding to the key and add it to the output\n if not isinstance(key, unicode_type): # python 2\n key = unicode(key, 'utf-8')\n output += key.replace('\\n', '\\n' + padding)\n\n # If we're a variable tag\n elif tag == 'variable':\n # Add the html escaped key to the output\n thing = _get_key(key, scopes)\n if thing is True and key == '.':\n # if we've coerced into a boolean by accident\n # (inverted tags do this)\n # then get the un-coerced object (next in the stack)\n thing = scopes[1]\n if not isinstance(thing, unicode_type):\n thing = unicode(str(thing), 'utf-8')\n output += _html_escape(thing)\n\n # If we're a no html escape tag\n elif tag == 'no escape':\n # Just lookup the key and add it\n thing = _get_key(key, scopes)\n if not isinstance(thing, unicode_type):\n thing = unicode(str(thing), 'utf-8')\n output += thing\n\n # If we're a section tag\n elif tag == 'section':\n # Get the sections scope\n scope = _get_key(key, scopes)\n\n # If the scope is a callable (as described in\n # https://mustache.github.io/mustache.5.html)\n if isinstance(scope, Callable):\n\n # Generate template text from tags\n text = unicode('', 'utf-8')\n tags = []\n for tag in tokens:\n if tag == ('end', key):\n break\n\n tags.append(tag)\n tag_type, tag_key = tag\n if tag_type == 'literal':\n text += tag_key\n elif tag_type == 'no escape':\n text += \"%s& %s %s\" % (def_ldel, tag_key, def_rdel)\n else:\n text += \"%s%s %s%s\" % (def_ldel, {\n 'commment': '!',\n 'section': '#',\n 'inverted section': '^',\n 'end': '/',\n 'partial': '>',\n 'set delimiter': '=',\n 'no escape': '&',\n 'variable': ''\n }[tag_type], tag_key, def_rdel)\n\n g_token_cache[text] = tags\n\n rend = scope(text, lambda template, data=None: render(template,\n data={},\n partials_path=partials_path,\n partials_ext=partials_ext,\n partials_dict=partials_dict,\n padding=padding,\n def_ldel=def_ldel, def_rdel=def_rdel,\n scopes=data and [data]+scopes or scopes))\n\n if python3:\n output += rend\n else: # python 2\n output += rend.decode('utf-8')\n\n # If the scope is a sequence, an iterator or generator but not\n # derived from a string\n elif isinstance(scope, (Sequence, Iterator)) and \\\n not isinstance(scope, string_type):\n # Then we need to do some looping\n\n # Gather up all the tags inside the section\n # (And don't be tricked by nested end tags with the same key)\n # TODO: This feels like it still has edge cases, no?\n tags = []\n tags_with_same_key = 0\n for tag in tokens:\n if tag == ('section', key):\n tags_with_same_key += 1\n if tag == ('end', key):\n tags_with_same_key -= 1\n if tags_with_same_key < 0:\n break\n tags.append(tag)\n\n # For every item in the scope\n for thing in scope:\n # Append it as the most recent scope and render\n new_scope = [thing] + scopes\n rend = render(template=tags, scopes=new_scope,\n partials_path=partials_path,\n partials_ext=partials_ext,\n partials_dict=partials_dict,\n def_ldel=def_ldel, def_rdel=def_rdel)\n\n if python3:\n output += rend\n else: # python 2\n output += rend.decode('utf-8')\n\n else:\n # Otherwise we're just a scope section\n scopes.insert(0, scope)\n\n # If we're an inverted section\n elif tag == 'inverted section':\n # Add the flipped scope to the scopes\n scope = _get_key(key, scopes)\n scopes.insert(0, not scope)\n\n # If we're a partial\n elif tag == 'partial':\n # Load the partial\n partial = _get_partial(key, partials_dict,\n partials_path, partials_ext)\n\n # Find what to pad the partial with\n left = output.split('\\n')[-1]\n part_padding = padding\n if left.isspace():\n part_padding += left\n\n # Render the partial\n part_out = render(template=partial, partials_path=partials_path,\n partials_ext=partials_ext,\n partials_dict=partials_dict,\n def_ldel=def_ldel, def_rdel=def_rdel,\n padding=part_padding, scopes=scopes)\n\n # If the partial was indented\n if left.isspace():\n # then remove the spaces from the end\n part_out = part_out.rstrip(' \\t')\n\n # Add the partials output to the ouput\n if python3:\n output += part_out\n else: # python 2\n output += part_out.decode('utf-8')\n\n if python3:\n return output\n else: # python 2\n return output.encode('utf-8')"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef bulk_add(\n self, item_id, ref_id=None, tags=None, time=None, title=None,\n url=None, wait=True\n ):\n '''\n Add a new item to the user's list\n http://getpocket.com/developer/docs/v3/modify#action_add\n\n '''", "response": "Bulk add a new item to the user s list\n "} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the request token that can be used to fetch the access token", "response": "def get_request_token(\n cls, consumer_key, redirect_uri='http://example.com/', state=None\n ):\n '''\n Returns the request token that can be used to fetch the access token\n\n '''\n headers = {\n 'X-Accept': 'application/json',\n }\n url = 'https://getpocket.com/v3/oauth/request'\n payload = {\n 'consumer_key': consumer_key,\n 'redirect_uri': redirect_uri,\n }\n\n if state:\n payload['state'] = state\n\n return cls._make_request(url, payload, headers)[0]['code']"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfetch access token from using the request token and consumer key and code.", "response": "def get_credentials(cls, consumer_key, code):\n '''\n Fetches access token from using the request token and consumer key\n\n '''\n headers = {\n 'X-Accept': 'application/json',\n }\n url = 'https://getpocket.com/v3/oauth/authorize'\n payload = {\n 'consumer_key': consumer_key,\n 'code': code,\n }\n\n return cls._make_request(url, payload, headers)[0]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nremove an object from the relation", "response": "def remove(self,obj):\n \"\"\"\n Remove an object from the relation\n \"\"\"\n relationship_table = self.params['relationship_table']\n with self.obj.backend.transaction(implicit = True):\n condition = and_(relationship_table.c[self.params['related_pk_field_name']] == obj.pk,\n relationship_table.c[self.params['pk_field_name']] == self.obj.pk)\n self.obj.backend.connection.execute(delete(relationship_table).where(condition))\n self._queryset = None"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef begin(self):\n if self.in_transaction: # we're already in a transaction...\n if self._auto_transaction:\n self._auto_transaction = False\n return\n self.commit()\n self.in_transaction = True\n for collection, store in self.stores.items():\n store.begin()\n indexes = self.indexes[collection]\n for index in indexes.values():\n index.begin()", "response": "Begin a new transaction."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef rollback(self, transaction = None):\n if not self.in_transaction:\n raise NotInTransaction\n for collection, store in self.stores.items():\n store.rollback()\n indexes = self.indexes[collection]\n indexes_to_rebuild = []\n for key, index in indexes.items():\n try:\n index.rollback()\n except NotInTransaction:\n # this index is \"dirty\" and needs to be rebuilt\n # (probably it has been created within a transaction)\n indexes_to_rebuild.append(key)\n if indexes_to_rebuild:\n self.rebuild_indexes(collection, indexes_to_rebuild)\n self.in_transaction = False", "response": "Roll back a transaction."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncommit all pending transactions to the database.", "response": "def commit(self,transaction = None):\n \"\"\"Commit all pending transactions to the database.\n\n .. admonition:: Warning\n\n This operation can be **expensive** in runtime if a large number of\n documents (>100.000) is contained in the database, since it will\n cause all database indexes to be written to disk.\n\n \"\"\"\n for collection in self.collections:\n store = self.get_collection_store(collection)\n store.commit()\n indexes = self.get_collection_indexes(collection)\n for index in indexes.values():\n index.commit()\n self.in_transaction = False\n self.begin()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a new index on the given collection or class with given parameters.", "response": "def create_index(self, cls_or_collection,\n params=None, fields=None, ephemeral=False, unique=False):\n \"\"\"Create new index on the given collection/class with given parameters.\n\n :param cls_or_collection:\n The name of the collection or the class for which to create an\n index\n :param params: The parameters of the index\n :param ephemeral: Whether to create a persistent or an ephemeral index\n :param unique: Whether the indexed field(s) must be unique\n\n `params` expects either a dictionary of parameters or a string value.\n In the latter case, it will interpret the string as the name of the key\n for which an index is to be created.\n\n If `ephemeral = True`, the index will be created only in memory and\n will not be written to disk when :py:meth:`.commit` is called. This is\n useful for optimizing query performance.\n\n ..notice::\n\n By default, BlitzDB will create ephemeral indexes for all keys over\n which you perform queries, so after you've run a query on a given\n key for the first time, the second run will usually be much faster.\n\n **Specifying keys**\n\n Keys can be specified just like in MongoDB, using a dot ('.') to\n specify nested keys.\n\n .. code-block:: python\n\n actor = Actor({'name' : 'Charlie Chaplin',\n 'foo' : {'value' : 'bar'}})\n\n If you want to create an index on `actor['foo']['value']` , you can\n just say\n\n .. code-block:: python\n\n backend.create_index(Actor,'foo.value')\n\n .. warning::\n\n Transcendental indexes (i.e. indexes transcending the boundaries of\n referenced objects) are currently not supported by Blitz, which\n means you can't create an index on an attribute value of a document\n that is embedded in another document.\n\n \"\"\"\n if params:\n return self.create_indexes(cls_or_collection, [params],\n ephemeral=ephemeral, unique=unique)\n elif fields:\n params = []\n if len(fields.items()) > 1:\n raise ValueError(\"File backend currently does not support multi-key indexes, sorry :/\")\n return self.create_indexes(cls_or_collection, [{'key': list(fields.keys())[0]}],\n ephemeral=ephemeral, unique=unique)\n else:\n raise AttributeError('You must either specify params or fields!')"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the primary key index for a given collection.", "response": "def get_pk_index(self, collection):\n \"\"\"Return the primary key index for a given collection.\n\n :param collection: the collection for which to return the primary index\n\n :returns: the primary key index of the given collection\n\n \"\"\"\n cls = self.collections[collection]\n\n if not cls.get_pk_name() in self.indexes[collection]:\n self.create_index(cls.get_pk_name(), collection)\n return self.indexes[collection][cls.get_pk_name()]"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nupdates the object with the specified fields.", "response": "def update(self, obj, set_fields = None, unset_fields = None, update_obj = True):\n \"\"\"\n We return the result of the save method (updates are not yet implemented here).\n \"\"\"\n if set_fields:\n if isinstance(set_fields,(list,tuple)):\n set_attributes = {}\n for key in set_fields:\n try:\n set_attributes[key] = get_value(obj,key)\n except KeyError:\n pass\n else:\n set_attributes = set_fields\n else:\n set_attributes = {}\n if unset_fields:\n unset_attributes = unset_fields\n else:\n unset_attributes = []\n\n self.call_hook('before_update',obj,set_attributes,unset_attributes)\n\n if update_obj:\n for key,value in set_attributes.items():\n set_value(obj,key,value)\n for key in unset_attributes:\n delete_value(obj,key)\n\n return self.save(obj,call_hook = False)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _canonicalize_query(self, query):\n\n \"\"\"\n Transform the query dictionary to replace e.g. documents with __ref__ fields.\n \"\"\"\n\n def transform_query(q):\n\n if isinstance(q, dict):\n nq = {}\n for key,value in q.items():\n nq[key] = transform_query(value)\n return nq\n elif isinstance(q, (list,QuerySet,tuple)):\n return [transform_query(x) for x in q]\n elif isinstance(q,Document):\n collection = self.get_collection_for_obj(q)\n ref = \"%s:%s\" % (collection,q.pk)\n return ref\n else:\n return q\n\n return transform_query(query)", "response": "Transform the query dictionary to replace e. g. documents with __ref__ fields."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef register(self, cls, parameters=None,overwrite = False):\n\n if cls in self.deprecated_classes and not overwrite:\n return False\n\n if parameters is None:\n parameters = {}\n if 'collection' in parameters:\n collection_name = parameters['collection']\n elif hasattr(cls.Meta,'collection'):\n collection_name = cls.Meta.collection\n else:\n collection_name = cls.__name__.lower()\n\n delete_list = []\n\n def register_class(collection_name,cls):\n self.collections[collection_name] = cls\n self.classes[cls] = parameters.copy()\n self.classes[cls]['collection'] = collection_name\n\n if collection_name in self.collections:\n old_cls = self.collections[collection_name]\n if (issubclass(cls,old_cls) and not (cls is old_cls)) or overwrite:\n logger.warning(\"Replacing class %s with %s for collection %s\" % (old_cls,cls,collection_name))\n self.deprecated_classes[old_cls] = self.classes[old_cls]\n del self.classes[old_cls]\n register_class(collection_name,cls)\n return True\n else:\n logger.debug(\"Registering class %s under collection %s\" % (cls,collection_name))\n register_class(collection_name,cls)\n return True\n\n return False", "response": "This method registers a new document class for use in the backend."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nserialize a given object to a dictionary.", "response": "def serialize(self, obj, convert_keys_to_str=False,\n embed_level=0,\n encoders=None,\n autosave=True,\n for_query=False,path = None):\n \"\"\"\n Serializes a given object, i.e. converts it to a representation that can be stored in the database.\n This usually involves replacing all `Document` instances by database references to them.\n\n :param obj: The object to serialize.\n :param convert_keys_to_str: If `True`, converts all dictionary keys to string (this is e.g. required for the MongoDB backend)\n :param embed_level: If `embed_level > 0`, instances of `Document` classes will be embedded instead of referenced.\n The value of the parameter will get decremented by 1 when calling `serialize` on child objects.\n :param autosave: Whether to automatically save embedded objects without a primary key to the database.\n :param for_query: If true, only the `pk` and `__collection__` attributes will be included in document references.\n\n :returns: The serialized object.\n \"\"\"\n\n if path is None:\n path = []\n\n def get_value(obj,key):\n key_fragments = key.split(\".\")\n current_dict = obj\n for key_fragment in key_fragments:\n current_dict = current_dict[key_fragment]\n return current_dict\n\n serialize_with_opts = lambda value,*args,**kwargs : self.serialize(value,*args,\n encoders = encoders,\n convert_keys_to_str = convert_keys_to_str,\n autosave = autosave,\n for_query = for_query,\n **kwargs)\n\n if encoders is None:\n encoders = []\n\n for encoder in self.standard_encoders+encoders:\n obj = encoder.encode(obj,path = path)\n\n def encode_as_str(obj):\n if six.PY3:\n return str(obj)\n else:\n if isinstance(obj,unicode):\n return obj\n elif isinstance(obj,str):\n return unicode(obj)\n else:\n return unicode(str(obj),errors='replace')\n\n if isinstance(obj, dict):\n output_obj = {}\n for key, value in obj.items():\n new_path = path[:]+[key]\n try:\n output_obj[encode_as_str(key) if convert_keys_to_str else key] = serialize_with_opts(value, embed_level=embed_level,path = new_path)\n except DoNotSerialize:\n pass\n elif isinstance(obj,six.string_types):\n output_obj = encode_as_str(obj)\n elif isinstance(obj, (list,tuple)):\n try:\n output_obj = [serialize_with_opts(x, embed_level=embed_level,path = path[:]+[i]) for i,x in enumerate(obj)]\n except DoNotSerialize:\n pass\n elif isinstance(obj, Document):\n collection = self.get_collection_for_obj(obj)\n if embed_level > 0:\n try:\n output_obj = self.serialize(obj, embed_level=embed_level-1)\n except obj.DoesNotExist:#cannot load object, ignoring...\n output_obj = self.serialize(obj.lazy_attributes, embed_level=embed_level-1)\n except DoNotSerialize:\n pass\n elif obj.embed:\n output_obj = self.serialize(obj)\n else:\n if obj.pk == None and autosave:\n obj.save(self)\n\n if obj._lazy:\n # We make sure that all attributes that are already present get included in the reference\n output_obj = {}\n if obj.get_pk_name() in output_obj:\n del output_obj[obj.get_pk_name()]\n output_obj['pk'] = obj.pk\n output_obj['__collection__'] = self.classes[obj.__class__]['collection']\n else:\n if for_query and not self._allow_documents_in_query:\n raise ValueError(\"Documents are not allowed in queries!\")\n if for_query:\n output_obj = {'$elemMatch' : {'pk':obj.pk,'__collection__':self.classes[obj.__class__]['collection']}}\n else:\n ref = \"%s:%s\" % (self.classes[obj.__class__]['collection'],str(obj.pk))\n output_obj = {'__ref__' : ref,'pk':obj.pk,'__collection__':self.classes[obj.__class__]['collection']}\n\n if hasattr(obj,'Meta') and hasattr(obj.Meta,'dbref_includes') and obj.Meta.dbref_includes:\n for include_key in obj.Meta.dbref_includes:\n try:\n value = get_value(obj,include_key)\n output_obj[include_key.replace(\".\",\"_\")] = value\n except KeyError:\n continue\n\n\n else:\n output_obj = obj\n return output_obj"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef deserialize(self, obj, encoders=None, embedded=False, create_instance=True):\n\n if not encoders:\n encoders = []\n\n for encoder in encoders + self.standard_encoders:\n obj = encoder.decode(obj)\n\n if isinstance(obj, dict):\n if create_instance and '__collection__' in obj and obj['__collection__'] in self.collections and 'pk' in obj:\n #for backwards compatibility\n attributes = copy.deepcopy(obj)\n del attributes['__collection__']\n if '__ref__' in attributes:\n del attributes['__ref__']\n if '__lazy__' in attributes:\n lazy = attributes['__lazy__']\n del attributes['__lazy__']\n else:\n lazy = True\n output_obj = self.create_instance(obj['__collection__'], attributes, lazy=lazy)\n else:\n output_obj = {}\n for key, value in obj.items():\n output_obj[key] = self.deserialize(value,encoders = encoders)\n elif isinstance(obj, (list,tuple)):\n output_obj = list(map(lambda x: self.deserialize(x), obj))\n else:\n output_obj = obj\n\n return output_obj", "response": "Deserializes a given object by lazy instances of the corresponding class."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates an instance of a Document class corresponding to the given collection name or class.", "response": "def create_instance(self, collection_or_class, attributes, lazy=False, call_hook=True, deserialize=True, db_loader=None):\n \"\"\"\n Creates an instance of a `Document` class corresponding to the given collection name or class.\n\n :param collection_or_class: The name of the collection or a reference to the class for which to create an instance.\n :param attributes: The attributes of the instance to be created\n :param lazy: Whether to create a `lazy` object or not.\n\n :returns: An instance of the requested Document class with the given attributes.\n \"\"\"\n creation_args = {\n 'backend' : self,\n 'autoload' : self._autoload_embedded,\n 'lazy' : lazy,\n 'db_loader' : db_loader\n }\n\n if collection_or_class in self.classes:\n cls = collection_or_class\n elif collection_or_class in self.collections:\n cls = self.collections[collection_or_class]\n else:\n raise AttributeError(\"Unknown collection or class: %s!\" % str(collection_or_class))\n\n #we deserialize the attributes that we receive\n if deserialize:\n deserialized_attributes = self.deserialize(attributes, create_instance=False)\n else:\n deserialized_attributes = attributes\n\n if 'constructor' in self.classes[cls]:\n obj = self.classes[cls]['constructor'](deserialized_attributes, **creation_args)\n else:\n obj = cls(deserialized_attributes, **creation_args)\n\n if call_hook:\n self.call_hook('after_load',obj)\n\n return obj"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_collection_for_cls(self, cls):\n if cls not in self.classes:\n if issubclass(cls, Document) and cls not in self.classes and cls not in self.deprecated_classes:\n self.autoregister(cls)\n else:\n raise AttributeError(\"Unknown object type: %s\" % cls.__name__)\n collection = self.classes[cls]['collection']\n return collection", "response": "Returns the collection name for a given document class."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the collection name for a given class name.", "response": "def get_collection_for_cls_name(self, cls_name):\n \"\"\"\n Returns the collection name for a given document class.\n\n :param cls: The document class for which to return the collection name.\n\n :returns: The collection name for the given class.\n \"\"\"\n for cls in self.classes:\n if cls.__name__ == cls_name:\n return self.classes[cls]['collection']\n raise AttributeError(\"Unknown class name: %s\" % cls_name)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the class for a given collection name.", "response": "def get_cls_for_collection(self, collection):\n \"\"\"\n Return the class for a given collection name.\n\n :param collection: The name of the collection for which to return the class.\n\n :returns: A reference to the class for the given collection name.\n \"\"\"\n for cls, params in self.classes.items():\n if params['collection'] == collection:\n return cls\n raise AttributeError(\"Unknown collection: %s\" % collection)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nclears index and reverse index.", "response": "def clear(self):\n \"\"\"Clear index.\"\"\"\n self._index = defaultdict(list)\n self._reverse_index = defaultdict(list)\n self._undefined_keys = {}"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets value to be indexed from document attributes.", "response": "def get_value(self, attributes,key = None):\n \"\"\"Get value to be indexed from document attributes.\n\n :param attributes: Document attributes\n :type attributes: dict\n :return: Value to be indexed\n :rtype: object\n\n \"\"\"\n\n value = attributes\n if key is None:\n key = self._splitted_key\n\n # A splitted key like 'a.b.c' goes into nested properties\n # and the value is retrieved recursively\n for i,elem in enumerate(key):\n if isinstance(value, (list,tuple)):\n #if this is a list, we return all matching values for the given list items\n return [self.get_value(v,key[i:]) for v in value]\n else:\n value = value[elem]\n\n return value"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsave index to store.", "response": "def save_to_store(self):\n \"\"\"Save index to store.\n\n :raise AttributeError: If no datastore is defined\n\n \"\"\"\n if not self._store:\n raise AttributeError('No datastore defined!')\n saved_data = self.save_to_data(in_place=True)\n data = Serializer.serialize(saved_data)\n self._store.store_blob(data, 'all_keys_with_undefined')"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_all_keys(self):\n all_keys = []\n for keys in self._index.values():\n all_keys.extend(keys)\n return all_keys", "response": "Get all keys indexed."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nload index from store.", "response": "def load_from_store(self):\n \"\"\"Load index from store.\n\n :return: Whether index was correctly loaded or not\n :rtype: bool\n :raise AttributeError: If no datastore is defined\n\n \"\"\"\n if not self._store:\n raise AttributeError('No datastore defined!')\n if self._store.has_blob('all_keys'):\n data = Serializer.deserialize(self._store.get_blob('all_keys'))\n self.load_from_data(data)\n return True\n elif self._store.has_blob('all_keys_with_undefined'):\n blob = self._store.get_blob('all_keys_with_undefined')\n data = Serializer.deserialize(blob)\n self.load_from_data(data, with_undefined=True)\n return True\n else:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef sort_keys(self, keys, order=QuerySet.ASCENDING):\n # to do: check that all reverse index values are unambiguous\n missing_keys = [\n key\n for key in keys\n if not len(self._reverse_index[key])\n ]\n keys_and_values = [\n (key, self._reverse_index[key][0])\n for key in keys\n if key not in missing_keys\n ]\n sorted_keys = [\n kv[0]\n for kv in sorted(\n keys_and_values,\n key=lambda x: x[1],\n reverse=True if order == QuerySet.DESCENDING else False)\n ]\n if order == QuerySet.ASCENDING:\n return missing_keys + sorted_keys\n elif order == QuerySet.DESCENDING:\n return sorted_keys + missing_keys\n else:\n raise ValueError('Unexpected order value: {:d}'.format(order))", "response": "Sort keys.\n\n Keys are sorted based on the value they are indexing.\n\n :param keys: Keys to be sorted\n :type keys: list(str)\n :param order: Order criteri (asending or descending)\n :type order: int\n :return: Sorted keys\n :rtype: list(str)\n :raise ValueError: If invalid order value is passed"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef save_to_data(self, in_place=False):\n if in_place:\n return [\n list(self._index.items()),\n list(self._undefined_keys.keys())\n ]\n return (\n [(key, values[:]) for key, values in self._index.items()],\n list(self._undefined_keys.keys()),\n )", "response": "Save index to data structure."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load_from_data(self, data, with_undefined=False):\n if with_undefined:\n defined_values, undefined_values = data\n else:\n defined_values = data\n undefined_values = None\n self._index = defaultdict(list, defined_values)\n self._reverse_index = defaultdict(list)\n for key, values in self._index.items():\n for value in values:\n self._reverse_index[value].append(key)\n if undefined_values:\n self._undefined_keys = {key: True for key in undefined_values}\n else:\n self._undefined_keys = {}", "response": "Load index structure from data."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting hash for a given value.", "response": "def get_hash_for(self, value):\n \"\"\"Get hash for a given value.\n\n :param value: The value to be indexed\n :type value: object\n :return: Hashed value\n :rtype: str\n\n \"\"\"\n if isinstance(value,dict) and '__ref__' in value:\n return self.get_hash_for(value['__ref__'])\n serialized_value = self._serializer(value)\n if isinstance(serialized_value, dict):\n # Hash each item and return the hash of all the hashes\n return hash(frozenset([\n self.get_hash_for(x)\n for x in serialized_value.items()\n ]))\n elif isinstance(serialized_value, (list,tuple)):\n # Hash each element and return the hash of all the hashes\n return hash(tuple([\n self.get_hash_for(x) for x in serialized_value\n ]))\n return value"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_keys_for(self, value):\n if callable(value):\n return value(self)\n hash_value = self.get_hash_for(value)\n return self._index[hash_value][:]", "response": "Get the keys for a given value."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add_hashed_value(self, hash_value, store_key):\n if self._unique and hash_value in self._index:\n raise NonUnique('Hash value {} already in index'.format(hash_value))\n if store_key not in self._index[hash_value]:\n self._index[hash_value].append(store_key)\n if hash_value not in self._reverse_index[store_key]:\n self._reverse_index[store_key].append(hash_value)", "response": "Adds a hashed value to the index."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef add_key(self, attributes, store_key):\n undefined = False\n try:\n value = self.get_value(attributes)\n except (KeyError, IndexError):\n undefined = True\n\n # We remove old values in _reverse_index\n self.remove_key(store_key)\n if not undefined:\n if isinstance(value, (list,tuple)):\n # We add an extra hash value for the list itself\n # (this allows for querying the whole list)\n values = value\n hash_value = self.get_hash_for(value)\n self.add_hashed_value(hash_value, store_key)\n else:\n values = [value]\n\n for value in values:\n hash_value = self.get_hash_for(value)\n self.add_hashed_value(hash_value, store_key)\n else:\n self.add_undefined(store_key)", "response": "Add key to the index."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef remove_key(self, store_key):\n if store_key in self._undefined_keys:\n del self._undefined_keys[store_key]\n if store_key in self._reverse_index:\n for value in self._reverse_index[store_key]:\n self._index[value].remove(store_key)\n del self._reverse_index[store_key]", "response": "Remove the key from the index."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndrops changes from current transaction.", "response": "def rollback(self):\n \"\"\"Drop changes from current transaction.\"\"\"\n if not self._in_transaction:\n raise NotInTransaction\n self._init_cache()\n self._in_transaction = False"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef add_hashed_value(self, hash_value, store_key):\n if hash_value not in self._add_cache[store_key]:\n self._add_cache[store_key].append(hash_value)\n if store_key not in self._reverse_add_cache[hash_value]:\n self._reverse_add_cache[hash_value].append(store_key)\n if store_key in self._remove_cache:\n del self._remove_cache[store_key]\n if store_key in self._undefined_cache:\n del self._undefined_cache[store_key]", "response": "Adds a hashed value to the index."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nremoves the key in the context of the current transaction.", "response": "def remove_key(self, store_key):\n \"\"\"Remove key in the context of the current transaction.\n\n :param store_key: The key for the document in the store\n :type store_key: str\n\n \"\"\"\n self._remove_cache[store_key] = True\n if store_key in self._add_cache:\n for hash_value in self._add_cache[store_key]:\n self._reverse_add_cache[hash_value].remove(store_key)\n del self._add_cache[store_key]\n if store_key in self._undefined_cache:\n del self._undefined_cache[store_key]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_keys_for(self, value, include_uncommitted=False):\n if not include_uncommitted:\n return super(TransactionalIndex, self).get_keys_for(value)\n else:\n keys = super(TransactionalIndex, self).get_keys_for(value)\n hash_value = self.get_hash_for(value)\n keys += self._reverse_add_cache[hash_value]\n return keys", "response": "Get the keys for a given value."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef filter(self, cls_or_collection, query, raw = False,only = None,include = None):\n\n if not isinstance(cls_or_collection, six.string_types):\n collection = self.get_collection_for_cls(cls_or_collection)\n cls = cls_or_collection\n else:\n collection = cls_or_collection\n cls = self.get_cls_for_collection(collection)\n\n table = self._collection_tables[collection]\n\n joins = defaultdict(dict)\n joins_list = []\n group_bys = []\n havings = []\n\n def compile_query(collection,query,table = None,path = None):\n if path is None:\n path = []\n\n \"\"\"\n This function emits a list of WHERE statements that can be used to retrieve\n \"\"\"\n\n if table is None:\n table = self._collection_tables[collection]\n\n where_statements = []\n\n if any([True if key.startswith('$') else False for key in query.keys()]):\n #this is a special operator query\n if len(query) > 1:\n raise AttributeError('Currently not supported!')\n operator = list(query.keys())[0][1:]\n if not operator in ('and','or','not'):\n raise AttributeError(\"Non-supported logical operator: $%s\" % operator)\n if operator in ('and','or'):\n where_statements = [sq for expr in query['$%s' % operator]\n for sq in compile_query(collection,expr,path = path)]\n if operator == 'and':\n return [and_(*where_statements)]\n else:\n return [or_(*where_statements)]\n elif operator == 'not':\n return [not_(*compile_query(collection,query['$not'],table = table,path = path))]\n\n def compile_one_to_many_query(key,query,field_name,related_table,count_column,path):\n\n def prepare_subquery(tail,query_dict):\n d = {}\n if not tail:\n if isinstance(query_dict,dict):\n return query_dict.copy()\n if not isinstance(query_dict,Document):\n raise AttributeError(\"Must be a document!\")\n if not query_dict.pk:\n raise AttributeError(\"Performing a query without a primary key!\")\n return {'pk' : query_dict.pk}\n return {tail : query_dict}\n\n tail = key[len(field_name)+1:]\n\n if isinstance(query,Document) and not tail:\n query = {'pk' : query.pk}\n\n #to do: implement $size and $not: {$size} operators...\n if isinstance(query,dict) and len(query) == 1 and list(query.keys())[0] in ('$all','$in','$elemMatch','$nin'):\n #this is an $in/$all/$nin query\n query_type = list(query.keys())[0][1:]\n subquery = list(query.values())[0]\n\n if query_type == 'elemMatch':\n queries = compile_query(params['collection'],\n prepare_subquery(tail,query['$elemMatch']),\n table = related_table,\n path = path)\n return queries\n else:\n if isinstance(subquery,(ManyToManyProxy,QuerySet)):\n if tail:\n #this query has a tail\n query = {tail : query}\n queries = compile_query(params['collection'],query,\n table = related_table,\n path = path)\n return queries\n #this is a query with a ManyToManyProxy/QuerySet\n if isinstance(subquery,ManyToManyProxy):\n qs = subquery.get_queryset()\n else:\n qs = subquery\n if not query_type in ('in','nin','all'):\n raise AttributeError\n if query_type == 'all':\n op = 'in'\n else:\n op = query_type\n if query_type == 'all':\n cnt = func.count(count_column)\n condition = cnt == qs.get_count_select()\n havings.append(condition)\n return [getattr(related_table.c['pk'],op+'_')(qs.get_select(columns = ['pk']))]\n elif isinstance(subquery,(list,tuple)):\n if subquery and isinstance(subquery[0],dict) and len(subquery[0]) == 1 and \\\n list(subquery[0].keys())[0] == '$elemMatch':\n queries = [sq for v in subquery for sq in compile_query(params['collection'],\n prepare_subquery(tail,v['$elemMatch']),\n table = related_table,\n path = path)]\n else:\n queries = [sq for v in subquery for sq in compile_query(params['collection'],\n prepare_subquery(tail,v),\n table = related_table,\n path = path)]\n where_statement = or_(*queries)\n\n if query_type == 'nin':\n where_statement = not_(where_statement)\n\n if query_type == 'all' and len(queries) > 1:\n cnt = func.count(count_column)\n havings.append(cnt == len(queries))\n\n return [where_statement]\n else:\n raise AttributeError(\"$in/$nin/$all query requires a list/tuple/QuerySet/ManyToManyProxy\")\n else:\n return compile_query(params['collection'],prepare_subquery(tail,query),\n table = related_table,\n path = path)\n\n def compile_many_to_many_query(key,value,field_name,params,relationship_table,path):\n\n related_collection = params['collection']\n related_table = self._collection_tables[related_collection]\n\n path_str = \".\".join(path)\n\n if path_str in joins[relationship_table]:\n relationship_table_alias = joins[relationship_table][path_str]\n else:\n relationship_table_alias = relationship_table.alias()\n joins[relationship_table][path_str] = relationship_table_alias\n joins_list.append((relationship_table_alias,\n relationship_table_alias.c[params['pk_field_name']] == table.c['pk']))\n\n if path_str in joins[related_table]:\n related_table_alias = joins[related_table][path_str]\n else:\n related_table_alias = related_table.alias()\n joins[related_table][path_str] = related_table_alias\n joins_list.append((related_table_alias,relationship_table_alias.c[params['related_pk_field_name']] == related_table_alias.c['pk']))\n\n return compile_one_to_many_query(key,value,field_name,related_table_alias,relationship_table_alias.c[params['pk_field_name']],new_path)\n\n def prepare_special_query(field_name,params,query):\n def sanitize(value):\n if isinstance(value,(list,tuple)):\n return [v.pk if isinstance(v,Document) else v for v in value]\n return value\n column_name = params['column']\n if '$not' in query:\n return [not_(*prepare_special_query(column_name,params,sanitize(query['$not'])))]\n elif '$in' in query:\n if not query['$in']:\n #we return an impossible condition since the $in query does not contain any values\n return [expression.cast(True,Boolean) == expression.cast(False,Boolean)]\n return [table.c[column_name].in_(sanitize(query['$in']))]\n elif '$nin' in query:\n if not query['$nin']:\n return [expression.cast(True,Boolean) == expression.cast(False,Boolean)]\n return [~table.c[column_name].in_(sanitize(query['$nin']))]\n elif '$eq' in query:\n return [table.c[column_name] == sanitize(query['$eq'])]\n elif '$ne' in query:\n return [table.c[column_name] != sanitize(query['$ne'])]\n elif '$gt' in query:\n return [table.c[column_name] > sanitize(query['$gt'])]\n elif '$gte' in query:\n return [table.c[column_name] >= sanitize(query['$gte'])]\n elif '$lt' in query:\n return [table.c[column_name] < sanitize(query['$lt'])]\n elif '$lte' in query:\n return [table.c[column_name] <= sanitize(query['$lte'])]\n elif '$exists' in query:\n if query['$exists']:\n return [table.c[column_name] != None]\n else:\n return [table.c[column_name] == None]\n elif '$like' in query:\n return [table.c[column_name].like(expression.cast(query['$like'],String))]\n elif '$ilike' in query:\n return [table.c[column_name].ilike(expression.cast(query['$ilike'],String))]\n elif '$regex' in query:\n if not self.engine.url.drivername in ('postgres','mysql','sqlite'):\n raise AttributeError(\"Regex queries not supported with %s engine!\" % self.engine.url.drivername)\n return [table.c[column_name].op('REGEXP')(expression.cast(query['$regex'],String))]\n else:\n raise AttributeError(\"Invalid query!\")\n\n #this is a normal, field-base query\n for key,value in query.items():\n for field_name,params in self._index_fields[collection].items():\n if key == field_name:\n if isinstance(value,re._pattern_type):\n value = {'$regex' : value.pattern}\n if isinstance(value,dict):\n #this is a special query\n where_statements.extend(prepare_special_query(field_name,params,value))\n else:\n #this is a normal value query\n where_statements.append(table.c[params['column']] == expression.cast(value,params['type']))\n break\n else:\n #we check the normal relationships\n for field_name,params in self._related_fields[collection].items():\n if key.startswith(field_name):\n\n head,tail = key[:len(field_name)],key[len(field_name)+1:]\n new_path = path + [head]\n path_str = \".\".join(new_path)\n #ManyToManyField\n if isinstance(params['field'],ManyToManyField):\n relationship_table = self._relationship_tables[collection][field_name]\n where_statements.extend(compile_many_to_many_query(key,value,field_name,params,relationship_table,path = new_path))\n elif isinstance(params['field'],ForeignKeyField):#this is a normal ForeignKey relation\n if key == field_name:\n #this is a ForeignKey query\n if isinstance(value,dict):\n if len(value) == 1:\n key,query = list(value.items())[0]\n if key == '$exists':\n if not isinstance(query,bool):\n raise AttributeError(\"$exists operator requires a Boolean operator\")\n if query:\n where_statements.append(table.c[params['column']] != None)\n else:\n where_statements.append(table.c[params['column']] == None)\n break\n elif not key in ('$in','$nin'):\n raise AttributeError(\"Invalid query!\")\n query_type = key[1:]\n else:\n raise AttributeError(\"Invalid query!\")\n else:\n query_type = 'exact'\n query = value\n if isinstance(query,(QuerySet,ManyToManyProxy)):\n if not query_type in ('in','nin'):\n raise AttributeError(\"QuerySet/ManyToManyProxy objects must be used in conjunction with $in/$nin when querying a ForeignKey relationship\")\n if isinstance(query,ManyToManyProxy):\n qs = query.get_queryset()\n else:\n qs = query\n if qs.count is not None and qs.count == 0:\n raise AttributeError(\"$in/$nin query with empty QuerySet/ManyToManyProxy!\")\n if qs.cls is not params['class']:\n raise AttributeError(\"Invalid QuerySet class!\")\n condition = getattr(table.c[params['column']],query_type+'_')(qs.get_select(columns = ['pk']))\n where_statements.append(condition)\n elif isinstance(query,(list,tuple)):\n if not query_type in ('in','nin'):\n raise AttributeError(\"Lists/tuples must be used in conjunction with $in/$nin when querying a ForeignKey relationship\")\n if not query:\n raise AttributeError(\"in/nin query with empty list!\")\n if query[0].__class__ is params['class']:\n if any((element.__class__ is not params['class'] for element in query)):\n raise AttributeError(\"Invalid document type in ForeignKey query\")\n where_statements.append(getattr(table.c[params['column']],query_type+'_')([expression.cast(doc.pk,params['type']) for doc in query]))\n else:\n where_statements.append(getattr(table.c[params['column']],query_type+'_')([expression.cast(element,params['type']) for element in query]))\n elif isinstance(query,Document):\n #we need an exact clas match here...\n if query.__class__ is not params['class']:\n raise AttributeError(\"Invalid Document class!\")\n where_statements.append(table.c[params['column']] == query.pk)\n else:\n where_statements.append(table.c[params['column']] == expression.cast(query,params['class'].Meta.PkType))\n else:\n #we query a sub-field of the relation\n related_table = self._collection_tables[params['collection']]\n\n if path_str in joins[related_table]:\n related_table_alias = joins[related_table][path_str]\n else:\n related_table_alias = related_table.alias()\n joins[related_table][path_str] = related_table_alias\n joins_list.append((related_table_alias,table.c[params['column']] == related_table_alias.c['pk']))\n where_statements.extend(compile_query(params['collection'],{tail : value},table = related_table_alias,path = new_path))\n elif isinstance(params['field'],OneToManyField):\n related_table = self._collection_tables[params['collection']]\n\n if path_str in joins[related_table]:\n related_table_alias = joins[related_table][path_str]\n else:\n related_table_alias = related_table.alias()\n joins[related_table][path_str] = related_table_alias\n joins_list.append((related_table_alias,related_table_alias.c[params['backref']['column']] == table.c['pk']))\n\n where_statements.extend(compile_one_to_many_query(key,value,field_name,related_table_alias,table.c.pk,new_path))\n break\n else:\n raise AttributeError(\"Query over non-indexed field %s in collection %s!\" % (key,collection))\n return where_statements\n\n compiled_query = compile_query(collection,query)\n\n if len(compiled_query) > 1:\n compiled_query = and_(*compiled_query)\n elif compiled_query:\n compiled_query = compiled_query[0]\n else:\n compiled_query = None\n\n return QuerySet(backend = self, table = table,\n joins = joins_list,\n cls = cls,\n condition = compiled_query,\n raw = raw,\n group_bys = group_bys,\n only = only,\n include = include,\n havings = havings\n )", "response": "This method returns a list of objects that correspond to a given set of properties."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ntransforms the query dictionary to replace e. g. documents with __ref__ fields.", "response": "def _canonicalize_query(self, query):\n\n \"\"\"\n Transform the query dictionary to replace e.g. documents with __ref__ fields.\n \"\"\"\n\n def transform_query(q):\n\n for encoder in self.query_encoders:\n q = encoder.encode(q,[])\n\n if isinstance(q, dict):\n nq = {}\n for key,value in q.items():\n new_key = key\n if isinstance(value,dict) and len(value) == 1 and list(value.keys())[0].startswith('$'):\n if list(value.keys())[0] in ('$all','$in'):\n if list(value.values())[0] and isinstance(list(value.values())[0][0],Document):\n if self._use_pk_based_refs:\n new_key+='.pk'\n else:\n new_key+='.__ref__'\n elif isinstance(value,Document):\n if self._use_pk_based_refs:\n new_key+='.pk'\n else:\n new_key+='.__ref__'\n nq[new_key] = transform_query(value)\n return nq\n elif isinstance(q, (list,QuerySet,tuple)):\n return [transform_query(x) for x in q]\n elif isinstance(q,Document):\n collection = self.get_collection_for_obj(q)\n if self._use_pk_based_refs:\n return q.pk\n else:\n return \"%s:%s\" % (collection,q.pk)\n else:\n return q\n\n return transform_query(query)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nfilter objects from the database that correspond to a given set of properties.", "response": "def filter(self, cls_or_collection, query, raw=False, only=None):\n \"\"\"\n Filter objects from the database that correspond to a given set of properties.\n\n See :py:meth:`blitzdb.backends.base.Backend.filter` for documentation of individual parameters\n\n .. note::\n\n This function supports most query operators that are available in MongoDB and returns\n a query set that is based on a MongoDB cursor.\n\n \"\"\"\n\n if not isinstance(cls_or_collection, six.string_types):\n collection = self.get_collection_for_cls(cls_or_collection)\n cls = cls_or_collection\n else:\n collection = cls_or_collection\n cls = self.get_cls_for_collection(collection)\n\n canonical_query = self._canonicalize_query(query)\n\n args = {}\n\n if only:\n if isinstance(only,tuple):\n args['projection'] = list(only)\n else:\n args['projection'] = only\n\n return QuerySet(self, cls, self.db[collection].find(canonical_query, **args), raw=raw, only=only)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngenerating boolean operator checking function.", "response": "def boolean_operator_query(boolean_operator):\n \"\"\"Generate boolean operator checking function.\"\"\"\n def _boolean_operator_query(expressions):\n \"\"\"Apply boolean operator to expressions.\"\"\"\n def _apply_boolean_operator(query_function, expressions=expressions):\n \"\"\"Return if expressions with boolean operator are satisfied.\"\"\"\n compiled_expressions = [compile_query(e) for e in expressions]\n return reduce(\n boolean_operator,\n [e(query_function) for e in compiled_expressions]\n )\n\n return _apply_boolean_operator\n return _boolean_operator_query"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfilter documents with a key that satisfies an expression.", "response": "def filter_query(key, expression):\n \"\"\"Filter documents with a key that satisfies an expression.\"\"\"\n if (isinstance(expression, dict)\n and len(expression) == 1\n and list(expression.keys())[0].startswith('$')):\n compiled_expression = compile_query(expression)\n elif callable(expression):\n def _filter(index, expression=expression):\n result = [store_key\n for value, store_keys in index.get_index().items()\n if expression(value)\n for store_key in store_keys]\n return result\n compiled_expression = _filter\n else:\n compiled_expression = expression\n\n def _get(query_function, key=key, expression=compiled_expression):\n \"\"\"Get document key and check against expression.\"\"\"\n return query_function(key, expression)\n\n return _get"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef not_query(expression):\n compiled_expression = compile_query(expression)\n\n def _not(index, expression=compiled_expression):\n \"\"\"Return store key for documents that satisfy expression.\"\"\"\n all_keys = index.get_all_keys()\n returned_keys = expression(index)\n return [key for key in all_keys if key not in returned_keys]\n\n return _not", "response": "Return a function that returns the store keys that satisfy the given logical not expression."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef comparison_operator_query(comparison_operator):\n def _comparison_operator_query(expression):\n \"\"\"Apply binary operator to expression.\"\"\"\n def _apply_comparison_operator(index, expression=expression):\n \"\"\"Return store key for documents that satisfy expression.\"\"\"\n ev = expression() if callable(expression) else expression\n return [\n store_key\n for value, store_keys\n in index.get_index().items()\n if comparison_operator(value, ev)\n for store_key in store_keys\n ]\n return _apply_comparison_operator\n return _comparison_operator_query", "response": "Generates a comparison operator checking function that returns a list of store keys that satisfy the given binary operator."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a function that checks that documents have a key that satisfies expression.", "response": "def exists_query(expression):\n \"\"\"Check that documents have a key that satisfies expression.\"\"\"\n def _exists(index, expression=expression):\n \"\"\"Return store key for documents that satisfy expression.\"\"\"\n ev = expression() if callable(expression) else expression\n if ev:\n return [\n store_key\n for store_keys\n in index.get_index().values()\n for store_key in store_keys\n ]\n else:\n return index.get_undefined_keys()\n\n return _exists"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a function that returns the store key for documents that satisfy the regular expression.", "response": "def regex_query(expression):\n \"\"\"Apply regular expression to result of expression.\"\"\"\n def _regex(index, expression=expression):\n \"\"\"Return store key for documents that satisfy expression.\"\"\"\n pattern = re.compile(expression)\n return [\n store_key\n for value, store_keys\n in index.get_index().items()\n if (isinstance(value, six.string_types)\n and re.match(pattern, value))\n for store_key in store_keys\n ]\n\n return _regex"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmatches arrays that contain all elements in the query.", "response": "def all_query(expression):\n \"\"\"Match arrays that contain all elements in the query.\"\"\"\n def _all(index, expression=expression):\n \"\"\"Return store key for documents that satisfy expression.\"\"\"\n ev = expression() if callable(expression) else expression\n try:\n iter(ev)\n except TypeError:\n raise AttributeError('$all argument must be an iterable!')\n\n hashed_ev = [index.get_hash_for(v) for v in ev]\n store_keys = set()\n\n if len(hashed_ev) == 0:\n return []\n store_keys = set(index.get_keys_for(hashed_ev[0]))\n for value in hashed_ev[1:]:\n store_keys &= set(index.get_keys_for(value))\n return list(store_keys)\n\n return _all"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a function that returns the store key for documents that satisfy the given expression.", "response": "def in_query(expression):\n \"\"\"Match any of the values that exist in an array specified in query.\"\"\"\n def _in(index, expression=expression):\n \"\"\"Return store key for documents that satisfy expression.\"\"\"\n ev = expression() if callable(expression) else expression\n try:\n iter(ev)\n except TypeError:\n raise AttributeError('$in argument must be an iterable!')\n hashed_ev = [index.get_hash_for(v) for v in ev]\n store_keys = set()\n\n for value in hashed_ev:\n store_keys |= set(index.get_keys_for(value))\n\n return list(store_keys)\n\n return _in"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncompiles each expression in query recursively.", "response": "def compile_query(query):\n \"\"\"Compile each expression in query recursively.\"\"\"\n if isinstance(query, dict):\n expressions = []\n for key, value in query.items():\n if key.startswith('$'):\n if key not in query_funcs:\n raise AttributeError('Invalid operator: {}'.format(key))\n expressions.append(query_funcs[key](value))\n else:\n expressions.append(filter_query(key, value))\n if len(expressions) > 1:\n return boolean_operator_query(operator.and_)(expressions)\n else:\n return (\n expressions[0]\n if len(expressions)\n else lambda query_function: query_function(None, None)\n )\n else:\n return query"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef environment_variable(key, value):\n\n if key not in os.environ:\n previous = None\n else:\n previous = os.environ[key]\n\n os.environ[key] = value\n\n yield\n\n if previous is None:\n del os.environ[key]\n else:\n os.environ[key] = previous", "response": "Temporarily overrides an environment variable."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nhook this module up to the given api.", "response": "def hookup(self, api):\n \"\"\" Hooks this module up to the given api. \"\"\"\n\n assert not hasattr(api, self.module_name), \"\"\"\n '{}' conflicts with existing attribute\n \"\"\".format(self.module_name)\n\n self.api = api\n\n setattr(api, self.module_name, self.execute)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nexecutes the module and returns the result.", "response": "def execute(self, *args, **kwargs):\n \"\"\" Puts args and kwargs in a way ansible can understand. Calls ansible\n and interprets the result.\n\n \"\"\"\n assert self.is_hooked_up, \"the module should be hooked up to the api\"\n\n if set_global_context:\n set_global_context(self.api.options)\n\n # legacy key=value pairs shorthand approach\n if args:\n self.module_args = module_args = self.get_module_args(args, kwargs)\n else:\n self.module_args = module_args = kwargs\n\n loader = DataLoader()\n inventory_manager = SourcelessInventoryManager(loader=loader)\n\n for host, port in self.api.hosts_with_ports:\n inventory_manager._inventory.add_host(host, group='all', port=port)\n\n for key, value in self.api.options.extra_vars.items():\n inventory_manager._inventory.set_variable('all', key, value)\n\n variable_manager = VariableManager(\n loader=loader, inventory=inventory_manager)\n\n play_source = {\n 'name': \"Suitable Play\",\n 'hosts': 'all',\n 'gather_facts': 'no',\n 'tasks': [{\n 'action': {\n 'module': self.module_name,\n 'args': module_args,\n },\n 'environment': self.api.environment,\n }]\n }\n\n try:\n play = Play.load(\n play_source,\n variable_manager=variable_manager,\n loader=loader,\n )\n\n if self.api.strategy:\n play.strategy = self.api.strategy\n\n log.info(\n u'running {}'.format(u'- {module_name}: {module_args}'.format(\n module_name=self.module_name,\n module_args=module_args\n ))\n )\n\n start = datetime.utcnow()\n task_queue_manager = None\n callback = SilentCallbackModule()\n\n # ansible uses various levels of verbosity (from -v to -vvvvvv)\n # offering various amounts of debug information\n #\n # we keep it a bit simpler by activating all of it during debug,\n # and falling back to the default of 0 otherwise\n verbosity = self.api.options.verbosity == logging.DEBUG and 6 or 0\n\n with ansible_verbosity(verbosity):\n\n # host_key_checking is special, since not each connection\n # plugin handles it the same way, we need to apply both\n # environment variable and Ansible constant when running a\n # command in the runner to be successful\n with host_key_checking(self.api.host_key_checking):\n kwargs = dict(\n inventory=inventory_manager,\n variable_manager=variable_manager,\n loader=loader,\n options=self.api.options,\n passwords=getattr(self.api.options, 'passwords', {}),\n stdout_callback=callback\n )\n\n if set_global_context:\n del kwargs['options']\n\n task_queue_manager = TaskQueueManager(**kwargs)\n\n try:\n task_queue_manager.run(play)\n except SystemExit:\n\n # Mitogen forks our process and exits it in one\n # instance before returning\n #\n # This is fine, but it does lead to a very messy exit\n # by py.test which will essentially return with a test\n # that is first successful and then failed as each\n # forked process dies.\n #\n # To avoid this we commit suicide if we are run inside\n # a pytest session. Normally this would just result\n # in a exit code of zero, which is good.\n if 'pytest' in sys.modules:\n try:\n atexit._run_exitfuncs()\n except Exception:\n pass\n os.kill(os.getpid(), signal.SIGKILL)\n\n raise\n finally:\n if task_queue_manager is not None:\n task_queue_manager.cleanup()\n\n if set_global_context:\n\n # Ansible 2.8 introduces a global context which persists\n # during the lifetime of the process - for Suitable this\n # singleton/cache needs to be cleared after each call\n # to make sure that API calls do not carry over state.\n #\n # The docs hint at a future inclusion of local contexts, which\n # would of course be preferable.\n from ansible.utils.context_objects import GlobalCLIArgs\n GlobalCLIArgs._Singleton__instance = None\n\n log.debug(u'took {} to complete'.format(datetime.utcnow() - start))\n\n return self.evaluate_results(callback)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nremoves a server from the list of servers.", "response": "def ignore_further_calls_to_server(self, server):\n \"\"\" Takes a server out of the list. \"\"\"\n log.error(u'ignoring further calls to {}'.format(server))\n self.api.servers.remove(server)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nevaluates the results of a runner call.", "response": "def evaluate_results(self, callback):\n \"\"\" prepare the result of runner call for use with RunnerResults. \"\"\"\n\n for server, result in callback.unreachable.items():\n log.error(u'{} could not be reached'.format(server))\n log.debug(u'ansible-output =>\\n{}'.format(pformat(result)))\n\n if self.api.ignore_unreachable:\n continue\n\n self.trigger_event(server, 'on_unreachable_host', (\n self, server\n ))\n\n for server, answer in callback.contacted.items():\n\n success = answer['success']\n result = answer['result']\n\n # none of the modules in our tests hit the 'failed' result\n # codepath (which seems to not be implemented by all modules)\n # seo we ignore this branch since it's rather trivial\n if result.get('failed'): # pragma: no cover\n success = False\n\n if 'rc' in result:\n if self.api.is_valid_return_code(result['rc']):\n success = True\n\n if not success:\n log.error(u'{} failed on {}'.format(self, server))\n log.debug(u'ansible-output =>\\n{}'.format(pformat(result)))\n\n if self.api.ignore_errors:\n continue\n\n self.trigger_event(server, 'on_module_error', (\n self, server, result\n ))\n\n # XXX this is a weird structure because RunnerResults still works\n # like it did with Ansible 1.x, where the results where structured\n # like this\n return RunnerResults({\n 'contacted': {\n server: answer['result']\n for server, answer in callback.contacted.items()\n }\n })"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef install_strategy_plugins(directories):\n if isinstance(directories, str):\n directories = directories.split(':')\n\n for directory in directories:\n strategy_loader.add_directory(directory)", "response": "Installs the given strategy plugins."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef valid_return_codes(self, *codes):\n previous_codes = self._valid_return_codes\n self._valid_return_codes = codes\n\n yield\n\n self._valid_return_codes = previous_codes", "response": "Sets codes which are considered valid when returned from a context module."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconstructing the URL with a given IP address.", "response": "def construct_url(ip_address: str) -> str:\r\n \"\"\"Construct the URL with a given IP address.\"\"\"\r\n if 'http://' not in ip_address and 'https://' not in ip_address:\r\n ip_address = '{}{}'.format('http://', ip_address)\r\n if ip_address[-1] == '/':\r\n ip_address = ip_address[:-1]\r\n return ip_address"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nupdating the data from the printer.", "response": "async def update(self) -> None:\r\n \"\"\"\r\n Retrieve the data from the printer.\r\n Throws ValueError if host does not support SyncThru\r\n \"\"\"\r\n url = '{}{}'.format(self.url, ENDPOINT)\r\n\r\n try:\r\n async with self._session.get(url) as response:\r\n json_dict = demjson.decode(await response.text(), strict=False)\r\n except aiohttp.ClientError:\r\n json_dict = {'status': {'status1': SyncThru.OFFLINE}}\r\n except demjson.JSONDecodeError:\r\n raise ValueError(\"Invalid host, does not support SyncThru.\")\r\n self.data = json_dict"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the model name of the printer.", "response": "def model(self):\r\n \"\"\"Return the model name of the printer.\"\"\"\r\n try:\r\n return self.data.get('identity').get('model_name')\r\n except (KeyError, AttributeError):\r\n return self.device_status_simple('')"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef location(self):\r\n try:\r\n return self.data.get('identity').get('location')\r\n except (KeyError, AttributeError):\r\n return self.device_status_simple('')", "response": "Return the location of the printer."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef serial_number(self):\r\n try:\r\n return self.data.get('identity').get('serial_num')\r\n except (KeyError, AttributeError):\r\n return self.device_status_simple('')", "response": "Return the serial number of the printer."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the hostname of the printer.", "response": "def hostname(self):\r\n \"\"\"Return the hostname of the printer.\"\"\"\r\n try:\r\n return self.data.get('identity').get('host_name')\r\n except (KeyError, AttributeError):\r\n return self.device_status_simple('')"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef device_status(self):\r\n try:\r\n return self.device_status_simple(\r\n self.data.get('status').get('status1'))\r\n except (KeyError, AttributeError):\r\n return self.device_status_simple('')", "response": "Return the status of the device as string."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef capability(self) -> Dict[str, Any]:\r\n try:\r\n return self.data.get('capability', {})\r\n except (KeyError, AttributeError):\r\n return {}", "response": "Return the capabilities of the printer."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the state of all toners cartridges.", "response": "def toner_status(self, filter_supported: bool = True) -> Dict[str, Any]:\r\n \"\"\"Return the state of all toners cartridges.\"\"\"\r\n toner_status = {}\r\n for color in self.COLOR_NAMES:\r\n try:\r\n toner_stat = self.data.get(\r\n '{}_{}'.format(SyncThru.TONER, color), {})\r\n if filter_supported and toner_stat.get('opt', 0) == 0:\r\n continue\r\n else:\r\n toner_status[color] = toner_stat\r\n except (KeyError, AttributeError):\r\n toner_status[color] = {}\r\n return toner_status"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the state of all input trays.", "response": "def input_tray_status(self,\r\n filter_supported: bool = True) -> Dict[int, Any]:\r\n \"\"\"Return the state of all input trays.\"\"\"\r\n tray_status = {}\r\n for i in range(1, 5):\r\n try:\r\n tray_stat = self.data.get('{}{}'.format(SyncThru.TRAY, i), {})\r\n if filter_supported and tray_stat.get('opt', 0) == 0:\r\n continue\r\n else:\r\n tray_status[i] = tray_stat\r\n except (KeyError, AttributeError):\r\n tray_status[i] = {}\r\n return tray_status"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the state of all output trays.", "response": "def output_tray_status(self) -> Dict[int, Dict[str, str]]:\r\n \"\"\"Return the state of all output trays.\"\"\"\r\n tray_status = {}\r\n try:\r\n tray_stat = self.data.get('outputTray', [])\r\n for i, stat in enumerate(tray_stat):\r\n tray_status[i] = {\r\n 'name': stat[0],\r\n 'capacity': stat[1],\r\n 'status': stat[2],\r\n }\r\n except (KeyError, AttributeError):\r\n tray_status = {}\r\n return tray_status"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the state of all drums.", "response": "def drum_status(self, filter_supported: bool = True) -> Dict[str, Any]:\r\n \"\"\"Return the state of all drums.\"\"\"\r\n drum_status = {}\r\n for color in self.COLOR_NAMES:\r\n try:\r\n drum_stat = self.data.get('{}_{}'.format(SyncThru.DRUM, color),\r\n {})\r\n if filter_supported and drum_stat.get('opt', 0) == 0:\r\n continue\r\n else:\r\n drum_status[color] = drum_stat\r\n except (KeyError, AttributeError):\r\n drum_status[color] = {}\r\n return drum_status"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncall func with args and kwargs and runs pdb.", "response": "def pdb_run(func: Callable, *args: Any, **kwargs: Any) -> None:\n \"\"\"\n Calls ``func(*args, **kwargs)``; if it raises an exception, break into\n the ``pdb`` debugger.\n \"\"\"\n # noinspection PyBroadException\n try:\n func(*args, **kwargs)\n except: # nopep8\n type_, value, tb = sys.exc_info()\n traceback.print_exc()\n pdb.post_mortem(tb)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_class_name_from_frame(fr: FrameType) -> Optional[str]:\n # http://stackoverflow.com/questions/2203424/python-how-to-retrieve-class-information-from-a-frame-object # noqa\n args, _, _, value_dict = inspect.getargvalues(fr)\n # we check the first parameter for the frame function is named 'self'\n if len(args) and args[0] == 'self':\n # in that case, 'self' will be referenced in value_dict\n instance = value_dict.get('self', None)\n if instance:\n # return its class\n cls = getattr(instance, '__class__', None)\n if cls:\n return cls.__name__\n return None\n # return None otherwise\n return None", "response": "This function attempts to retrieve the class name from a frame object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn details about the caller of this function.", "response": "def get_caller_name(back: int = 0) -> str:\n \"\"\"\n Return details about the CALLER OF THE CALLER (plus n calls further back)\n of this function.\n\n So, if your function calls :func:`get_caller_name`, it will return the\n name of the function that called your function! (Or ``back`` calls further\n back.)\n\n Example:\n\n .. code-block:: python\n\n from cardinal_pythonlib.debugging import get_caller_name\n\n def who_am_i():\n return get_caller_name()\n\n class MyClass(object):\n def classfunc(self):\n print(\"I am: \" + who_am_i())\n print(\"I was called by: \" + get_caller_name())\n print(\"That was called by: \" + get_caller_name(back=1))\n\n def f2():\n x = MyClass()\n x.classfunc()\n\n def f1():\n f2()\n\n f1()\n\n will produce:\n\n .. code-block:: none\n\n I am: MyClass.classfunc\n I was called by: f2\n That was called by: f1\n\n \"\"\"\n # http://stackoverflow.com/questions/5067604/determine-function-name-from-within-that-function-without-using-traceback # noqa\n try:\n # noinspection PyProtectedMember\n frame = sys._getframe(back + 2)\n except ValueError:\n # Stack isn't deep enough.\n return '?'\n function_name = frame.f_code.co_name\n class_name = get_class_name_from_frame(frame)\n if class_name:\n return \"{}.{}\".format(class_name, function_name)\n return function_name"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsend details about a Python object to the log.", "response": "def debug_object(obj, log_level: int = logging.DEBUG) -> None:\n \"\"\"\n Sends details about a Python to the log, specifically its ``repr()``\n representation, and all of its attributes with their name, value, and type.\n\n Args:\n obj: object to debug\n log_level: log level to use; default is ``logging.DEBUG``\n \"\"\"\n msgs = [\"For {o!r}:\".format(o=obj)]\n for attrname in dir(obj):\n attribute = getattr(obj, attrname)\n msgs.append(\"- {an!r}: {at!r}, of type {t!r}\".format(\n an=attrname, at=attribute, t=type(attribute)))\n log.log(log_level, \"{}\", \"\\n\".join(msgs))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef from_db_value(self, value, expression, connection, context):\n if value is None:\n return value\n return json_decode(value)", "response": "Called in all circumstances when the data is loaded from the\n database including in aggregates and values() calls."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse(xml_input, encoding=None, expat=expat, process_namespaces=False,\n namespace_separator=':', **kwargs):\n \"\"\"Parse the given XML input and convert it into a dictionary.\n\n `xml_input` can either be a `string` or a file-like object.\n\n If `xml_attribs` is `True`, element attributes are put in the dictionary\n among regular child elements, using `@` as a prefix to avoid collisions. If\n set to `False`, they are just ignored.\n\n Simple example::\n\n >>> import xmltodict\n >>> doc = xmltodict.parse(\\\"\\\"\\\"\n ... <a prop=\"x\">\n ... <b>1</b>\n ... <b>2</b>\n ... </a>\n ... \\\"\\\"\\\")\n >>> doc['a']['@prop']\n u'x'\n >>> doc['a']['b']\n [u'1', u'2']\n\n If `item_depth` is `0`, the function returns a dictionary for the root\n element (default behavior). Otherwise, it calls `item_callback` every time\n an item at the specified depth is found and returns `None` in the end\n (streaming mode).\n\n The callback function receives two parameters: the `path` from the document\n root to the item (name-attribs pairs), and the `item` (dict). If the\n callback's return value is false-ish, parsing will be stopped with the\n :class:`ParsingInterrupted` exception.\n\n Streaming example::\n\n >>> def handle(path, item):\n ... print 'path:%s item:%s' % (path, item)\n ... return True\n ...\n >>> xmltodict.parse(\\\"\\\"\\\"\n ... <a prop=\"x\">\n ... <b>1</b>\n ... <b>2</b>\n ... </a>\\\"\\\"\\\", item_depth=2, item_callback=handle)\n path:[(u'a', {u'prop': u'x'}), (u'b', None)] item:1\n path:[(u'a', {u'prop': u'x'}), (u'b', None)] item:2\n\n The optional argument `postprocessor` is a function that takes `path`,\n `key` and `value` as positional arguments and returns a new `(key, value)`\n pair where both `key` and `value` may have changed. Usage example::\n\n >>> def postprocessor(path, key, value):\n ... try:\n ... return key + ':int', int(value)\n ... except (ValueError, TypeError):\n ... return key, value\n >>> xmltodict.parse('<a><b>1</b><b>2</b><b>x</b></a>',\n ... postprocessor=postprocessor)\n OrderedDict([(u'a', OrderedDict([(u'b:int', [1, 2]), (u'b', u'x')]))])\n\n You can pass an alternate version of `expat` (such as `defusedexpat`) by\n using the `expat` parameter. E.g:\n\n >>> import defusedexpat\n >>> xmltodict.parse('<a>hello</a>', expat=defusedexpat.pyexpat)\n OrderedDict([(u'a', u'hello')])\n\n You can use the force_list argument to force lists to be created even\n when there is only a single child of a given level of hierarchy. The\n force_list argument is a tuple of keys. If the key for a given level\n of hierarchy is in the force_list argument, that level of hierarchy\n will have a list as a child (even if there is only one sub-element).\n The index_keys operation takes precendence over this. This is applied\n after any user-supplied postprocessor has already run.\n\n For example, given this input:\n <servers>\n <server>\n <name>host1</name>\n <os>Linux</os>\n <interfaces>\n <interface>\n <name>em0</name>\n <ip_address>10.0.0.1</ip_address>\n </interface>\n </interfaces>\n </server>\n </servers>\n\n If called with force_list=('interface',), it will produce\n this dictionary:\n {'servers':\n {'server':\n {'name': 'host1',\n 'os': 'Linux'},\n 'interfaces':\n {'interface':\n [ {'name': 'em0', 'ip_address': '10.0.0.1' } ] } } }\n\n `force_list` can also be a callable that receives `path`, `key` and\n `value`. This is helpful in cases where the logic that decides whether\n a list should be forced is more complex.\n \"\"\"\n handler = _DictSAXHandler(namespace_separator=namespace_separator,\n **kwargs)\n if isinstance(xml_input, _unicode):\n if not encoding:\n encoding = 'utf-8'\n xml_input = xml_input.encode(encoding)\n if not process_namespaces:\n namespace_separator = None\n parser = expat.ParserCreate(\n encoding,\n namespace_separator\n )\n try:\n parser.ordered_attributes = True\n except AttributeError:\n # Jython's expat does not support ordered_attributes\n pass\n parser.StartElementHandler = handler.startElement\n parser.EndElementHandler = handler.endElement\n parser.CharacterDataHandler = handler.characters\n parser.buffer_text = True\n try:\n parser.ParseFile(xml_input)\n except (TypeError, AttributeError):\n parser.Parse(xml_input, True)\n return handler.item", "response": "Parse the given XML input and return a dictionary."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nmatching C ++ class declarations.", "response": "def cxxRecordDecl(*args):\n \"\"\"Matches C++ class declarations.\n\n >>> from glud import *\n >>> config = '''\n ... class W;\n ... template<typename T> class X {};\n ... struct Y {};\n ... union Z {};\n ... '''\n >>> m = cxxRecordDecl()\n >>> for c in walk(m, parse_string(config).cursor):\n ... print(c.spelling)\n W\n X\n \"\"\"\n kinds = [\n CursorKind.CLASS_DECL,\n CursorKind.CLASS_TEMPLATE,\n ]\n inner = [ PredMatcher(is_kind(k)) for k in kinds ]\n return allOf(anyOf(*inner), *args)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nmatch class struct and union declarations.", "response": "def recordDecl(*args):\n \"\"\"Matches class, struct, and union declarations.\n\n >>> from glud import *\n >>> config = '''\n ... class W;\n ... template<typename T> class X {};\n ... struct Y {};\n ... union Z {};\n ... '''\n >>> m = recordDecl()\n >>> for c in walk(m, parse_string(config).cursor):\n ... print(c.spelling)\n W\n X\n Y\n Z\n \"\"\"\n kinds = [\n CursorKind.STRUCT_DECL,\n CursorKind.UNION_DECL,\n CursorKind.CLASS_DECL,\n CursorKind.CLASS_TEMPLATE,\n ]\n inner = [ PredMatcher(is_kind(k)) for k in kinds ]\n return allOf(anyOf(*inner), *args)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef convert_to_bool(x: Any, default: bool = None) -> bool:\n if isinstance(x, bool):\n return x\n\n if not x: # None, zero, blank string...\n return default\n\n try:\n return int(x) != 0\n except (TypeError, ValueError):\n pass\n\n try:\n return float(x) != 0\n except (TypeError, ValueError):\n pass\n\n if not isinstance(x, str):\n raise Exception(\"Unknown thing being converted to bool: {!r}\".format(x))\n\n x = x.upper()\n if x in [\"Y\", \"YES\", \"T\", \"TRUE\"]:\n return True\n if x in [\"N\", \"NO\", \"F\", \"FALSE\"]:\n return False\n\n raise Exception(\"Unknown thing being converted to bool: {!r}\".format(x))", "response": "Converts a string or integer to a boolean."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef convert_to_int(x: Any, default: int = None) -> int:\n try:\n return int(x)\n except (TypeError, ValueError):\n return default", "response": "Transforms its input into an integer or returns default."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef convert_attrs_to_bool(obj: Any,\n attrs: Iterable[str],\n default: bool = None) -> None:\n \"\"\"\n Applies :func:`convert_to_bool` to the specified attributes of an object,\n modifying it in place.\n \"\"\"\n for a in attrs:\n setattr(obj, a, convert_to_bool(getattr(obj, a), default=default))", "response": "Applies convert_to_bool to the specified attributes of an object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef convert_attrs_to_uppercase(obj: Any, attrs: Iterable[str]) -> None:\n for a in attrs:\n value = getattr(obj, a)\n if value is None:\n continue\n setattr(obj, a, value.upper())", "response": "Converts the specified attributes of an object to upper case."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef convert_attrs_to_lowercase(obj: Any, attrs: Iterable[str]) -> None:\n for a in attrs:\n value = getattr(obj, a)\n if value is None:\n continue\n setattr(obj, a, value.lower())", "response": "Converts the specified attributes of an object to lower case."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef convert_attrs_to_int(obj: Any,\n attrs: Iterable[str],\n default: int = None) -> None:\n \"\"\"\n Applies :func:`convert_to_int` to the specified attributes of an object,\n modifying it in place.\n \"\"\"\n for a in attrs:\n value = convert_to_int(getattr(obj, a), default=default)\n setattr(obj, a, value)", "response": "Applies convert_to_int to the specified attributes of an object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndecode a string into a byte array.", "response": "def hex_xformat_decode(s: str) -> Optional[bytes]:\n \"\"\"\n Reverse :func:`hex_xformat_encode`.\n\n The parameter is a hex-encoded BLOB like\n\n .. code-block:: none\n\n \"X'CDE7A24B1A9DBA3148BCB7A0B9DA5BB6A424486C'\"\n\n Original purpose and notes:\n\n - SPECIAL HANDLING for BLOBs: a string like ``X'01FF'`` means a hex-encoded\n BLOB. Titanium is rubbish at BLOBs, so we encode them as special string\n literals.\n - SQLite uses this notation: https://sqlite.org/lang_expr.html\n - Strip off the start and end and convert it to a byte array:\n http://stackoverflow.com/questions/5649407\n \"\"\"\n if len(s) < 3 or not s.startswith(\"X'\") or not s.endswith(\"'\"):\n return None\n return binascii.unhexlify(s[2:-1])"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndecode base64 encoded string.", "response": "def base64_64format_decode(s: str) -> Optional[bytes]:\n \"\"\"\n Reverse :func:`base64_64format_encode`.\n\n Original purpose and notes:\n\n - THIS IS ANOTHER WAY OF DOING BLOBS: base64 encoding, e.g. a string like\n ``64'cGxlYXN1cmUu'`` is a base-64-encoded BLOB (the ``64'...'`` bit is my\n representation).\n - regex from http://stackoverflow.com/questions/475074\n - better one from http://www.perlmonks.org/?node_id=775820\n\n \"\"\"\n if len(s) < 4 or not s.startswith(\"64'\") or not s.endswith(\"'\"):\n return None\n return base64.b64decode(s[3:-1])"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfetches a setting from the environment.", "response": "def environ_setting(name, default=None, required=True):\n \"\"\"\n Fetch setting from the environment. The bahavior of the setting if it\n is not in environment is as follows:\n\n 1. If it is required and the default is None, raise Exception\n 2. If it is requried and a default exists, return default\n 3. If it is not required and default is None, return None\n 4. If it is not required and default exists, return default\n \"\"\"\n if name not in os.environ and default is None:\n message = \"The {0} ENVVAR is not set.\".format(name)\n if required:\n raise ImproperlyConfigured(message)\n else:\n warnings.warn(ConfigurationMissing(message))\n\n return os.environ.get(name, default)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef load(klass):\n config = klass()\n for path in klass.CONF_PATHS:\n if os.path.exists(path):\n with open(path, 'r') as conf:\n config.configure(yaml.safe_load(conf))\n return config", "response": "Loads the configuration from YAML files specified by the CONF_PATH module\n variable."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nallow updating of the configuration of the object via a dictionary of configuration terms or a configuration object. Generally speaking this method is utilized to configure the object from a JSON or a YAML parsing.", "response": "def configure(self, conf={}):\n \"\"\"\n Allows updating of the configuration via a dictionary of\n configuration terms or a configuration object. Generally speaking,\n this method is utilized to configure the object from a JSON or\n YAML parsing.\n \"\"\"\n if not conf: return\n if isinstance(conf, Configuration):\n conf = dict(conf.options())\n for key, value in conf.items():\n opt = self.get(key, None)\n if isinstance(opt, Configuration):\n opt.configure(value)\n else:\n setattr(self, key, value)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef options(self):\n keys = self.__class__.__dict__.copy()\n keys.update(self.__dict__)\n keys = sorted(keys.keys())\n\n for opt in keys:\n val = self.get(opt)\n if val is not None:\n yield opt, val", "response": "Returns an iterable of sorted option names and values for all the configuration directives in the class."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nformatting a datetime. date to ISO - 8601 basic format.", "response": "def formatdt(date: datetime.date, include_time: bool = True) -> str:\n \"\"\"\n Formats a ``datetime.date`` to ISO-8601 basic format, to minute accuracy\n with no timezone (or, if ``include_time`` is ``False``, omit the time).\n \"\"\"\n if include_time:\n return date.strftime(\"%Y-%m-%dT%H:%M\")\n else:\n return date.strftime(\"%Y-%m-%d\")"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting a datetime. timedelta object into other units.", "response": "def convert_duration(duration: datetime.timedelta,\n units: str) -> Optional[float]:\n \"\"\"\n Convert a ``datetime.timedelta`` object -- a duration -- into other\n units. Possible units:\n\n ``s``, ``sec``, ``seconds``\n ``m``, ``min``, ``minutes``\n ``h``, ``hr``, ``hours``\n ``d``, ``days``\n ``w``, ``weeks``\n ``y``, ``years``\n \"\"\"\n if duration is None:\n return None\n s = duration.total_seconds()\n if units in ['s', 'sec', 'seconds']:\n return s\n if units in ['m', 'min', 'minutes']:\n return s / SECONDS_PER_MINUTE\n if units in ['h', 'hr', 'hours']:\n return s / SECONDS_PER_HOUR\n if units in ['d', 'days']:\n return s / SECONDS_PER_DAY\n if units in ['w', 'weeks']:\n return s / SECONDS_PER_WEEK\n if units in ['y', 'years']:\n return s / SECONDS_PER_YEAR\n raise ValueError(\"Unknown units: {}\".format(units))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef overlaps(self, other: \"Interval\") -> bool:\n return not(self.end <= other.start or self.start >= other.end)", "response": "Returns True if this interval overlaps other."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef contiguous(self, other: \"Interval\") -> bool:\n return not(self.end < other.start or self.start > other.end)", "response": "Is this interval contiguous?"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns True if the interval contains the given time.", "response": "def contains(self, time: datetime.datetime,\n inclusive: bool = True) -> bool:\n \"\"\"\n Does the interval contain a momentary time?\n\n Args:\n time: the ``datetime.datetime`` to check\n inclusive: use inclusive rather than exclusive range checks?\n \"\"\"\n if inclusive:\n return self.start <= time <= self.end\n else:\n return self.start < time < self.end"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning True if this interval is contained within the other.", "response": "def within(self, other: \"Interval\", inclusive: bool = True) -> bool:\n \"\"\"\n Is this interval contained within the other?\n\n Args:\n other: the :class:`Interval` to check\n inclusive: use inclusive rather than exclusive range checks?\n \"\"\"\n if not other:\n return False\n if inclusive:\n return self.start >= other.start and self.end <= other.end\n else:\n return self.start > other.start and self.end < other.end"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef intersection(self, other: \"Interval\") -> Optional[\"Interval\"]:\n if not self.contiguous(other):\n return None\n return Interval(\n max(self.start, other.start),\n min(self.end, other.end)\n )", "response": "Returns an interval representing the intersection of this and the other."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a list of intervals produced by using times as a list of knives to slice this interval.", "response": "def cut(self, times: Union[datetime.datetime,\n List[datetime.datetime]]) -> List[\"Interval\"]:\n \"\"\"\n Returns a list of intervals produced by using times (a list of\n ``datetime.datetime`` objects, or a single such object) as a set of\n knives to slice this interval.\n \"\"\"\n if not isinstance(times, list):\n # Single time\n time = times\n if not self.contains(time):\n return []\n return [\n Interval(self.start, time),\n Interval(time, self.end)\n ]\n else:\n # Multiple times\n times = [t for t in times if self.contains(t)] # discard others\n times.sort()\n times = [self.start] + times + [self.end]\n intervals = []\n for i in range(len(times) - 1):\n intervals.append(Interval(times[i], times[i + 1]))\n return intervals"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns an interval covering the given date.", "response": "def wholeday(date: datetime.date) -> \"Interval\":\n \"\"\"\n Returns an :class:`Interval` covering the date given (midnight at the\n start of that day to midnight at the start of the next day).\n \"\"\"\n start = datetime.datetime.combine(date, datetime.time())\n return Interval(\n start,\n start + datetime.timedelta(days=1)\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns an :class:`Interval` representing daytime on the date given.", "response": "def daytime(date: datetime.date,\n daybreak: datetime.time = datetime.time(NORMAL_DAY_START_H),\n nightfall: datetime.time = datetime.time(NORMAL_DAY_END_H)) \\\n -> \"Interval\":\n \"\"\"\n Returns an :class:`Interval` representing daytime on the date given.\n \"\"\"\n return Interval(\n datetime.datetime.combine(date, daybreak),\n datetime.datetime.combine(date, nightfall),\n )"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn an interval representing the date range given from startdate to enddate.", "response": "def dayspan(startdate: datetime.date,\n enddate: datetime.date,\n include_end: bool = True) -> Optional[\"Interval\"]:\n \"\"\"\n Returns an :class:`Interval` representing the date range given, from\n midnight at the start of the first day to midnight at the end of the\n last (i.e. at the start of the next day after the last), or if\n include_end is False, 24h before that.\n\n If the parameters are invalid, returns ``None``.\n \"\"\"\n if enddate < startdate:\n return None\n if enddate == startdate and include_end:\n return None\n start_dt = datetime.datetime.combine(startdate, datetime.time())\n end_dt = datetime.datetime.combine(enddate, datetime.time())\n if include_end:\n end_dt += datetime.timedelta(days=1)\n return Interval(start_dt, end_dt)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the part of this interval that falls on the given date.", "response": "def component_on_date(self, date: datetime.date) -> Optional[\"Interval\"]:\n \"\"\"\n Returns the part of this interval that falls on the date given, or\n ``None`` if the interval doesn't have any part during that date.\n \"\"\"\n return self.intersection(Interval.wholeday(date))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a tuple of the duration of this interval that falls into day and night.", "response": "def day_night_duration(\n self,\n daybreak: datetime.time = datetime.time(NORMAL_DAY_START_H),\n nightfall: datetime.time = datetime.time(NORMAL_DAY_END_H)) \\\n -> Tuple[datetime.timedelta, datetime.timedelta]:\n \"\"\"\n Returns a ``(day, night)`` tuple of ``datetime.timedelta`` objects\n giving the duration of this interval that falls into day and night\n respectively.\n \"\"\"\n daytotal = datetime.timedelta()\n nighttotal = datetime.timedelta()\n startdate = self.start.date()\n enddate = self.end.date()\n ndays = (enddate - startdate).days + 1\n for i in range(ndays):\n date = startdate + datetime.timedelta(days=i)\n component = self.component_on_date(date)\n # ... an interval on a single day\n day = Interval.daytime(date, daybreak, nightfall)\n daypart = component.intersection(day)\n if daypart is not None:\n daytotal += daypart.duration()\n nighttotal += component.duration() - daypart.duration()\n else:\n nighttotal += component.duration()\n return daytotal, nighttotal"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the number of hours outside the given time range.", "response": "def duration_outside_nwh(\n self,\n starttime: datetime.time = datetime.time(NORMAL_DAY_START_H),\n endtime: datetime.time = datetime.time(NORMAL_DAY_END_H),\n weekdays_only: bool = False,\n weekends_only: bool = False) -> datetime.timedelta:\n \"\"\"\n Returns a duration (a ``datetime.timedelta`` object) representing the\n number of hours outside normal working hours.\n\n This is not simply a subset of :meth:`day_night_duration`, because\n weekends are treated differently (they are always out of hours).\n\n The options allow the calculation of components on weekdays or weekends\n only.\n \"\"\"\n if weekdays_only and weekends_only:\n raise ValueError(\"Can't have weekdays_only and weekends_only\")\n ooh = datetime.timedelta() # ooh = out of (normal) hours\n startdate = self.start.date()\n enddate = self.end.date()\n ndays = (enddate - startdate).days + 1\n for i in range(ndays):\n date = startdate + datetime.timedelta(days=i)\n component = self.component_on_date(date)\n # ... an interval on a single day\n if not is_normal_working_day(date):\n if weekdays_only:\n continue\n ooh += component.duration() # all is out-of-normal-hours\n else:\n if weekends_only:\n continue\n normalday = Interval.daytime(date, starttime, endtime)\n normalpart = component.intersection(normalday)\n if normalpart is not None:\n ooh += component.duration() - normalpart.duration()\n else:\n ooh += component.duration()\n return ooh"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef saturdays_of_weekends(self) -> Set[datetime.date]:\n startdate = self.start.date()\n enddate = self.end.date()\n ndays = (enddate - startdate).days + 1\n saturdays = set()\n for i in range(ndays):\n date = startdate + datetime.timedelta(days=i)\n if is_saturday(date):\n saturdays.add(date)\n elif is_sunday(date):\n saturdays.add(date - datetime.timedelta(days=1))\n return saturdays", "response": "Returns the set of dates of all Saturdays that this calendar contains."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef copy(self, no_overlap: bool = None,\n no_contiguous: bool = None) -> \"IntervalList\":\n \"\"\"\n Makes and returns a copy of the :class:`IntervalList`. The\n ``no_overlap``/``no_contiguous`` parameters can be changed.\n\n Args:\n no_overlap: merge intervals that overlap (now and on subsequent\n addition)?\n no_contiguous: if ``no_overlap`` is set, merge intervals that are\n contiguous too?\n \"\"\"\n if no_overlap is None:\n no_overlap = self.no_overlap\n if no_contiguous is None:\n no_contiguous = self.no_contiguous\n return IntervalList(self.intervals, no_overlap=no_overlap,\n no_contiguous=no_contiguous)", "response": "Makes and returns a copy of the : class : IntervalList."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding an interval to the list.", "response": "def add(self, interval: Interval) -> None:\n \"\"\"\n Adds an interval to the list. If ``self.no_overlap`` is True, as is the\n default, it will merge any overlapping intervals thus created.\n \"\"\"\n if interval is None:\n return\n if not isinstance(interval, Interval):\n raise TypeError(\n \"Attempt to insert non-Interval into IntervalList\")\n self.intervals.append(interval)\n self._tidy()"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nremoves overlaps etc. and sorts.", "response": "def _tidy(self) -> None:\n \"\"\"\n Removes overlaps, etc., and sorts.\n \"\"\"\n if self.no_overlap:\n self.remove_overlap(self.no_contiguous) # will sort\n else:\n self._sort()"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _remove_overlap_sub(self, also_remove_contiguous: bool) -> bool:\n # Returns\n for i in range(len(self.intervals)):\n for j in range(i + 1, len(self.intervals)):\n first = self.intervals[i]\n second = self.intervals[j]\n if also_remove_contiguous:\n test = first.contiguous(second)\n else:\n test = first.overlaps(second)\n if test:\n newint = first.union(second)\n self.intervals.pop(j)\n self.intervals.pop(i) # note that i must be less than j\n self.intervals.append(newint)\n return True\n return False", "response": "Called by _remove_overlap to remove the first overlap found."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nremove any overlapping intervals.", "response": "def remove_overlap(self, also_remove_contiguous: bool = False) -> None:\n \"\"\"\n Merges any overlapping intervals.\n\n Args:\n also_remove_contiguous: treat contiguous (as well as overlapping)\n intervals as worthy of merging?\n \"\"\"\n overlap = True\n while overlap:\n overlap = self._remove_overlap_sub(also_remove_contiguous)\n self._sort()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _any_overlap_or_contiguous(self, test_overlap: bool) -> bool:\n for i in range(len(self.intervals)):\n for j in range(i + 1, len(self.intervals)):\n first = self.intervals[i]\n second = self.intervals[j]\n if test_overlap:\n test = first.overlaps(second)\n else:\n test = first.contiguous(second)\n if test:\n return True\n return False", "response": "Test if any of the intervals overlap?"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef start_datetime(self) -> Optional[datetime.datetime]:\n if not self.intervals:\n return None\n return self.intervals[0].start", "response": "Returns the start date of the set of intervals or None if there is no start date."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef end_datetime(self) -> Optional[datetime.datetime]:\n if not self.intervals:\n return None\n return max([x.end for x in self.intervals])", "response": "Returns the end date of the set of intervals or None if there is no end date."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the start date of the set of intervals or None if there is no start date.", "response": "def start_date(self) -> Optional[datetime.date]:\n \"\"\"\n Returns the start date of the set of intervals, or ``None`` if empty.\n \"\"\"\n if not self.intervals:\n return None\n return self.start_datetime().date()"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef end_date(self) -> Optional[datetime.date]:\n if not self.intervals:\n return None\n return self.end_datetime().date()", "response": "Returns the end date of the set of intervals or None if there is no end date."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the earliest start of an interval running to the latest end of the anatomical set.", "response": "def extent(self) -> Optional[Interval]:\n \"\"\"\n Returns an :class:`Interval` running from the earliest start of an\n interval in this list to the latest end. Returns ``None`` if we are\n empty.\n \"\"\"\n if not self.intervals:\n return None\n return Interval(self.start_datetime(), self.end_datetime())"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the total sum of all the time - counted durations of all the items in the set.", "response": "def total_duration(self) -> datetime.timedelta:\n \"\"\"\n Returns a ``datetime.timedelta`` object with the total sum of\n durations. If there is overlap, time will be double-counted, so beware!\n \"\"\"\n total = datetime.timedelta()\n for interval in self.intervals:\n total += interval.duration()\n return total"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_overlaps(self) -> \"IntervalList\":\n overlaps = IntervalList()\n for i in range(len(self.intervals)):\n for j in range(i + 1, len(self.intervals)):\n first = self.intervals[i]\n second = self.intervals[j]\n ol = first.intersection(second)\n if ol is not None:\n overlaps.add(ol)\n return overlaps", "response": "Returns an intervalList containing intervals representing periods of overlap between intervals in this one."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef durations(self) -> List[datetime.timedelta]:\n return [x.duration() for x in self.intervals]", "response": "Returns a list of datetime. timedelta objects representing the durations of each interval in our list."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the duration of the longest interval.", "response": "def longest_duration(self) -> Optional[datetime.timedelta]:\n \"\"\"\n Returns the duration of the longest interval, or None if none.\n \"\"\"\n if not self.intervals:\n return None\n return max(self.durations())"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the longest interval in the archive or None if none.", "response": "def longest_interval(self) -> Optional[Interval]:\n \"\"\"\n Returns the longest interval, or ``None`` if none.\n \"\"\"\n longest_duration = self.longest_duration()\n for i in self.intervals:\n if i.duration() == longest_duration:\n return i\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the first interval that starts with the start parameter or None if no interval is found.", "response": "def first_interval_starting(self, start: datetime.datetime) -> \\\n Optional[Interval]:\n \"\"\"\n Returns our first interval that starts with the ``start`` parameter, or\n ``None``.\n \"\"\"\n for i in self.intervals:\n if i.start == start:\n return i\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the first interval that ends with the end parameter or None if no interval is found.", "response": "def first_interval_ending(self, end: datetime.datetime) \\\n -> Optional[Interval]:\n \"\"\"\n Returns our first interval that ends with the ``end`` parameter, or\n ``None``.\n \"\"\"\n for i in self.intervals:\n if i.end == end:\n return i\n return None"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning all the gaps between intervals as an : class:`IntervalList.", "response": "def gaps(self) -> \"IntervalList\":\n \"\"\"\n Returns all the gaps between intervals, as an :class:`IntervalList`.\n \"\"\"\n if len(self.intervals) < 2:\n return IntervalList(None)\n gaps = []\n for i in range(len(self.intervals) - 1):\n gap = Interval(\n self.intervals[i].end,\n self.intervals[i + 1].start\n )\n gaps.append(gap)\n return IntervalList(gaps)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef subset(self, interval: Interval,\n flexibility: int = 2) -> \"IntervalList\":\n \"\"\"\n Returns an IntervalList that's a subset of this one, only containing\n intervals that meet the \"interval\" parameter criterion. What \"meet\"\n means is defined by the ``flexibility`` parameter.\n\n ``flexibility == 0``: permits only wholly contained intervals:\n\n .. code-block:: none\n\n interval:\n I----------------I\n\n intervals in self that will/won't be returned:\n\n N---N N---N Y---Y N---N N---N\n N---N N---N\n\n ``flexibility == 1``: permits overlapping intervals as well:\n\n .. code-block:: none\n\n I----------------I\n\n N---N Y---Y Y---Y Y---Y N---N\n N---N N---N\n\n ``flexibility == 2``: permits adjoining intervals as well:\n\n .. code-block:: none\n\n I----------------I\n\n N---N Y---Y Y---Y Y---Y N---N\n Y---Y Y---Y\n \"\"\"\n if flexibility not in [0, 1, 2]:\n raise ValueError(\"subset: bad flexibility value\")\n permitted = []\n for i in self.intervals:\n if flexibility == 0:\n ok = i.start > interval.start and i.end < interval.end\n elif flexibility == 1:\n ok = i.end > interval.start and i.start < interval.end\n else:\n ok = i.end >= interval.start and i.start <= interval.end\n if ok:\n permitted.append(i)\n return IntervalList(permitted)", "response": "Returns an IntervalList that s a subset of this one with only containing intervals that meet the interval criterion."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef gap_subset(self, interval: Interval,\n flexibility: int = 2) -> \"IntervalList\":\n \"\"\"\n Returns an IntervalList that's a subset of this one, only containing\n *gaps* between intervals that meet the interval criterion.\n\n See :meth:`subset` for the meaning of parameters.\n \"\"\"\n return self.gaps().subset(interval, flexibility)", "response": "Returns an IntervalList that s a subset of this one only containing gaps between intervals that meet the interval criterion."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the number of weekends that the intervals collectively touch .", "response": "def n_weekends(self) -> int:\n \"\"\"\n Returns the number of weekends that the intervals collectively touch\n (where \"touching a weekend\" means \"including time on a Saturday or a\n Sunday\").\n \"\"\"\n saturdays = set()\n for interval in self.intervals:\n saturdays.update(interval.saturdays_of_weekends())\n return len(saturdays)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef duration_outside_nwh(\n self,\n starttime: datetime.time = datetime.time(NORMAL_DAY_START_H),\n endtime: datetime.time = datetime.time(NORMAL_DAY_END_H)) \\\n -> datetime.timedelta:\n \"\"\"\n Returns the total duration outside normal working hours, i.e.\n evenings/nights, weekends (and Bank Holidays).\n \"\"\"\n total = datetime.timedelta()\n for interval in self.intervals:\n total += interval.duration_outside_nwh(starttime, endtime)\n return total", "response": "Returns the total duration of the sequence of time intervals that are outside the normal working hours."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef max_consecutive_days(self) -> Optional[Tuple[int, Interval]]:\n if len(self.intervals) == 0:\n return None\n startdate = self.start_date()\n enddate = self.end_date()\n seq = ''\n ndays = (enddate - startdate).days + 1\n for i in range(ndays):\n date = startdate + datetime.timedelta(days=i)\n wholeday = Interval.wholeday(date)\n if any([x.overlaps(wholeday) for x in self.intervals]):\n seq += '+'\n else:\n seq += ' '\n # noinspection PyTypeChecker\n longest = max(seq.split(), key=len)\n longest_len = len(longest)\n longest_idx = seq.index(longest)\n longest_interval = Interval.dayspan(\n startdate + datetime.timedelta(days=longest_idx),\n startdate + datetime.timedelta(days=longest_idx + longest_len)\n )\n return longest_len, longest_interval", "response": "Returns the length of the longest sequence of days in which all days include the longest interval."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _sufficient_gaps(self,\n startdate: datetime.date,\n enddate: datetime.date,\n requiredgaps: List[datetime.timedelta],\n flexibility: int) -> Tuple[bool, Optional[Interval]]:\n \"\"\"\n Are there sufficient gaps (specified by ``requiredgaps``) in the date\n range specified? This is a worker function for :meth:`sufficient_gaps`.\n \"\"\"\n requiredgaps = list(requiredgaps) # make a copy\n interval = Interval.dayspan(startdate, enddate, include_end=True)\n # log.debug(\">>> _sufficient_gaps\")\n gaps = self.gap_subset(interval, flexibility)\n gapdurations = gaps.durations()\n gaplist = gaps.list()\n gapdurations.sort(reverse=True) # longest gap first\n requiredgaps.sort(reverse=True) # longest gap first\n # log.debug(\"... gaps = {}\".format(gaps))\n # log.debug(\"... gapdurations = {}\".format(gapdurations))\n # log.debug(\"... requiredgaps = {}\".format(requiredgaps))\n while requiredgaps:\n # log.debug(\"... processing gap\")\n if not gapdurations:\n # log.debug(\"<<< no gaps left\")\n return False, None\n if gapdurations[0] < requiredgaps[0]:\n # log.debug(\"<<< longest gap is too short\")\n return False, self.first_interval_ending(gaplist[0].start)\n gapdurations.pop(0)\n requiredgaps.pop(0)\n gaplist.pop(0)\n # ... keeps gaplist and gapdurations mapped to each other\n # log.debug(\"<<< success\")\n return True, None", "response": "This is a worker function for the _sufficient_gaps method. It returns True if there are sufficient gaps in the date range specified?"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef sufficient_gaps(self,\n every_n_days: int,\n requiredgaps: List[datetime.timedelta],\n flexibility: int = 2) \\\n -> Tuple[bool, Optional[Interval]]:\n \"\"\"\n Are gaps present sufficiently often?\n For example:\n\n .. code-block:: python\n\n every_n_days=21\n requiredgaps=[\n datetime.timedelta(hours=62),\n datetime.timedelta(hours=48),\n ]\n\n ... means \"is there at least one 62-hour gap and one (separate) 48-hour\n gap in every possible 21-day sequence within the IntervalList?\n\n - If ``flexibility == 0``: gaps must be WHOLLY WITHIN the interval.\n\n - If ``flexibility == 1``: gaps may OVERLAP the edges of the interval.\n\n - If ``flexibility == 2``: gaps may ABUT the edges of the interval.\n\n Returns ``(True, None)`` or ``(False, first_failure_interval)``.\n \"\"\"\n if len(self.intervals) < 2:\n return False, None\n startdate = self.start_date()\n enddate = self.end_date()\n ndays = (enddate - startdate).days + 1\n if ndays <= every_n_days:\n # Our interval is too short, or just right\n return self._sufficient_gaps(startdate, enddate, requiredgaps,\n flexibility)\n for i in range(ndays - every_n_days):\n j = i + every_n_days\n a = startdate + datetime.timedelta(days=i)\n b = startdate + datetime.timedelta(days=j)\n sufficient, ffi = self._sufficient_gaps(a, b, requiredgaps,\n flexibility)\n if not sufficient:\n return False, ffi\n return True, None", "response": "Check if a set of gaps are present sufficiently often."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef cumulative_time_to(self,\n when: datetime.datetime) -> datetime.timedelta:\n \"\"\"\n Returns the cumulative time contained in our intervals up to the\n specified time point.\n \"\"\"\n assert self.no_overlap, self._ONLY_FOR_NO_INTERVAL\n cumulative = datetime.timedelta()\n for interval in self.intervals:\n if interval.start >= when:\n break\n elif interval.end <= when:\n # complete interval precedes \"when\"\n cumulative += interval.duration()\n else: # start < when < end\n cumulative += when - interval.start\n return cumulative", "response": "Returns the cumulative time contained in our intervals up to the given time point."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef cumulative_gaps_to(self,\n when: datetime.datetime) -> datetime.timedelta:\n \"\"\"\n Return the cumulative time within our gaps, up to ``when``.\n \"\"\"\n gaps = self.gaps()\n return gaps.cumulative_time_to(when)", "response": "Return the cumulative time within our gaps up to when."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the time after our last interval but before when.", "response": "def time_afterwards_preceding(\n self, when: datetime.datetime) -> Optional[datetime.timedelta]:\n \"\"\"\n Returns the time after our last interval, but before ``when``.\n If ``self`` is an empty list, returns ``None``.\n \"\"\"\n if self.is_empty():\n return None\n end_time = self.end_datetime()\n if when <= end_time:\n return datetime.timedelta()\n else:\n return when - end_time"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the cumulative time before and during the given time.", "response": "def cumulative_before_during_after(self,\n start: datetime.datetime,\n when: datetime.datetime) -> \\\n Tuple[datetime.timedelta,\n datetime.timedelta,\n datetime.timedelta]:\n \"\"\"\n For a given time, ``when``, returns the cumulative time\n \n - after ``start`` but before ``self`` begins, prior to ``when``;\n - after ``start`` and during intervals represented by ``self``, prior\n to ``when``;\n - after ``start`` and after at least one interval represented by\n ``self`` has finished, and not within any intervals represented by\n ``self``, and prior to ``when``.\n \n Args:\n start: the start time of interest (e.g. before ``self`` begins)\n when: the time of interest\n \n Returns:\n tuple: ``before, during, after``\n\n Illustration\n \n .. code-block:: none\n \n \n start: S\n self: X---X X---X X---X X---X\n \n when: W\n \n before: ----\n during: ----- ----- -----\n after: ------- ------- ----\n\n \"\"\"\n assert self.no_overlap, (\n \"Only implemented for IntervalList objects with no_overlap == True\"\n )\n no_time = datetime.timedelta()\n earliest_interval_start = self.start_datetime()\n\n # Easy special cases\n if when <= start:\n return no_time, no_time, no_time\n if self.is_empty() or when <= earliest_interval_start:\n return when - start, no_time, no_time\n\n # Now we can guarantee:\n # - \"self\" is a non-empty list\n # - start < when\n # - earliest_interval_start < when\n\n # Before\n if earliest_interval_start < start:\n before = no_time\n else:\n before = earliest_interval_start - start\n\n # During\n during = self.cumulative_time_to(when)\n\n after = (\n self.cumulative_gaps_to(when) +\n self.time_afterwards_preceding(when)\n )\n\n return before, during, after"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef repr_parameter(param: inspect.Parameter) -> str:\n return (\n \"Parameter(name={name}, annotation={annotation}, kind={kind}, \"\n \"default={default}\".format(\n name=param.name, annotation=param.annotation, kind=param.kind,\n default=param.default)\n )", "response": "Provides a repr - style representation of a function parameter."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a representation of a function s name in a namespace.", "response": "def get_namespace(fn: Callable, namespace: Optional[str]) -> str:\n \"\"\"\n Returns a representation of a function's name (perhaps within a namespace),\n like\n\n .. code-block:: none\n\n mymodule:MyClass.myclassfunc # with no namespace\n mymodule:MyClass.myclassfunc|somenamespace # with a namespace\n\n Args:\n fn: a function\n namespace: an optional namespace, which can be of any type but is\n normally a ``str``; if not ``None``, ``str(namespace)`` will be\n added to the result. See\n https://dogpilecache.readthedocs.io/en/latest/api.html#dogpile.cache.region.CacheRegion.cache_on_arguments\n \"\"\" # noqa\n # See hidden attributes with dir(fn)\n # noinspection PyUnresolvedReferences\n return \"{module}:{name}{extra}\".format(\n module=fn.__module__,\n name=fn.__qualname__, # __qualname__ includes class name, if present\n extra=\"|{}\".format(namespace) if namespace is not None else \"\",\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef kw_fkg_allowing_type_hints(\n namespace: Optional[str],\n fn: Callable,\n to_str: Callable[[Any], str] = repr) -> Callable[[Any], str]:\n \"\"\"\n As for :func:`fkg_allowing_type_hints`, but allowing keyword arguments.\n\n For ``kwargs`` passed in, we will build a ``dict`` of all argname (key) to\n argvalue (values) pairs, including default args from the argspec, and then\n alphabetize the list before generating the key.\n\n NOTE ALSO that once we have keyword arguments, we should be using\n :func:`repr`, because we need to distinguish\n\n .. code-block:: python\n\n kwargs = {'p': 'another', 'q': 'thing'}\n # ... which compat.string_type will make into\n # p=another q=thing\n # ... from\n kwargs = {'p': 'another q=thing'}\n\n Also modified to make the cached function unique per INSTANCE for normal\n methods of a class.\n \"\"\"\n\n namespace = get_namespace(fn, namespace)\n\n sig = inspect.signature(fn)\n parameters = list(sig.parameters.values()) # convert from odict_values\n argnames = [p.name for p in parameters\n if p.kind == inspect.Parameter.POSITIONAL_OR_KEYWORD]\n has_self = bool(argnames and argnames[0] in ('self', 'cls'))\n\n if DEBUG_INTERNALS:\n log.debug(\n \"At start of kw_fkg_allowing_type_hints: namespace={namespace},\"\n \"parameters=[{parameters}], argnames={argnames}, \"\n \"has_self={has_self}, fn={fn}\",\n namespace=namespace,\n parameters=\", \".join(repr_parameter(p) for p in parameters),\n argnames=repr(argnames),\n has_self=has_self,\n fn=repr(fn),\n )\n\n def generate_key(*args: Any, **kwargs: Any) -> str:\n as_kwargs = {} # type: Dict[str, Any]\n loose_args = [] # type: List[Any] # those captured by *args\n # 1. args: get the name as well.\n for idx, arg in enumerate(args):\n if idx >= len(argnames):\n # positional argument to be scooped up with *args\n loose_args.append(arg)\n else:\n # normal plain positional argument\n if has_self and idx == 0: # \"self\" or \"cls\" initial argument\n argvalue = hex(id(arg))\n else:\n argvalue = arg\n as_kwargs[argnames[idx]] = argvalue\n # 1b. args with no name\n if loose_args:\n as_kwargs['*args'] = loose_args\n # '*args' is guaranteed not to be a parameter name in its own right\n # 2. kwargs\n as_kwargs.update(kwargs)\n # 3. default values\n for param in parameters:\n if param.default != inspect.Parameter.empty:\n if param.name not in as_kwargs:\n as_kwargs[param.name] = param.default\n # 4. sorted by name\n # ... but also incorporating the name of the argument, because once\n # we allow the arbitrary **kwargs format, order is no longer\n # sufficient to discriminate\n # fn(p=\"another\", q=\"thing\")\n # from\n # fn(r=\"another\", s=\"thing\")\n argument_values = [\"{k}={v}\".format(k=key, v=to_str(as_kwargs[key]))\n for key in sorted(as_kwargs.keys())]\n key = namespace + '|' + \" \".join(argument_values)\n if DEBUG_INTERNALS:\n log.debug(\"kw_fkg_allowing_type_hints.generate_key() -> {!r}\", key)\n return key\n\n return generate_key", "response": "A function that allows keyword arguments to be used in a cached function."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the GPG version", "response": "def gpg_version():\n \"\"\"Returns the GPG version\"\"\"\n cmd = flatten([gnupg_bin(), \"--version\"])\n output = stderr_output(cmd)\n output = output \\\n .split('\\n')[0] \\\n .split(\" \")[2] \\\n .split('.')\n return tuple([int(x) for x in output])"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef polite_string(a_string):\n if is_py3() and hasattr(a_string, 'decode'):\n try:\n return a_string.decode('utf-8')\n except UnicodeDecodeError:\n return a_string\n\n return a_string", "response": "Returns a string that should work in both Py3 and Py2"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nflatten iterables This is super ugly. There must be a cleaner py2 / 3 way to handle this.", "response": "def actually_flatten(iterable):\n \"\"\"Flatten iterables\n This is super ugly. There must be a cleaner py2/3 way\n of handling this.\"\"\"\n remainder = iter(iterable)\n while True:\n first = next(remainder) # pylint: disable=R1708\n # Python 2/3 compat\n is_iter = isinstance(first, collections.Iterable)\n try:\n basestring\n except NameError:\n basestring = str # pylint: disable=W0622\n\n if is_py3() and is_iter and not_a_string(first):\n remainder = IT.chain(first, remainder)\n elif (not is_py3()) and is_iter and not isinstance(first, basestring):\n remainder = IT.chain(first, remainder)\n else:\n yield polite_string(first)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef passphrase_file(passphrase=None):\n cmd = []\n pass_file = None\n if not passphrase and 'CRYPTORITO_PASSPHRASE_FILE' in os.environ:\n pass_file = os.environ['CRYPTORITO_PASSPHRASE_FILE']\n if not os.path.isfile(pass_file):\n raise CryptoritoError('CRYPTORITO_PASSPHRASE_FILE is invalid')\n elif passphrase:\n tmpdir = ensure_tmpdir()\n pass_file = \"%s/p_pass\" % tmpdir\n p_handle = open(pass_file, 'w')\n p_handle.write(passphrase)\n p_handle.close()\n\n if pass_file:\n cmd = cmd + [\"--batch\", \"--passphrase-file\", pass_file]\n\n vsn = gpg_version()\n if vsn[0] >= 2 and vsn[1] >= 1:\n cmd = cmd + [\"--pinentry-mode\", \"loopback\"]\n\n return cmd", "response": "Read passphrase from a file."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning appropriate arguments for GNUPGHOME.", "response": "def gnupg_home():\n \"\"\"Returns appropriate arguments if GNUPGHOME is set\"\"\"\n if 'GNUPGHOME' in os.environ:\n gnupghome = os.environ['GNUPGHOME']\n if not os.path.isdir(gnupghome):\n raise CryptoritoError(\"Invalid GNUPGHOME directory\")\n\n return [\"--homedir\", gnupghome]\n else:\n return []"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef fingerprint_from_keybase(fingerprint, kb_obj):\n if 'public_keys' in kb_obj and \\\n 'pgp_public_keys' in kb_obj['public_keys']:\n for key in kb_obj['public_keys']['pgp_public_keys']:\n keyprint = fingerprint_from_var(key).lower()\n fingerprint = fingerprint.lower()\n if fingerprint == keyprint or \\\n keyprint.startswith(fingerprint) or \\\n keyprint.endswith(fingerprint):\n return {\n 'fingerprint': keyprint,\n 'bundle': key\n }\n\n return None", "response": "Extracts a key matching a specific fingerprint from a Keybase API response"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nlooks up a public key from a username", "response": "def key_from_keybase(username, fingerprint=None):\n \"\"\"Look up a public key from a username\"\"\"\n url = keybase_lookup_url(username)\n resp = requests.get(url)\n if resp.status_code == 200:\n j_resp = json.loads(polite_string(resp.content))\n if 'them' in j_resp and len(j_resp['them']) == 1:\n kb_obj = j_resp['them'][0]\n if fingerprint:\n return fingerprint_from_keybase(fingerprint, kb_obj)\n else:\n if 'public_keys' in kb_obj \\\n and 'pgp_public_keys' in kb_obj['public_keys']:\n key = kb_obj['public_keys']['primary']\n return massage_key(key)\n\n return None"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef has_gpg_key(fingerprint):\n if len(fingerprint) > 8:\n fingerprint = fingerprint[-8:]\n\n fingerprint = fingerprint.upper()\n cmd = flatten([gnupg_bin(), gnupg_home(), \"--list-public-keys\"])\n lines = stderr_output(cmd).split('\\n')\n return len([key for key in lines if key.find(fingerprint) > -1]) == 1", "response": "Checks to see if we have this gpg key"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fingerprint_from_var(var):\n vsn = gpg_version()\n cmd = flatten([gnupg_bin(), gnupg_home()])\n if vsn[0] >= 2 and vsn[1] < 1:\n cmd.append(\"--with-fingerprint\")\n\n output = polite_string(stderr_with_input(cmd, var)).split('\\n')\n if not output[0].startswith('pub'):\n raise CryptoritoError('probably an invalid gpg key')\n\n if vsn[0] >= 2 and vsn[1] < 1:\n return output[1] \\\n .split('=')[1] \\\n .replace(' ', '')\n\n return output[1].strip()", "response": "Extract a fingerprint from a GPG public key"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef fingerprint_from_file(filename):\n cmd = flatten([gnupg_bin(), gnupg_home(), filename])\n outp = stderr_output(cmd).split('\\n')\n if not outp[0].startswith('pub'):\n raise CryptoritoError('probably an invalid gpg key')\n\n return outp[1].strip()", "response": "Extract a fingerprint from a GPG public key file"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ngenerate debug - level appropriate stderr context for returning the stderr handle.", "response": "def stderr_handle():\n \"\"\"Generate debug-level appropriate stderr context for\n executing things through subprocess. Normally stderr gets\n sent to dev/null but when debugging it is sent to stdout.\"\"\"\n gpg_stderr = None\n handle = None\n if LOGGER.getEffectiveLevel() > logging.DEBUG:\n handle = open(os.devnull, 'wb')\n gpg_stderr = handle\n\n return handle, gpg_stderr"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef stderr_output(cmd):\n\n handle, gpg_stderr = stderr_handle()\n try:\n output = subprocess.check_output(cmd, stderr=gpg_stderr) # nosec\n if handle:\n handle.close()\n\n return str(polite_string(output))\n except subprocess.CalledProcessError as exception:\n LOGGER.debug(\"GPG Command %s\", ' '.join(exception.cmd))\n LOGGER.debug(\"GPG Output %s\", exception.output)\n raise CryptoritoError('GPG Execution')", "response": "Wraps the execution of check_output in a way that returns the output of the command and closes the stderr handle."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef stderr_with_input(cmd, stdin):\n handle, gpg_stderr = stderr_handle()\n LOGGER.debug(\"GPG command %s\", ' '.join(cmd))\n try:\n gpg_proc = subprocess.Popen(cmd, # nosec\n stdout=subprocess.PIPE,\n stdin=subprocess.PIPE,\n stderr=gpg_stderr)\n\n output, _err = gpg_proc.communicate(polite_bytes(stdin))\n\n if handle:\n handle.close()\n\n return output\n except subprocess.CalledProcessError as exception:\n return gpg_error(exception, 'GPG variable encryption error')\n except OSError as exception:\n raise CryptoritoError(\"File %s not found\" % exception.filename)", "response": "Runs a command passing something in stdin and returning\nTaxonomy whatever came out from stdout"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef import_gpg_key(key):\n if not key:\n raise CryptoritoError('Invalid GPG Key')\n\n key_fd, key_filename = mkstemp(\"cryptorito-gpg-import\")\n key_handle = os.fdopen(key_fd, 'w')\n\n key_handle.write(polite_string(key))\n key_handle.close()\n cmd = flatten([gnupg_bin(), gnupg_home(), \"--import\", key_filename])\n output = stderr_output(cmd)\n msg = 'gpg: Total number processed: 1'\n output_bits = polite_string(output).split('\\n')\n return len([line for line in output_bits if line == msg]) == 1", "response": "Imports a GPG key into a single file."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nexports a GPG key and returns it", "response": "def export_gpg_key(key):\n \"\"\"Exports a GPG key and returns it\"\"\"\n cmd = flatten([gnupg_bin(), gnupg_verbose(), gnupg_home(),\n \"--export\", key])\n handle, gpg_stderr = stderr_handle()\n try:\n gpg_proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, # nosec\n stderr=gpg_stderr)\n output, _err = gpg_proc.communicate()\n if handle:\n handle.close()\n\n return portable_b64encode(output)\n except subprocess.CalledProcessError as exception:\n LOGGER.debug(\"GPG Command %s\", ' '.join(exception.cmd))\n LOGGER.debug(\"GPG Output %s\", exception.output)\n raise CryptoritoError('GPG encryption error')"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nencrypts a file using the given keys", "response": "def encrypt(source, dest, keys):\n \"\"\"Encrypts a file using the given keys\"\"\"\n cmd = flatten([gnupg_bin(), \"--armor\", \"--output\", dest, gnupg_verbose(),\n gnupg_home(), recipients_args(keys),\n \"--encrypt\", source])\n\n stderr_output(cmd)\n return True"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nattempting to encrypt a variable", "response": "def encrypt_var(source, keys):\n \"\"\"Attempts to encrypt a variable\"\"\"\n cmd = flatten([gnupg_bin(), \"--armor\", \"--encrypt\", gnupg_verbose(),\n recipients_args(keys)])\n output = stderr_with_input(cmd, source)\n return output"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nhandling the output of subprocess errors", "response": "def gpg_error(exception, message):\n \"\"\"Handles the output of subprocess errors\n in a way that is compatible with the log level\"\"\"\n LOGGER.debug(\"GPG Command %s\", ' '.join([str(x) for x in exception.cmd]))\n LOGGER.debug(\"GPG Output %s\", exception.output)\n raise CryptoritoError(message)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nattempt to decrypt a variable", "response": "def decrypt_var(source, passphrase=None):\n \"\"\"Attempts to decrypt a variable\"\"\"\n cmd = [gnupg_bin(), \"--decrypt\", gnupg_home(), gnupg_verbose(),\n passphrase_file(passphrase)]\n\n return stderr_with_input(flatten(cmd), source)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef decrypt(source, dest=None, passphrase=None):\n if not os.path.exists(source):\n raise CryptoritoError(\"Encrypted file %s not found\" % source)\n\n cmd = [gnupg_bin(), gnupg_verbose(), \"--decrypt\", gnupg_home(),\n passphrase_file(passphrase)]\n\n if dest:\n cmd.append([\"--output\", dest])\n\n cmd.append([source])\n stderr_output(flatten(cmd))\n return True", "response": "Attempts to decrypt a file"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef is_base64(string):\n return (not re.match('^[0-9]+$', string)) and \\\n (len(string) % 4 == 0) and \\\n re.match('^[A-Za-z0-9+/]+[=]{0,2}$', string)", "response": "Determines whether or not a string is likely to\n be base64 encoded binary nonsense"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef portable_b64encode(thing):\n if is_py3():\n try:\n some_bits = bytes(thing, 'utf-8')\n except TypeError:\n some_bits = thing\n\n return polite_string(b64encode(some_bits).decode('utf-8'))\n\n return polite_string(b64encode(thing))", "response": "Wrap b64encode for Python 2 & 3"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef download_if_not_exists(url: str, filename: str,\n skip_cert_verify: bool = True,\n mkdir: bool = True) -> None:\n \"\"\"\n Downloads a URL to a file, unless the file already exists.\n \"\"\"\n if os.path.isfile(filename):\n log.info(\"No need to download, already have: {}\", filename)\n return\n if mkdir:\n directory, basename = os.path.split(os.path.abspath(filename))\n mkdir_p(directory)\n download(url=url,\n filename=filename,\n skip_cert_verify=skip_cert_verify)", "response": "Downloads a URL to a file unless the file already exists."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfetches a Git repository and returns a list of the new ones.", "response": "def git_clone(prettyname: str, url: str, directory: str,\n branch: str = None,\n commit: str = None,\n clone_options: List[str] = None,\n run_func: Callable[[List[str]], Any] = None) -> bool:\n \"\"\"\n Fetches a Git repository, unless we have it already.\n\n Args:\n prettyname: name to display to user\n url: URL\n directory: destination directory\n branch: repository branch\n commit: repository commit tag\n clone_options: additional options to pass to ``git clone``\n run_func: function to use to call an external command\n\n Returns:\n did we need to do anything?\n \"\"\"\n run_func = run_func or subprocess.check_call\n clone_options = clone_options or [] # type: List[str]\n if os.path.isdir(directory):\n log.info(\"Not re-cloning {} Git repository: using existing source \"\n \"in {}\".format(prettyname, directory))\n return False\n log.info(\"Fetching {} source from {} into {}\",\n prettyname, url, directory)\n require_executable(GIT)\n gitargs = [GIT, \"clone\"] + clone_options\n if branch:\n gitargs += [\"--branch\", branch]\n gitargs += [url, directory]\n run_func(gitargs)\n if commit:\n log.info(\"Resetting {} local Git repository to commit {}\",\n prettyname, commit)\n run_func([GIT,\n \"-C\", directory,\n \"reset\", \"--hard\", commit])\n # Using a Git repository that's not in the working directory:\n # https://stackoverflow.com/questions/1386291/git-git-dir-not-working-as-expected # noqa\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nunpacks a TAR file into a specified directory.", "response": "def untar_to_directory(tarfile: str,\n directory: str,\n verbose: bool = False,\n gzipped: bool = False,\n skip_if_dir_exists: bool = True,\n run_func: Callable[[List[str]], Any] = None,\n chdir_via_python: bool = True) -> None:\n \"\"\"\n Unpacks a TAR file into a specified directory.\n\n Args:\n tarfile: filename of the ``.tar`` file\n directory: destination directory\n verbose: be verbose?\n gzipped: is the ``.tar`` also gzipped, e.g. a ``.tar.gz`` file?\n skip_if_dir_exists: don't do anything if the destrination directory\n exists?\n run_func: function to use to call an external command\n chdir_via_python: change directory via Python, not via ``tar``.\n Consider using this via Windows, because Cygwin ``tar`` v1.29 falls\n over when given a Windows path for its ``-C`` (or ``--directory``)\n option.\n \"\"\"\n if skip_if_dir_exists and os.path.isdir(directory):\n log.info(\"Skipping extraction of {} as directory {} exists\",\n tarfile, directory)\n return\n log.info(\"Extracting {} -> {}\", tarfile, directory)\n require_executable(TAR)\n mkdir_p(directory)\n args = [TAR, \"-x\"] # -x: extract\n if verbose:\n args.append(\"-v\") # -v: verbose\n if gzipped:\n args.append(\"-z\") # -z: decompress using gzip\n if platform.system() != \"Darwin\": # OS/X tar doesn't support --force-local\n args.append(\"--force-local\") # allows filenames with colons in (Windows!) # noqa\n args.extend([\"-f\", tarfile]) # -f: filename follows\n if chdir_via_python:\n with pushd(directory):\n run_func(args)\n else:\n # chdir via tar\n args.extend([\"-C\", directory]) # -C: change to directory\n run_func(args)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef make_copy_paste_env(env: Dict[str, str]) -> str:\n windows = platform.system() == \"Windows\"\n cmd = \"set\" if windows else \"export\"\n return (\n \"\\n\".join(\n \"{cmd} {k}={v}\".format(\n cmd=cmd,\n k=k,\n v=env[k] if windows else subprocess.list2cmdline([env[k]])\n ) for k in sorted(env.keys())\n )\n )", "response": "Convert an environment into a set of commands that can be copied and pasted on the build platform to recreate that environment."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef run(args: List[str],\n env: Dict[str, str] = None,\n capture_stdout: bool = False,\n echo_stdout: bool = True,\n capture_stderr: bool = False,\n echo_stderr: bool = True,\n debug_show_env: bool = True,\n encoding: str = sys.getdefaultencoding(),\n allow_failure: bool = False,\n **kwargs) -> Tuple[str, str]:\n \"\"\"\n Runs an external process, announcing it.\n\n Optionally, retrieves its ``stdout`` and/or ``stderr`` output (if not\n retrieved, the output will be visible to the user).\n\n Args:\n args: list of command-line arguments (the first being the executable)\n\n env: operating system environment to use (if ``None``, the current OS\n environment will be used)\n\n capture_stdout: capture the command's ``stdout``?\n\n echo_stdout: allow the command's ``stdout`` to go to ``sys.stdout``?\n\n capture_stderr: capture the command's ``stderr``?\n\n echo_stderr: allow the command's ``stderr`` to go to ``sys.stderr``?\n\n debug_show_env: be verbose and show the environment used before calling\n\n encoding: encoding to use to translate the command's output\n\n allow_failure: if ``True``, continues if the command returns a\n non-zero (failure) exit code; if ``False``, raises an error if\n that happens\n\n kwargs: additional arguments to :func:`teed_call`\n\n Returns:\n a tuple: ``(stdout, stderr)``. If the output wasn't captured, an empty\n string will take its place in this tuple.\n \"\"\"\n cwd = os.getcwd()\n # log.debug(\"External command Python form: {}\", args)\n copy_paste_cmd = subprocess.list2cmdline(args)\n csep = \"=\" * 79\n esep = \"-\" * 79\n effective_env = env or os.environ\n if debug_show_env:\n log.debug(\n \"Environment for the command that follows:\\n\"\n \"{esep}\\n\"\n \"{env}\\n\"\n \"{esep}\".format(esep=esep, env=make_copy_paste_env(effective_env))\n )\n log.info(\n \"Launching external command:\\n\"\n \"{csep}\\n\"\n \"WORKING DIRECTORY: {cwd}\\n\"\n \"PYTHON ARGS: {args!r}\\n\"\n \"COMMAND: {cmd}\\n\"\n \"{csep}\".format(csep=csep, cwd=cwd, cmd=copy_paste_cmd,\n args=args)\n )\n try:\n with io.StringIO() as out, io.StringIO() as err:\n stdout_targets = [] # type: List[TextIO]\n stderr_targets = [] # type: List[TextIO]\n if capture_stdout:\n stdout_targets.append(out)\n if echo_stdout:\n stdout_targets.append(sys.stdout)\n if capture_stderr:\n stderr_targets.append(err)\n if echo_stderr:\n stderr_targets.append(sys.stderr)\n retcode = teed_call(args,\n stdout_targets=stdout_targets,\n stderr_targets=stderr_targets,\n encoding=encoding,\n env=env,\n **kwargs)\n stdout = out.getvalue()\n stderr = err.getvalue()\n if retcode != 0 and not allow_failure:\n # subprocess.check_call() and check_output() raise\n # CalledProcessError if the called process returns a non-zero\n # return code.\n raise subprocess.CalledProcessError(returncode=retcode,\n cmd=args,\n output=stdout,\n stderr=stderr)\n log.debug(\"\\n{csep}\\nFINISHED SUCCESSFULLY: {cmd}\\n{csep}\",\n cmd=copy_paste_cmd, csep=csep)\n return stdout, stderr\n except FileNotFoundError:\n require_executable(args[0]) # which is missing, so we'll see some help\n raise\n except subprocess.CalledProcessError:\n log.critical(\n \"Command that failed:\\n\"\n \"[ENVIRONMENT]\\n\"\n \"{env}\\n\"\n \"\\n\"\n \"[DIRECTORY] {cwd}\\n\"\n \"[PYTHON ARGS] {args}\\n\"\n \"[COMMAND] {cmd}\".format(\n cwd=cwd,\n env=make_copy_paste_env(effective_env),\n cmd=copy_paste_cmd,\n args=args\n )\n )\n raise", "response": "Runs an external process and returns its output."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef fetch(args: List[str], env: Dict[str, str] = None,\n encoding: str = sys.getdefaultencoding()) -> str:\n \"\"\"\n Run a command and returns its stdout.\n\n Args:\n args: the command-line arguments\n env: the operating system environment to use\n encoding: the encoding to use for ``stdout``\n\n Returns:\n the command's ``stdout`` output\n\n \"\"\"\n stdout, _ = run(args, env=env, capture_stdout=True,\n echo_stdout=False, encoding=encoding)\n log.debug(stdout)\n return stdout", "response": "Run a command and returns its stdout."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef dump_connection_info(engine: Engine, fileobj: TextIO = sys.stdout) -> None:\n meta = MetaData(bind=engine)\n writeline_nl(fileobj, sql_comment('Database info: {}'.format(meta)))", "response": "Dumps some connection info as an SQL comment. Obscures passwords."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsends schema-creating DDL from the metadata to the dump engine. This makes ``CREATE TABLE`` statements. Args: metadata: SQLAlchemy :class:`MetaData` dialect_name: string name of SQL dialect to generate DDL in fileobj: file-like object to send DDL to checkfirst: if ``True``, use ``CREATE TABLE IF NOT EXISTS`` or equivalent.", "response": "def dump_ddl(metadata: MetaData,\n dialect_name: str,\n fileobj: TextIO = sys.stdout,\n checkfirst: bool = True) -> None:\n \"\"\"\n Sends schema-creating DDL from the metadata to the dump engine.\n This makes ``CREATE TABLE`` statements.\n\n Args:\n metadata: SQLAlchemy :class:`MetaData`\n dialect_name: string name of SQL dialect to generate DDL in\n fileobj: file-like object to send DDL to\n checkfirst: if ``True``, use ``CREATE TABLE IF NOT EXISTS`` or\n equivalent.\n \"\"\"\n # http://docs.sqlalchemy.org/en/rel_0_8/faq.html#how-can-i-get-the-create-table-drop-table-output-as-a-string # noqa\n # http://stackoverflow.com/questions/870925/how-to-generate-a-file-with-ddl-in-the-engines-sql-dialect-in-sqlalchemy # noqa\n # https://github.com/plq/scripts/blob/master/pg_dump.py\n # noinspection PyUnusedLocal\n def dump(querysql, *multiparams, **params):\n compsql = querysql.compile(dialect=engine.dialect)\n writeline_nl(fileobj, \"{sql};\".format(sql=compsql))\n\n writeline_nl(fileobj,\n sql_comment(\"Schema (for dialect {}):\".format(dialect_name)))\n engine = create_engine('{dialect}://'.format(dialect=dialect_name),\n strategy='mock', executor=dump)\n metadata.create_all(engine, checkfirst=checkfirst)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a new SQLAlchemy mapper for an existing table.", "response": "def quick_mapper(table: Table) -> Type[DeclarativeMeta]:\n \"\"\"\n Makes a new SQLAlchemy mapper for an existing table.\n See\n http://www.tylerlesmann.com/2009/apr/27/copying-databases-across-platforms-sqlalchemy/\n \n Args:\n table: SQLAlchemy :class:`Table` object\n\n Returns:\n a :class:`DeclarativeMeta` class\n\n \"\"\" # noqa\n # noinspection PyPep8Naming\n Base = declarative_base()\n\n class GenericMapper(Base):\n __table__ = table\n\n # noinspection PyTypeChecker\n return GenericMapper"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_literal_query(statement: Union[Query, Executable],\n bind: Connectable = None) -> str:\n \"\"\"\n Takes an SQLAlchemy statement and produces a literal SQL version, with\n values filled in.\n \n As per\n http://stackoverflow.com/questions/5631078/sqlalchemy-print-the-actual-query\n \n Notes:\n - for debugging purposes *only*\n - insecure; you should always separate queries from their values\n - please also note that this function is quite slow \n \n Args:\n statement: the SQL statement (a SQLAlchemy object) to use\n bind: if the statement is unbound, you will need to specify an object\n here that supports SQL execution \n\n Returns:\n a string literal version of the query.\n\n \"\"\" # noqa\n # log.debug(\"statement: {!r}\", statement)\n # log.debug(\"statement.bind: {!r}\", statement.bind)\n if isinstance(statement, Query):\n if bind is None:\n bind = statement.session.get_bind(statement._mapper_zero_or_none())\n statement = statement.statement\n elif bind is None:\n bind = statement.bind\n if bind is None: # despite all that\n raise ValueError(\"Attempt to call get_literal_query with an unbound \"\n \"statement and no 'bind' parameter\")\n\n # noinspection PyUnresolvedReferences\n dialect = bind.dialect\n compiler = statement._compiler(dialect)\n\n class LiteralCompiler(compiler.__class__):\n # noinspection PyMethodMayBeStatic\n def visit_bindparam(self,\n bindparam: BindParameter,\n within_columns_clause: bool = False,\n literal_binds: bool = False,\n **kwargs) -> str:\n return super().render_literal_bindparam(\n bindparam,\n within_columns_clause=within_columns_clause,\n literal_binds=literal_binds,\n **kwargs\n )\n\n # noinspection PyUnusedLocal\n def render_literal_value(self, value: Any, type_) -> str:\n \"\"\"Render the value of a bind parameter as a quoted literal.\n\n This is used for statement sections that do not accept bind\n paramters on the target driver/database.\n\n This should be implemented by subclasses using the quoting services\n of the DBAPI.\n \"\"\"\n if isinstance(value, str):\n value = value.replace(\"'\", \"''\")\n return \"'%s'\" % value\n elif value is None:\n return \"NULL\"\n elif isinstance(value, (float, int)):\n return repr(value)\n elif isinstance(value, decimal.Decimal):\n return str(value)\n elif (isinstance(value, datetime.datetime) or\n isinstance(value, datetime.date) or\n isinstance(value, datetime.time) or\n isinstance(value, pendulum.DateTime) or\n isinstance(value, pendulum.Date) or\n isinstance(value, pendulum.Time)):\n # All have an isoformat() method.\n return \"'{}'\".format(value.isoformat())\n # return (\n # \"TO_DATE('%s','YYYY-MM-DD HH24:MI:SS')\"\n # % value.strftime(\"%Y-%m-%d %H:%M:%S\")\n # )\n else:\n raise NotImplementedError(\n \"Don't know how to literal-quote value %r\" % value)\n\n compiler = LiteralCompiler(dialect, statement)\n return compiler.process(statement) + \";\"", "response": "Returns a literal SQL version of the given SQLAlchemy statement."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndumps the data from the database to the file - like object fileobj.", "response": "def dump_table_as_insert_sql(engine: Engine,\n table_name: str,\n fileobj: TextIO,\n wheredict: Dict[str, Any] = None,\n include_ddl: bool = False,\n multirow: bool = False) -> None:\n \"\"\"\n Reads a table from the database, and writes SQL to replicate the table's\n data to the output ``fileobj``.\n\n Args:\n engine: SQLAlchemy :class:`Engine`\n table_name: name of the table\n fileobj: file-like object to write to\n wheredict: optional dictionary of ``{column_name: value}`` to use as\n ``WHERE`` filters\n include_ddl: if ``True``, include the DDL to create the table as well\n multirow: write multi-row ``INSERT`` statements\n \"\"\"\n # http://stackoverflow.com/questions/5631078/sqlalchemy-print-the-actual-query # noqa\n # http://docs.sqlalchemy.org/en/latest/faq/sqlexpressions.html\n # http://www.tylerlesmann.com/2009/apr/27/copying-databases-across-platforms-sqlalchemy/ # noqa\n # https://github.com/plq/scripts/blob/master/pg_dump.py\n log.info(\"dump_data_as_insert_sql: table_name={}\", table_name)\n writelines_nl(fileobj, [\n SEP1,\n sql_comment(\"Data for table: {}\".format(table_name)),\n SEP2,\n sql_comment(\"Filters: {}\".format(wheredict)),\n ])\n dialect = engine.dialect\n if not dialect.supports_multivalues_insert:\n multirow = False\n if multirow:\n log.warning(\"dump_data_as_insert_sql: multirow parameter substitution \"\n \"not working yet\")\n multirow = False\n\n # literal_query = make_literal_query_fn(dialect)\n\n meta = MetaData(bind=engine)\n log.debug(\"... retrieving schema\")\n table = Table(table_name, meta, autoload=True)\n if include_ddl:\n log.debug(\"... producing DDL\")\n dump_ddl(table.metadata, dialect_name=engine.dialect.name,\n fileobj=fileobj)\n # NewRecord = quick_mapper(table)\n # columns = table.columns.keys()\n log.debug(\"... fetching records\")\n # log.debug(\"meta: {}\", meta) # obscures password\n # log.debug(\"table: {}\", table)\n # log.debug(\"table.columns: {!r}\", table.columns)\n # log.debug(\"multirow: {}\", multirow)\n query = select(table.columns)\n if wheredict:\n for k, v in wheredict.items():\n col = table.columns.get(k)\n query = query.where(col == v)\n # log.debug(\"query: {}\", query)\n cursor = engine.execute(query)\n if multirow:\n row_dict_list = []\n for r in cursor:\n row_dict_list.append(dict(r))\n # log.debug(\"row_dict_list: {}\", row_dict_list)\n if row_dict_list:\n statement = table.insert().values(row_dict_list)\n # log.debug(\"statement: {!r}\", statement)\n # insert_str = literal_query(statement)\n insert_str = get_literal_query(statement, bind=engine)\n # NOT WORKING FOR MULTIROW INSERTS. ONLY SUBSTITUTES FIRST ROW.\n writeline_nl(fileobj, insert_str)\n else:\n writeline_nl(fileobj, sql_comment(\"No data!\"))\n else:\n found_one = False\n for r in cursor:\n found_one = True\n row_dict = dict(r)\n statement = table.insert(values=row_dict)\n # insert_str = literal_query(statement)\n insert_str = get_literal_query(statement, bind=engine)\n # log.debug(\"row_dict: {}\", row_dict)\n # log.debug(\"insert_str: {}\", insert_str)\n writeline_nl(fileobj, insert_str)\n if not found_one:\n writeline_nl(fileobj, sql_comment(\"No data!\"))\n writeline_nl(fileobj, SEP2)\n log.debug(\"... done\")"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nwrites SQL to replicate the entire database to the output file - like object.", "response": "def dump_database_as_insert_sql(engine: Engine,\n fileobj: TextIO = sys.stdout,\n include_ddl: bool = False,\n multirow: bool = False) -> None:\n \"\"\"\n Reads an entire database and writes SQL to replicate it to the output\n file-like object.\n\n Args:\n engine: SQLAlchemy :class:`Engine`\n fileobj: file-like object to write to\n include_ddl: if ``True``, include the DDL to create the table as well\n multirow: write multi-row ``INSERT`` statements\n \"\"\"\n for tablename in get_table_names(engine):\n dump_table_as_insert_sql(\n engine=engine,\n table_name=tablename,\n fileobj=fileobj,\n include_ddl=include_ddl,\n multirow=multirow\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef dump_orm_object_as_insert_sql(engine: Engine,\n obj: object,\n fileobj: TextIO) -> None:\n \"\"\"\n Takes a SQLAlchemy ORM object, and writes ``INSERT`` SQL to replicate it\n to the output file-like object.\n\n Args:\n engine: SQLAlchemy :class:`Engine`\n obj: SQLAlchemy ORM object to write\n fileobj: file-like object to write to\n \"\"\"\n # literal_query = make_literal_query_fn(engine.dialect)\n insp = inspect(obj)\n # insp: an InstanceState\n # http://docs.sqlalchemy.org/en/latest/orm/internals.html#sqlalchemy.orm.state.InstanceState # noqa\n # insp.mapper: a Mapper\n # http://docs.sqlalchemy.org/en/latest/orm/mapping_api.html#sqlalchemy.orm.mapper.Mapper # noqa\n\n # Don't do this:\n # table = insp.mapper.mapped_table\n # Do this instead. The method above gives you fancy data types like list\n # and Arrow on the Python side. We want the bog-standard datatypes drawn\n # from the database itself.\n meta = MetaData(bind=engine)\n table_name = insp.mapper.mapped_table.name\n # log.debug(\"table_name: {}\", table_name)\n table = Table(table_name, meta, autoload=True)\n # log.debug(\"table: {}\", table)\n\n # NewRecord = quick_mapper(table)\n # columns = table.columns.keys()\n query = select(table.columns)\n # log.debug(\"query: {}\", query)\n for orm_pkcol in insp.mapper.primary_key:\n core_pkcol = table.columns.get(orm_pkcol.name)\n pkval = getattr(obj, orm_pkcol.name)\n query = query.where(core_pkcol == pkval)\n # log.debug(\"query: {}\", query)\n cursor = engine.execute(query)\n row = cursor.fetchone() # should only be one...\n row_dict = dict(row)\n # log.debug(\"obj: {}\", obj)\n # log.debug(\"row_dict: {}\", row_dict)\n statement = table.insert(values=row_dict)\n # insert_str = literal_query(statement)\n insert_str = get_literal_query(statement, bind=engine)\n writeline_nl(fileobj, insert_str)", "response": "Dump an object as an INSERT SQL statement."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef bulk_insert_extras(dialect_name: str,\n fileobj: TextIO,\n start: bool) -> None:\n \"\"\"\n Writes bulk ``INSERT`` preamble (start=True) or end (start=False).\n\n For MySQL, this temporarily switches off autocommit behaviour and index/FK\n checks, for speed, then re-enables them at the end and commits.\n\n Args:\n dialect_name: SQLAlchemy dialect name (see :class:`SqlaDialectName`)\n fileobj: file-like object to write to\n start: if ``True``, write preamble; if ``False``, write end\n \"\"\"\n lines = []\n if dialect_name == SqlaDialectName.MYSQL:\n if start:\n lines = [\n \"SET autocommit=0;\",\n \"SET unique_checks=0;\",\n \"SET foreign_key_checks=0;\",\n ]\n else:\n lines = [\n \"SET foreign_key_checks=1;\",\n \"SET unique_checks=1;\",\n \"COMMIT;\",\n ]\n writelines_nl(fileobj, lines)", "response": "Writes bulk INSERT preamble or end."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndumps an object and all related objects in a tree to a file - like object.", "response": "def dump_orm_tree_as_insert_sql(engine: Engine,\n baseobj: object,\n fileobj: TextIO) -> None:\n \"\"\"\n Sends an object, and all its relations (discovered via \"relationship\"\n links) as ``INSERT`` commands in SQL, to ``fileobj``.\n\n Args:\n engine: SQLAlchemy :class:`Engine`\n baseobj: starting SQLAlchemy ORM object\n fileobj: file-like object to write to\n\n Problem: foreign key constraints.\n \n - MySQL/InnoDB doesn't wait to the end of a transaction to check FK\n integrity (which it should):\n http://stackoverflow.com/questions/5014700/in-mysql-can-i-defer-referential-integrity-checks-until-commit # noqa\n - PostgreSQL can.\n - Anyway, slightly ugly hacks...\n https://dev.mysql.com/doc/refman/5.5/en/optimizing-innodb-bulk-data-loading.html\n - Not so obvious how we can iterate through the list of ORM objects and\n guarantee correct insertion order with respect to all FKs.\n \"\"\" # noqa\n writeline_nl(\n fileobj,\n sql_comment(\"Data for all objects related to the first below:\"))\n bulk_insert_extras(engine.dialect.name, fileobj, start=True)\n for part in walk_orm_tree(baseobj):\n dump_orm_object_as_insert_sql(engine, part, fileobj)\n bulk_insert_extras(engine.dialect.name, fileobj, start=False)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfetching next page from scroll API.", "response": "def next(self):\n \"\"\"Fetch next page from scroll API.\"\"\"\n d = None\n if self._first_results:\n d = succeed(EsUtils.extract_hits(self._first_results))\n self._first_results = None\n elif self._scroll_id:\n d = self._scroll_next_results()\n else:\n raise StopIteration()\n return d"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef reformat_python_docstrings(top_dirs: List[str],\n correct_copyright_lines: List[str],\n show_only: bool = True,\n rewrite: bool = False,\n process_only_filenum: int = None) -> None:\n \"\"\"\n Walk a directory, finding Python files and rewriting them.\n\n Args:\n top_dirs: list of directories to descend into\n correct_copyright_lines:\n list of lines (without newlines) representing the copyright\n docstring block, including the transition lines of equals\n symbols\n show_only: show results (to stdout) only; don't rewrite\n rewrite: write the changes\n process_only_filenum: only process this file number (1-based index);\n for debugging only\n \"\"\"\n filenum = 0\n for top_dir in top_dirs:\n for dirpath, dirnames, filenames in walk(top_dir):\n for filename in filenames:\n fullname = join(dirpath, filename)\n extension = splitext(filename)[1]\n if extension != PYTHON_EXTENSION:\n # log.debug(\"Skipping non-Python file: {}\", fullname)\n continue\n\n filenum += 1\n\n if process_only_filenum and filenum != process_only_filenum:\n continue\n\n log.info(\"Processing file {}: {}\", filenum, fullname)\n proc = PythonProcessor(\n full_path=fullname,\n top_dir=top_dir,\n correct_copyright_lines=correct_copyright_lines)\n if show_only:\n proc.show()\n elif rewrite:\n proc.rewrite_file()", "response": "Walk a directory and reformat the copyright - related Python docstrings."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreading the source file and adds the lines to the source_lines list.", "response": "def _read_source(self) -> None:\n \"\"\"\n Reads the source file.\n \"\"\"\n with open(self.full_path, \"rt\") as f:\n for linenum, line_with_nl in enumerate(f.readlines(), start=1):\n line_without_newline = (\n line_with_nl[:-1] if line_with_nl.endswith(NL)\n else line_with_nl\n )\n if TAB in line_without_newline:\n self._warn(\"Tab character at line {}\".format(linenum))\n if CR in line_without_newline:\n self._warn(\"Carriage return character at line {} \"\n \"(Windows CR+LF endings?)\".format(linenum))\n self.source_lines.append(line_without_newline)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates an internal representation of the destination file.", "response": "def _create_dest(self) -> None:\n \"\"\"\n Creates an internal representation of the destination file.\n\n This is where the thinking happens\n \"\"\"\n in_body = False\n in_docstring = False\n in_copyright = False\n copyright_done = False\n docstring_done = False\n swallow_blanks_and_filename_in_docstring = False\n for linenum, sl in enumerate(self.source_lines, start=1):\n dl = sl\n\n if dl.endswith(SPACE):\n self._debug(\"Line {} ends in whitespace\".format(linenum))\n dl = dl.rstrip()\n\n if not in_body:\n\n if linenum == 1:\n # Shebang\n if not dl.startswith(SHEBANG_START):\n self._warn(\"File does not start with shebang; \"\n \"first line was {!r}\".format(dl))\n self._too_risky()\n return\n if dl != CORRECT_SHEBANG:\n self._debug(\"Rewriting shebang; was {!r}\".format(dl))\n dl = CORRECT_SHEBANG\n\n if (linenum == 2 and dl.startswith(HASH_SPACE) and\n dl.endswith(PYTHON_EXTENSION)):\n self._debug(\n \"Removing filename comment: {!r}\".format(dl))\n dl = None\n\n elif TRIPLE_DOUBLEQUOTE in dl:\n if (not dl.startswith(TRIPLE_DOUBLEQUOTE) and\n not dl.startswith(RAW_TRIPLE_DOUBLEQUOTE)):\n self._warn(\n \"Triple-quote not at start of line, as follows\")\n self._debug_line(linenum, dl)\n self._too_risky()\n return\n if in_docstring: # docstring finishing\n in_docstring = False\n docstring_done = True\n in_body = True\n # ... and keep dl, so we write the end of the\n # docstring, potentially with e.g. \"# noqa\" on the end\n elif not docstring_done: # docstring starting\n in_docstring = True\n # self._critical(\"adding our new docstring\")\n # Write our new docstring's start\n tdq = \"\" # stops linter moaning\n if dl.startswith(TRIPLE_DOUBLEQUOTE):\n tdq = TRIPLE_DOUBLEQUOTE\n elif dl.startswith(RAW_TRIPLE_DOUBLEQUOTE):\n tdq = RAW_TRIPLE_DOUBLEQUOTE\n else:\n assert \"Bug!\"\n self.dest_lines.append(tdq)\n self.dest_lines.append(self.advertised_filename)\n self.dest_lines.append(BLANK)\n self.dest_lines.extend(self.correct_copyright_lines)\n self.dest_lines.append(BLANK)\n swallow_blanks_and_filename_in_docstring = True\n if dl == tdq:\n dl = None # don't write another triple-quote line\n else:\n dl = dl[len(tdq):]\n\n elif in_docstring:\n # Reading within the source docstring\n\n if dl == TRANSITION:\n if in_copyright: # copyright finishing\n in_copyright = False\n copyright_done = True\n dl = None # we've already replaced with our own\n elif not copyright_done:\n in_copyright = True\n dl = None # we've already replaced with our own\n\n elif in_copyright:\n dl = None # we've already replaced with our own\n\n elif dl == RST_COMMENT_LINE:\n dl = None # remove these\n\n elif swallow_blanks_and_filename_in_docstring:\n # self._debug_line(linenum, dl)\n if dl == BLANK or dl == self.advertised_filename:\n dl = None\n elif copyright_done:\n swallow_blanks_and_filename_in_docstring = False\n\n elif not dl.startswith(HASH) and not dl == BLANK:\n in_body = True\n\n if not docstring_done:\n # The source file didn't have a docstring!\n new_docstring_lines = [\n BLANK,\n TRIPLE_DOUBLEQUOTE,\n self.advertised_filename,\n BLANK,\n ] + self.correct_copyright_lines + [\n BLANK,\n MISSING_RST_TITLE,\n BLANK,\n TRIPLE_DOUBLEQUOTE\n ]\n self._warn(\"File had no docstring; adding one. \"\n \"Will need manual edit to add RST title. \"\n \"Search for {!r}\".format(MISSING_RST_TITLE))\n self.dest_lines[1:1] = new_docstring_lines\n\n if dl is not None:\n # self._debug_line(linenum, dl, \"adding \")\n self.dest_lines.append(dl)\n\n self.needs_rewriting = self.dest_lines != self.source_lines"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _debug_line(linenum: int, line: str, extramsg: str = \"\") -> None:\n log.critical(\"{}Line {}: {!r}\", extramsg, linenum, line)", "response": "Writes a debugging report on a line."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nrewrite the source file.", "response": "def rewrite_file(self) -> None:\n \"\"\"\n Rewrites the source file.\n \"\"\"\n if not self.needs_rewriting:\n return\n self._info(\"Rewriting file\")\n with open(self.full_path, \"w\") as outfile:\n self._write(outfile)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nwrite the converted output to a destination.", "response": "def _write(self, destination: TextIO) -> None:\n \"\"\"\n Writes the converted output to a destination.\n \"\"\"\n for line in self.dest_lines:\n destination.write(line + NL)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef contains_duplicates(values: Iterable[Any]) -> bool:\n for v in Counter(values).values():\n if v > 1:\n return True\n return False", "response": "Returns True if the iterable contains any duplicate values."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a list of indexes of x in order of the data x.", "response": "def index_list_for_sort_order(x: List[Any], key: Callable[[Any], Any] = None,\n reverse: bool = False) -> List[int]:\n \"\"\"\n Returns a list of indexes of ``x``, IF ``x`` WERE TO BE SORTED.\n\n Args:\n x: data\n key: function to be applied to the data to generate a sort key; this\n function is passed as the ``key=`` parameter to :func:`sorted`;\n the default is ``itemgetter(1)``\n reverse: reverse the sort order?\n\n Returns:\n list of integer index values\n\n Example:\n\n .. code-block:: python\n\n z = [\"a\", \"c\", \"b\"]\n index_list_for_sort_order(z) # [0, 2, 1]\n index_list_for_sort_order(z, reverse=True) # [1, 2, 0]\n q = [(\"a\", 9), (\"b\", 8), (\"c\", 7)]\n index_list_for_sort_order(q, key=itemgetter(1))\n\n \"\"\"\n def key_with_user_func(idx_val: Tuple[int, Any]):\n return key(idx_val[1])\n if key:\n sort_key = key_with_user_func\n # see the simpler version below\n else:\n sort_key = itemgetter(1)\n # enumerate, below, will return tuples of (index, value), so\n # itemgetter(1) means sort by the value\n index_value_list = sorted(enumerate(x), key=sort_key, reverse=reverse)\n return [i for i, _ in index_value_list]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef sort_list_by_index_list(x: List[Any], indexes: List[int]) -> None:\n x[:] = [x[i] for i in indexes]", "response": "Re - orders x by the list of indexes of x in place."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef flatten_list(x: List[Any]) -> List[Any]:\n return [item for sublist in x for item in sublist]", "response": "Converts a list of lists into a flat list."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a list of all the unique elements in the input list.", "response": "def unique_list(seq: Iterable[Any]) -> List[Any]:\n \"\"\"\n Returns a list of all the unique elements in the input list.\n\n Args:\n seq: input list\n\n Returns:\n list of unique elements\n\n As per\n http://stackoverflow.com/questions/480214/how-do-you-remove-duplicates-from-a-list-in-whilst-preserving-order\n\n \"\"\" # noqa\n seen = set()\n seen_add = seen.add\n return [x for x in seq if not (x in seen or seen_add(x))]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef chunks(l: List[Any], n: int) -> Iterable[List[Any]]:\n for i in range(0, len(l), n):\n yield l[i:i + n]", "response": "Yields n - sized chunks from l."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef escape_newlines(s: str) -> str:\n if not s:\n return s\n s = s.replace(\"\\\\\", r\"\\\\\") # replace \\ with \\\\\n s = s.replace(\"\\n\", r\"\\n\") # escape \\n; note ord(\"\\n\") == 10\n s = s.replace(\"\\r\", r\"\\r\") # escape \\r; note ord(\"\\r\") == 13\n return s", "response": "Escapes newlines in a string."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef unescape_newlines(s: str) -> str:\n # See also http://stackoverflow.com/questions/4020539\n if not s:\n return s\n d = \"\" # the destination string\n in_escape = False\n for i in range(len(s)):\n c = s[i] # the character being processed\n if in_escape:\n if c == \"r\":\n d += \"\\r\"\n elif c == \"n\":\n d += \"\\n\"\n else:\n d += c\n in_escape = False\n else:\n if c == \"\\\\\":\n in_escape = True\n else:\n d += c\n return d", "response": "Unescapes newlines in a string."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nescapes CR LF tab and backslashes.", "response": "def escape_tabs_newlines(s: str) -> str:\n \"\"\"\n Escapes CR, LF, tab, and backslashes.\n\n Its counterpart is :func:`unescape_tabs_newlines`.\n \"\"\"\n if not s:\n return s\n s = s.replace(\"\\\\\", r\"\\\\\") # replace \\ with \\\\\n s = s.replace(\"\\n\", r\"\\n\") # escape \\n; note ord(\"\\n\") == 10\n s = s.replace(\"\\r\", r\"\\r\") # escape \\r; note ord(\"\\r\") == 13\n s = s.replace(\"\\t\", r\"\\t\") # escape \\t; note ord(\"\\t\") == 9\n return s"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts a list of integers or hex range strings like 0061 - 007A to a string.", "response": "def _unicode_def_src_to_str(srclist: List[Union[str, int]]) -> str:\n \"\"\"\n Used to create :data:`UNICODE_CATEGORY_STRINGS`.\n\n Args:\n srclist: list of integers or hex range strings like ``\"0061-007A\"``\n\n Returns:\n a string with all characters described by ``srclist``: either the\n character corresponding to the integer Unicode character number, or\n all characters corresponding to the inclusive range described\n \"\"\"\n charlist = [] # type: List[str]\n for src in srclist:\n if isinstance(src, int):\n charlist.append(chr(src))\n else:\n # Range like \"0041-005A\"\n first, last = [int(x, 16) for x in src.split(\"-\")]\n charlist += [chr(x) for x in range(first, last + 1)]\n return \"\".join(charlist)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef report(*arrays, **kwargs):\n name = kwargs.pop(\"name\",None)\n\n grouped = len(arrays) > 1\n if grouped:\n arr = N.concatenate(arrays)\n components = [PCAOrientation(a)\n for a in arrays]\n else:\n arr = arrays[0]\n components = []\n\n #r = LinearOrientation(arr)\n pca = PCAOrientation(arr)\n\n distances = list(distance_from_group(components,pca))\n\n kwargs = dict(\n levels=[1,2,3],\n alpha=[0.8,0.5,0.2],\n linewidth=2)\n\n #ellipse=error_ellipse(pca)\n\n kwargs = dict(n=500,levels=[1,2], ellipse=True)\n stereonet_data = dict(\n main=pca.error_coords(**kwargs),\n components=[i.error_coords(**kwargs)\n for i in components])\n\n t = env.get_template(\"report.html\")\n\n return t.render(\n name=name,\n pca=pca,\n stereonet_data=stereonet_data,\n angular_errors=tuple(N.degrees(i)\n for i in pca.angular_errors()[::-1]),\n aligned=plot_aligned(pca),\n distances=distances)", "response": "Outputs a standalone HTML report card for a single set of measurements."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nchecks that stream record is actual tweet with coordinates", "response": "def validate_geotweet(self, record):\n \"\"\" check that stream record is actual tweet with coordinates \"\"\"\n if record and self._validate('user', record) \\\n and self._validate('coordinates', record):\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncompute the half angle subtended for an elliptical conic from the origin or an arbitrary viewpoint.", "response": "def angle_subtended(ell, **kwargs):\n \"\"\"\n Compute the half angle subtended (or min and max angles)\n for an offset elliptical conic\n from the origin or an arbitrary viewpoint.\n\n kwargs:\n tangent Return tangent instead of angle (default false)\n viewpoint Defaults to origin\n \"\"\"\n return_tangent = kwargs.pop('tangent',False)\n\n con, transform, offset = ell.projection(**kwargs)\n v = N.linalg.norm(N.array(con.major_axes()),axis=1)\n A = N.sort(v)[::-1] # Sort highest values first\n A = N.squeeze(A)\n B = N.linalg.norm(offset)\n if return_tangent: return A/B\n return N.arctan2(A,B)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef from_axes(cls,axes):\n ax = list(axes)\n #ax[-1] *= -1 # Not sure what is going on here...\n arr = N.diag(ax + [-1])\n return arr.view(cls)", "response": "Get elliptical conic from axis - aligned elliptical conic"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget elliptical conic from axis - aligned elliptical conic", "response": "def from_semiaxes(cls,axes):\n \"\"\"\n Get axis-aligned elliptical conic from axis lenths\n This can be converted into a hyperbola by getting the dual conic\n \"\"\"\n ax = list(1/N.array(axes)**2)\n #ax[-1] *= -1 # Not sure what is going on here...\n arr = N.diag(ax + [-1])\n return arr.view(cls)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef contains(ell, p, shell_only=False):\n v = augment(p)\n _ = ell.solve(v)\n return N.allclose(_,0) if shell_only else _ <= 0", "response": "Check to see if a point is inside the conic s entry set."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets the major axes of the ellipsoids", "response": "def major_axes(ell):\n \"\"\"\n Gets major axes of ellipsoids\n \"\"\"\n _ = ell[:-1,:-1]\n U,s,V = N.linalg.svd(_)\n scalar = -(ell.sum()-_.sum())\n return N.sqrt(s*scalar)*V"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ntranslating a conic by a vector", "response": "def translate(conic, vector):\n \"\"\"\n Translates a conic by a vector\n \"\"\"\n # Translation matrix\n T = N.identity(len(conic))\n T[:-1,-1] = -vector\n return conic.transform(T)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef polar_plane(conic, vector):\n pole = augment(vector)\n return dot(conic,pole).view(Plane)", "response": "Calculates the polar plane to a given vector."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef pole(conic, plane):\n v = dot(N.linalg.inv(conic),plane)\n return v[:-1]/v[-1]", "response": "Calculates the pole of a polar plane for a given conic section and a given plane."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef guess_file_name_stream_type_header(args):\n ftype = None\n fheader = None\n if isinstance(args, (tuple, list)):\n if len(args) == 2:\n fname, fstream = args\n elif len(args) == 3:\n fname, fstream, ftype = args\n else:\n fname, fstream, ftype, fheader = args\n else:\n fname, fstream = guess_filename_stream(args)\n ftype = guess_content_type(fname)\n\n if isinstance(fstream, (str, bytes, bytearray)):\n fdata = fstream\n else:\n fdata = fstream.read()\n return fname, fdata, ftype, fheader", "response": "Guess filename stream file type and file header from args."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nbuilds the body for a multipart / form - data request.", "response": "def encode_params(self, data=None, files=None, **kwargs):\n \"\"\"\n Build the body for a multipart/form-data request.\n Will successfully encode files when passed as a dict or a list of\n tuples. Order is retained if data is a list of tuples but arbitrary\n if parameters are supplied as a dict.\n The tuples may be string (filepath), 2-tuples (filename, fileobj), 3-tuples (filename, fileobj, contentype)\n or 4-tuples (filename, fileobj, contentype, custom_headers).\n \"\"\"\n if isinstance(data, basestring):\n raise ValueError(\"Data must not be a string.\")\n\n # optional args\n boundary = kwargs.get(\"boundary\", None)\n output_str = kwargs.get(\"output_str\", self.output_str)\n\n new_fields = []\n fields = to_key_val_list(data or {})\n files = to_key_val_list(files or {})\n\n for field, value in fields:\n ctype = None\n if isinstance(value, (tuple, list)) and len(value) == 2:\n val, ctype = value\n else:\n val = value\n\n if isinstance(val, basestring) or not hasattr(val, '__iter__'):\n val = [val]\n for v in val:\n # Don't call str() on bytestrings: in Py3 it all goes wrong.\n if not isinstance(v, bytes):\n v = to_string(v, lang=output_str)\n\n field = field.decode('utf-8') if isinstance(field, bytes) else field\n v = v.encode('utf-8') if isinstance(v, str) else v\n\n rf = RequestField(name=field, data=v)\n rf.make_multipart(content_type=ctype)\n new_fields.append(rf)\n\n for (k, v) in files:\n fn, fdata, ft, fh = guess_file_name_stream_type_header(v)\n rf = RequestField(name=k, data=fdata, filename=fn, headers=fh)\n rf.make_multipart(content_type=ft)\n new_fields.append(rf)\n\n if boundary is None:\n boundary = self.boundary\n body, content_type = encode_multipart_formdata(new_fields, boundary=boundary)\n\n return body, content_type"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef encode_params(self, data=None, **kwargs):\n collection_format = kwargs.get(\"collection_format\", self.collection_format)\n output_str = kwargs.get(\"output_str\", self.output_str)\n sort = kwargs.get(\"sort\", self.sort)\n\n if data is None:\n return \"\", self.content_type\n elif isinstance(data, (str, bytes)):\n return data, self.content_type\n elif hasattr(data, 'read'):\n return data, self.content_type\n elif collection_format == 'multi' and hasattr(data, '__iter__'):\n result = []\n for k, vs in to_key_val_list(data, sort=sort):\n if isinstance(vs, basestring) or not hasattr(vs, '__iter__'):\n vs = [vs]\n for v in vs:\n result.append(\n (k.encode('utf-8') if isinstance(k, str) else k,\n v.encode('utf-8') if isinstance(v, str) else to_string(v, lang=output_str)))\n return urlencode(result, doseq=True), self.content_type\n elif collection_format == 'encoded' and hasattr(data, '__iter__'):\n return urlencode(data, doseq=False), self.content_type\n elif hasattr(data, '__iter__'):\n results = []\n for k, vs in to_key_val_dict(data).items():\n if isinstance(vs, list):\n v = self.COLLECTION_SEPARATORS[collection_format].join(quote_plus(e) for e in vs)\n key = k + '[]'\n else:\n v = quote_plus(vs)\n key = k\n results.append(\"%s=%s\" % (key, v))\n\n return '&'.join(results), self.content_type\n else:\n return data, self.content_type", "response": "Encode parameters in a piece of data."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef encode_params(self, data=None, **kwargs):\n charset = kwargs.get(\"charset\", self.charset)\n collection_format = kwargs.get(\"collection_format\", self.collection_format)\n output_str = kwargs.get(\"output_str\", self.output_str)\n\n if data is None:\n return \"\", self.get_content_type(charset)\n elif isinstance(data, (str, bytes)):\n return data, self.get_content_type(charset)\n elif hasattr(data, 'read'):\n return data, self.get_content_type(charset)\n elif collection_format == 'multi' and hasattr(data, '__iter__'):\n result = []\n for k, vs in to_key_val_list(data):\n if isinstance(vs, basestring) or not hasattr(vs, '__iter__'):\n vs = [vs]\n for v in vs:\n result.append(b\"=\".join([self._encode(k, charset), self._encode(v, charset, output_str)]))\n return b'\\n'.join(result), self.get_content_type(charset)\n elif collection_format == 'plain' and hasattr(data, '__iter__'):\n results = []\n for k, vs in to_key_val_dict(data).items():\n results.append(b\"=\".join([self._encode(k, charset), self._encode(vs, charset, output_str)]))\n\n return b'\\n'.join(results), self.get_content_type(charset)\n elif hasattr(data, '__iter__'):\n results = []\n for k, vs in to_key_val_dict(data).items():\n if isinstance(vs, list):\n v = self.COLLECTION_SEPARATORS[collection_format].join(e for e in vs)\n key = k + '[]'\n else:\n v = vs\n key = k\n results.append(b\"=\".join([self._encode(key, charset), self._encode(v, charset, output_str)]))\n\n return b\"\\n\".join(results), self.get_content_type(charset)\n else:\n return str(data).encode(charset) if charset else str(data), self.get_content_type(charset)", "response": "Builds the body for a text / plain request."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nbuilds the body for a application / json request.", "response": "def encode_params(self, data=None, **kwargs):\n \"\"\"\n Build the body for a application/json request.\n \"\"\"\n if isinstance(data, basestring):\n raise ValueError(\"Data must not be a string.\")\n if data is None:\n return b\"\", self.content_type\n\n fields = to_key_val_dict(data or \"\")\n try:\n body = json.dumps(fields)\n except:\n body = json.dumps(fields, encoding='latin-1')\n\n return str(body).encode(), self.content_type"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsplitting command - line into pieces.", "response": "def cmdline_split(s: str, platform: Union[int, str] = 'this') -> List[str]:\n \"\"\"\n As per\n https://stackoverflow.com/questions/33560364/python-windows-parsing-command-lines-with-shlex.\n\n Multi-platform variant of ``shlex.split()`` for command-line splitting.\n For use with ``subprocess``, for ``argv`` injection etc. Using fast REGEX.\n \n Args:\n s:\n string to split\n platform:\n - ``'this'`` = auto from current platform;\n - ``1`` = POSIX;\n - ``0`` = Windows/CMD\n - (other values reserved)\n \"\"\" # noqa\n if platform == 'this':\n platform = (sys.platform != 'win32') # RNC: includes 64-bit Windows\n if platform == 1: # POSIX\n re_cmd_lex = r'''\"((?:\\\\[\"\\\\]|[^\"])*)\"|'([^']*)'|(\\\\.)|(&&?|\\|\\|?|\\d?\\>|[<])|([^\\s'\"\\\\&|<>]+)|(\\s+)|(.)''' # noqa\n elif platform == 0: # Windows/CMD\n re_cmd_lex = r'''\"((?:\"\"|\\\\[\"\\\\]|[^\"])*)\"?()|(\\\\\\\\(?=\\\\*\")|\\\\\")|(&&?|\\|\\|?|\\d?>|[<])|([^\\s\"&|<>]+)|(\\s+)|(.)''' # noqa\n else:\n raise AssertionError('unknown platform %r' % platform)\n\n args = []\n accu = None # collects pieces of one arg\n for qs, qss, esc, pipe, word, white, fail in re.findall(re_cmd_lex, s):\n if word:\n pass # most frequent\n elif esc:\n word = esc[1]\n elif white or pipe:\n if accu is not None:\n args.append(accu)\n if pipe:\n args.append(pipe)\n accu = None\n continue\n elif fail:\n raise ValueError(\"invalid or incomplete shell string\")\n elif qs:\n word = qs.replace('\\\\\"', '\"').replace('\\\\\\\\', '\\\\')\n if platform == 0:\n word = word.replace('\"\"', '\"')\n else:\n word = qss # may be even empty; must be last\n\n accu = (accu or '') + word\n\n if accu is not None:\n args.append(accu)\n\n return args"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nlike scoreatpercentile but can take and return array of percentiles.", "response": "def percentiles(a, pcts, axis=None):\n \"\"\"Like scoreatpercentile but can take and return array of percentiles.\n Parameters\n ----------\n a : array\n data\n pcts : sequence of percentile values\n percentile or percentiles to find score at\n axis : int or None\n if not None, computes scores over this axis\n Returns\n -------\n scores: array\n array of scores at requested percentiles\n first dimension is length of object passed to ``pcts``\n \"\"\"\n scores = []\n try:\n n = len(pcts)\n except TypeError:\n pcts = [pcts]\n n = 0\n for i, p in enumerate(pcts):\n if axis is None:\n score = stats.scoreatpercentile(a.ravel(), p)\n else:\n score = N.apply_along_axis(stats.scoreatpercentile, axis, a, p)\n scores.append(score)\n scores = N.asarray(scores)\n if not n:\n scores = scores.squeeze()\n return scores"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a percentile range from an array of values.", "response": "def ci(a, which=95, axis=None):\n \"\"\"Return a percentile range from an array of values.\"\"\"\n p = 50 - which / 2, 50 + which / 2\n return percentiles(a, p, axis)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nretrieving a string value from a ConfigParser instance", "response": "def get_config_string_option(parser: ConfigParser,\n section: str,\n option: str,\n default: str = None) -> str:\n \"\"\"\n Retrieves a string value from a parser.\n\n Args:\n parser: instance of :class:`ConfigParser`\n section: section name within config file\n option: option (variable) name within that section\n default: value to return if option is absent\n\n Returns:\n string value\n\n Raises:\n ValueError: if the section is absent\n\n \"\"\"\n if not parser.has_section(section):\n raise ValueError(\"config missing section: \" + section)\n return parser.get(section, option, fallback=default)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef read_config_string_options(obj: Any,\n parser: ConfigParser,\n section: str,\n options: Iterable[str],\n default: str = None) -> None:\n \"\"\"\n Reads config options and writes them as attributes of ``obj``, with\n attribute names as per ``options``.\n\n Args:\n obj: the object to modify\n parser: instance of :class:`ConfigParser`\n section: section name within config file\n options: option (variable) names within that section\n default: value to use for any missing options\n\n Returns:\n\n \"\"\"\n # enforce_str removed; ConfigParser always returns strings unless asked\n # specifically\n for o in options:\n setattr(obj, o, get_config_string_option(parser, section, o,\n default=default))", "response": "Reads config options and writes them as attributes of obj with the names of the options as per options."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a list of strings that are multi - line option values from a parser.", "response": "def get_config_multiline_option(parser: ConfigParser,\n section: str,\n option: str,\n default: List[str] = None) -> List[str]:\n \"\"\"\n Retrieves a multi-line string value from a parser as a list of strings\n (one per line, ignoring blank lines).\n\n Args:\n parser: instance of :class:`ConfigParser`\n section: section name within config file\n option: option (variable) name within that section\n default: value to return if option is absent (``None`` is mapped to\n ``[]``)\n\n Returns:\n list of strings\n\n Raises:\n ValueError: if the section is absent\n\n \"\"\"\n default = default or []\n if not parser.has_section(section):\n raise ValueError(\"config missing section: \" + section)\n try:\n multiline = parser.get(section, option)\n values = [x.strip() for x in multiline.splitlines() if x.strip()]\n return values\n except NoOptionError:\n return default"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nread options from a multiline section.", "response": "def read_config_multiline_options(obj: Any,\n parser: ConfigParser,\n section: str,\n options: Iterable[str]) -> None:\n \"\"\"\n This is to :func:`read_config_string_options` as\n :func:`get_config_multiline_option` is to :func:`get_config_string_option`.\n \"\"\"\n for o in options:\n setattr(obj, o, get_config_multiline_option(parser, section, o))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_config_bool_option(parser: ConfigParser,\n section: str,\n option: str,\n default: bool = None) -> bool:\n \"\"\"\n Retrieves a boolean value from a parser.\n\n Args:\n parser: instance of :class:`ConfigParser`\n section: section name within config file\n option: option (variable) name within that section\n default: value to return if option is absent\n\n Returns:\n string value\n\n Raises:\n ValueError: if the section is absent\n\n \"\"\"\n if not parser.has_section(section):\n raise ValueError(\"config missing section: \" + section)\n return parser.getboolean(section, option, fallback=default)", "response": "Retrieves a boolean value from a ConfigParser instance"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_config_parameter(config: ConfigParser,\n section: str,\n param: str,\n fn: Callable[[Any], Any],\n default: Any) -> Any:\n \"\"\"\n Fetch parameter from ``configparser`` ``.INI`` file.\n\n Args:\n config: :class:`ConfigParser` object\n section: section name within config file\n param: name of parameter within section\n fn: function to apply to string parameter (e.g. ``int``)\n default: default value\n\n Returns:\n parameter value, or ``None`` if ``default is None``, or ``fn(default)``\n \"\"\"\n try:\n value = fn(config.get(section, param))\n except (TypeError, ValueError, NoOptionError):\n log.warning(\n \"Configuration variable {} not found or improper in section [{}]; \"\n \"using default of {!r}\", param, section, default)\n if default is None:\n value = default\n else:\n value = fn(default)\n return value", "response": "Get a parameter from configparser. INI file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_config_parameter_boolean(config: ConfigParser,\n section: str,\n param: str,\n default: bool) -> bool:\n \"\"\"\n Get Boolean parameter from ``configparser`` ``.INI`` file.\n\n Args:\n config: :class:`ConfigParser` object\n section: section name within config file\n param: name of parameter within section\n default: default value\n Returns:\n parameter value, or default\n \"\"\"\n try:\n value = config.getboolean(section, param)\n except (TypeError, ValueError, NoOptionError):\n log.warning(\n \"Configuration variable {} not found or improper in section [{}]; \"\n \"using default of {!r}\", param, section, default)\n value = default\n return value", "response": "Get Boolean parameter from ConfigParser object."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets the loglevel parameter from configparser. INI file e. g. logging. DEBUG to logging. INFO or logging. WARNING.", "response": "def get_config_parameter_loglevel(config: ConfigParser,\n section: str,\n param: str,\n default: int) -> int:\n \"\"\"\n Get ``loglevel`` parameter from ``configparser`` ``.INI`` file, e.g.\n mapping ``'debug'`` to ``logging.DEBUG``.\n\n Args:\n config: :class:`ConfigParser` object\n section: section name within config file\n param: name of parameter within section\n default: default value\n Returns:\n parameter value, or default\n \"\"\"\n try:\n value = config.get(section, param).lower()\n if value == \"debug\":\n return logging.DEBUG # 10\n elif value == \"info\":\n return logging.INFO\n elif value in [\"warn\", \"warning\"]:\n return logging.WARN\n elif value == \"error\":\n return logging.ERROR\n elif value in [\"critical\", \"fatal\"]:\n return logging.CRITICAL # 50\n else:\n raise ValueError\n except (TypeError, ValueError, NoOptionError, AttributeError):\n log.warning(\n \"Configuration variable {} not found or improper in section [{}]; \"\n \"using default of {!r}\", param, section, default)\n return default"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting a multi - line string parameter from configparser. INI file.", "response": "def get_config_parameter_multiline(config: ConfigParser,\n section: str,\n param: str,\n default: List[str]) -> List[str]:\n \"\"\"\n Get multi-line string parameter from ``configparser`` ``.INI`` file,\n as a list of strings (one per line, ignoring blank lines).\n\n Args:\n config: :class:`ConfigParser` object\n section: section name within config file\n param: name of parameter within section\n default: default value\n Returns:\n parameter value, or default\n \"\"\"\n try:\n multiline = config.get(section, param)\n lines = [x.strip() for x in multiline.splitlines()]\n return [line for line in lines if line]\n except (TypeError, ValueError, NoOptionError):\n log.warning(\n \"Configuration variable {} not found or improper in section [{}]; \"\n \"using default of {!r}\", param, section, default)\n return default"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef is_definition(cursor):\n defn = cursor.get_definition()\n return (defn is not None) and (cursor.location == defn.location)", "response": "Test if a cursor refers to a definition\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting a cone of asymptotes for hyperbola", "response": "def asymptotes(hyp, n=1000):\n \"\"\"\n Gets a cone of asymptotes for hyperbola\n \"\"\"\n assert N.linalg.norm(hyp.center()) == 0\n\n u = N.linspace(0,2*N.pi,n)\n _ = N.ones(len(u))\n angles = N.array([N.cos(u),N.sin(u),_]).T\n return dot(angles,hyp[:-1,:-1])"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef generic_service_main(cls: Type[WindowsService], name: str) -> None:\n argc = len(sys.argv)\n if argc == 1:\n try:\n print(\"Trying to start service directly...\")\n evtsrc_dll = os.path.abspath(servicemanager.__file__)\n # noinspection PyUnresolvedReferences\n servicemanager.PrepareToHostSingle(cls) # <-- sets up the service\n # noinspection PyUnresolvedReferences\n servicemanager.Initialize(name, evtsrc_dll)\n # noinspection PyUnresolvedReferences\n servicemanager.StartServiceCtrlDispatcher()\n except win32service.error as details:\n print(\"Failed: {}\".format(details))\n # print(repr(details.__dict__))\n errnum = details.winerror\n if errnum == winerror.ERROR_FAILED_SERVICE_CONTROLLER_CONNECT:\n win32serviceutil.usage()\n elif argc == 2 and sys.argv[1] == 'debug':\n s = cls()\n s.run_debug()\n else:\n win32serviceutil.HandleCommandLine(cls)", "response": "Main function for the generic service."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the full name of the process.", "response": "def fullname(self) -> str:\n \"\"\"\n Description of the process.\n \"\"\"\n fullname = \"Process {}/{} ({})\".format(self.procnum, self.nprocs,\n self.details.name)\n if self.running:\n fullname += \" (PID={})\".format(self.process.pid)\n return fullname"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef debug(self, msg: str) -> None:\n if self.debugging:\n s = \"{}: {}\".format(self.fullname, msg)\n log.debug(s)", "response": "Write a debug message to the Python disk log."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef info(self, msg: str) -> None:\n # noinspection PyUnresolvedReferences\n s = \"{}: {}\".format(self.fullname, msg)\n servicemanager.LogInfoMsg(s)\n if self.debugging:\n log.info(s)", "response": "Write an info message to the Windows Application log."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef warning(self, msg: str) -> None:\n # Log messages go to the Windows APPLICATION log.\n # noinspection PyUnresolvedReferences\n s = \"{}: {}\".format(self.fullname, msg)\n servicemanager.LogWarningMsg(s)\n if self.debugging:\n log.warning(s)", "response": "Write a warning message to the Windows Application log."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef error(self, msg: str) -> None:\n # noinspection PyUnresolvedReferences\n s = \"{}: {}\".format(self.fullname, msg)\n servicemanager.LogErrorMsg(s)\n if self.debugging:\n log.warning(s)", "response": "Write an error message to the Windows Application log."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nopen Python disk logs.", "response": "def open_logs(self) -> None:\n \"\"\"\n Open Python disk logs.\n \"\"\"\n if self.details.logfile_out:\n self.stdout = open(self.details.logfile_out, 'a')\n else:\n self.stdout = None\n if self.details.logfile_err:\n if self.details.logfile_err == self.details.logfile_out:\n self.stderr = subprocess.STDOUT\n else:\n self.stderr = open(self.details.logfile_err, 'a')\n else:\n self.stderr = None"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nclosing Python disk logs.", "response": "def close_logs(self) -> None:\n \"\"\"\n Close Python disk logs.\n \"\"\"\n if self.stdout is not None:\n self.stdout.close()\n self.stdout = None\n if self.stderr is not None and self.stderr != subprocess.STDOUT:\n self.stderr.close()\n self.stderr = None"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef start(self) -> None:\n if self.running:\n return\n self.info(\"Starting: {} (with logs stdout={}, stderr={})\".format(\n self.details.procargs,\n self.details.logfile_out,\n self.details.logfile_err))\n self.open_logs()\n creationflags = CREATE_NEW_PROCESS_GROUP if WINDOWS else 0\n # self.warning(\"creationflags: {}\".format(creationflags))\n self.process = subprocess.Popen(self.details.procargs, stdin=None,\n stdout=self.stdout, stderr=self.stderr,\n creationflags=creationflags)\n self.running = True", "response": "Starts a subprocess. Optionally routes its output to our disk logs."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef stop(self) -> None:\n if not self.running:\n return\n try:\n self.wait(timeout_s=0)\n # If we get here: stopped already\n except subprocess.TimeoutExpired: # still running\n for kill_level in self.ALL_KILL_LEVELS:\n tried_to_kill = self._terminate(level=kill_level) # please stop\n if tried_to_kill:\n try:\n self.wait(timeout_s=self.kill_timeout_sec)\n break\n except subprocess.TimeoutExpired: # failed to close\n self.warning(\"Subprocess didn't stop when asked\")\n pass # carry on escalating\n self.close_logs()\n self.running = False", "response": "Stops a subprocess. Waits. Asks less nicely. Waits."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _terminate(self, level: int) -> bool:\n if not self.running:\n return True\n\n # Already closed by itself?\n try:\n self.wait(0)\n return True\n except subprocess.TimeoutExpired: # failed to close\n pass\n\n # SEE NOTES ABOVE. This is tricky under Windows.\n\n suffix = \" [to child process {}]\".format(self.process.pid)\n if level == self.KILL_LEVEL_CTRL_C_OR_SOFT_KILL:\n if WINDOWS:\n success = 0 != ctypes.windll.kernel32.GenerateConsoleCtrlEvent(\n CTRL_C_EVENT, self.process.pid)\n if success:\n self.info(\"Sent CTRL-C to request stop\" + suffix)\n # ... but that doesn't mean it'll stop...\n else:\n self.info(\"Failed to send CTRL-C\" + suffix)\n return success\n else:\n self.warning(\"Asking process to stop (SIGTERM)\" + suffix)\n self.process.terminate() # soft kill under POSIX\n return True\n\n elif level == self.KILL_LEVEL_CTRL_BREAK:\n if not WINDOWS:\n return False\n success = 0 != ctypes.windll.kernel32.GenerateConsoleCtrlEvent(\n CTRL_BREAK_EVENT, self.process.pid)\n if success:\n self.info(\"Sent CTRL-BREAK to request stop\" + suffix)\n else:\n self.info(\"Failed to send CTRL-BREAK\" + suffix)\n return success\n\n elif level == self.KILL_LEVEL_TASKKILL:\n if not WINDOWS:\n return False\n retcode = self._taskkill(force=False) # does its own info messages\n return retcode == winerror.ERROR_SUCCESS\n\n elif level == self.KILL_LEVEL_TASKKILL_FORCE:\n if not WINDOWS:\n return False\n retcode = self._taskkill(force=True) # does its own info messages\n return retcode == winerror.ERROR_SUCCESS\n\n elif level == self.KILL_LEVEL_HARD_KILL:\n # Last resort\n self._kill() # may do TASKKILL /F or some other method\n return True\n\n else:\n raise ValueError(\"Bad kill level requested\")", "response": "Terminate the current process."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nexecute a Windows ``TASKKILL /pid PROCESS_ID /t`` command (``/t`` for \"tree kill\" = \"kill all children\"). Args: force: also add ``/f`` (forcefully) Returns: return code from ``TASKKILL`` **Test code:** Firstly we need a program that won't let itself be killed. Save this as ``nokill.py``: .. code-block:: python #!/usr/bin/env python import logging import time import os from cardinal_pythonlib.logs import main_only_quicksetup_rootlogger from cardinal_pythonlib.signalfunc import trap_ctrl_c_ctrl_break main_only_quicksetup_rootlogger(level=logging.DEBUG) trap_ctrl_c_ctrl_break() while True: print(\"Process ID is {}; time is {} s\".format(os.getpid(), time.clock())) time.sleep(1) Now run that with ``python nokill.py``. It should resist CTRL-C and CTRL-BREAK. Start another command prompt in which to play with ``TASKKILL``. .. code-block:: bat REM Firstly, avoid this single-ampersand syntax: REM taskkill /im notepad.exe & echo %errorlevel% REM ... as it prints the WRONG (previous?) errorlevel. notepad.exe taskkill /im notepad.exe echo %errorlevel% REM ... 0 for success (Windows 10), e.g. REM 'SUCCESS: Sent termination signal to the process \"notepad.exe\" with PID 6988.' taskkill /im notepad.exe echo %errorlevel% REM ... 128 for \"not found\" (Windows 10), e.g. REM 'ERROR: The process \"notepad.exe\" not found.' REM Now run notepad.exe as Administrator taskkill /im notepad.exe & echo %errorlevel% REM ... 1 for \"access denied\" (Windows 10) REM Now kill the nokill.py process by its PID (e.g. 11892 here): taskkill /pid 11892 echo %errorlevel% REM ... 1 for \"not allowed\" (Windows 10), e.g. REM 'ERROR: The process with PID 11892 could not be terminated.' REM 'Reason: This process can only be terminated forcefully (with /F option).' REM Now forcefully: taskkill /pid 11892 /f echo %errorlevel% REM ... 0 for success (Windows 10), e.g. REM 'SUCCESS: The process with PID 11892 has been terminated.'", "response": "def _taskkill(self, force: bool = False) -> int:\n \"\"\"\n Executes a Windows ``TASKKILL /pid PROCESS_ID /t`` command\n (``/t`` for \"tree kill\" = \"kill all children\").\n\n Args:\n force: also add ``/f`` (forcefully)\n\n Returns:\n return code from ``TASKKILL``\n\n **Test code:**\n\n Firstly we need a program that won't let itself be killed. Save this as\n ``nokill.py``:\n\n .. code-block:: python\n\n #!/usr/bin/env python\n\n import logging\n import time\n import os\n from cardinal_pythonlib.logs import main_only_quicksetup_rootlogger\n from cardinal_pythonlib.signalfunc import trap_ctrl_c_ctrl_break\n\n main_only_quicksetup_rootlogger(level=logging.DEBUG)\n trap_ctrl_c_ctrl_break()\n\n while True:\n print(\"Process ID is {}; time is {} s\".format(os.getpid(), time.clock()))\n time.sleep(1)\n\n Now run that with ``python nokill.py``. It should resist CTRL-C and\n CTRL-BREAK. Start another command prompt in which to play with\n ``TASKKILL``.\n\n .. code-block:: bat\n\n REM Firstly, avoid this single-ampersand syntax:\n REM taskkill /im notepad.exe & echo %errorlevel%\n REM ... as it prints the WRONG (previous?) errorlevel.\n\n notepad.exe\n taskkill /im notepad.exe\n echo %errorlevel%\n REM ... 0 for success (Windows 10), e.g.\n REM 'SUCCESS: Sent termination signal to the process \"notepad.exe\" with PID 6988.'\n\n taskkill /im notepad.exe\n echo %errorlevel%\n REM ... 128 for \"not found\" (Windows 10), e.g.\n REM 'ERROR: The process \"notepad.exe\" not found.'\n\n REM Now run notepad.exe as Administrator\n taskkill /im notepad.exe & echo %errorlevel%\n REM ... 1 for \"access denied\" (Windows 10)\n\n REM Now kill the nokill.py process by its PID (e.g. 11892 here):\n taskkill /pid 11892\n echo %errorlevel%\n REM ... 1 for \"not allowed\" (Windows 10), e.g.\n REM 'ERROR: The process with PID 11892 could not be terminated.'\n REM 'Reason: This process can only be terminated forcefully (with /F option).'\n\n REM Now forcefully:\n taskkill /pid 11892 /f\n echo %errorlevel%\n REM ... 0 for success (Windows 10), e.g.\n REM 'SUCCESS: The process with PID 11892 has been terminated.'\n \n \"\"\" # noqa\n args = [\n \"taskkill\", # built in to Windows XP and higher\n \"/pid\", str(self.process.pid),\n \"/t\", # tree kill: kill all children\n ]\n if force:\n args.append(\"/f\") # forcefully\n callname = \" \".join(args)\n retcode = subprocess.call(args)\n # http://stackoverflow.com/questions/18682681/what-are-exit-codes-from-the-taskkill-utility # noqa\n if retcode == winerror.ERROR_SUCCESS: # 0\n self.info(\"Killed with \" + repr(callname))\n elif retcode == winerror.ERROR_INVALID_FUNCTION: # 1\n self.warning(\n repr(callname) +\n \" failed (error code 1 = ERROR_INVALID_FUNCTION; \"\n \"can mean 'Access denied', or 'This process can only be \"\n \"terminated forcefully (with /F option)').\")\n elif retcode == winerror.ERROR_WAIT_NO_CHILDREN: # 128\n self.warning(\n repr(callname) +\n \" failed (error code 128 = ERROR_WAIT_NO_CHILDREN \"\n \"= 'There are no child processes to wait for', but also \"\n \"occurs when the process doesn't exist, and when processes \"\n \"require a forceful [/F] termination)\")\n elif retcode == winerror.ERROR_EA_LIST_INCONSISTENT: # 255\n self.warning(\n repr(callname) +\n \" failed (error code 255 = ERROR_EA_LIST_INCONSISTENT \"\n \"= 'The extended attributes are inconsistent.')\")\n else:\n self.warning(callname + \" failed: error code {}\".format(retcode))\n return retcode"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nwaits for the child process to finish.", "response": "def wait(self, timeout_s: float = None) -> int:\n \"\"\"\n Wait for up to ``timeout_s`` for the child process to finish.\n\n Args:\n timeout_s: maximum time to wait or ``None`` to wait forever\n\n Returns:\n process return code; or ``0`` if it wasn't running, or ``1`` if\n it managed to exit without a return code\n\n Raises:\n subprocess.TimeoutExpired: if the process continues to run\n\n \"\"\"\n if not self.running:\n return 0\n retcode = self.process.wait(timeout=timeout_s)\n # We won't get further unless the process has stopped.\n if retcode is None:\n self.error(\"Subprocess finished, but return code was None\")\n retcode = 1 # we're promising to return an int\n elif retcode == 0:\n self.info(\"Subprocess finished cleanly (return code 0).\")\n else:\n self.error(\n \"Subprocess finished, but FAILED (return code {}). \"\n \"Logs were: {} (stdout), {} (stderr)\".format(\n retcode,\n self.details.logfile_out,\n self.details.logfile_err))\n self.running = False\n return retcode"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef info(self, msg: str) -> None:\n # noinspection PyUnresolvedReferences\n servicemanager.LogInfoMsg(str(msg))\n if self.debugging:\n log.info(msg)", "response": "Write an info message to the Windows Application log."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef error(self, msg: str) -> None:\n # noinspection PyUnresolvedReferences\n servicemanager.LogErrorMsg(str(msg))\n if self.debugging:\n log.error(msg)", "response": "Write an error message to the Windows Application log."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalling when the service is being shut down.", "response": "def SvcStop(self) -> None:\n \"\"\"\n Called when the service is being shut down.\n \"\"\"\n # tell the SCM we're shutting down\n # noinspection PyUnresolvedReferences\n self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)\n # fire the stop event\n win32event.SetEvent(self.h_stop_event)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncalling when the service is started.", "response": "def SvcDoRun(self) -> None:\n \"\"\"\n Called when the service is started.\n \"\"\"\n # No need to self.ReportServiceStatus(win32service.SERVICE_RUNNING);\n # that is done by the framework (see win32serviceutil.py).\n # Similarly, no need to report a SERVICE_STOP_PENDING on exit.\n # noinspection PyUnresolvedReferences\n self.debug(\"Sending PYS_SERVICE_STARTED message\")\n # noinspection PyUnresolvedReferences\n servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,\n servicemanager.PYS_SERVICE_STARTED,\n (self._svc_name_, ''))\n\n # self.test_service() # test service\n self.main() # real service\n\n # noinspection PyUnresolvedReferences\n servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,\n servicemanager.PYS_SERVICE_STOPPED,\n (self._svc_name_, ''))\n # noinspection PyUnresolvedReferences\n self.ReportServiceStatus(win32service.SERVICE_STOPPED)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef main(self) -> None:\n # Actual main service code.\n try:\n self.service()\n except Exception as e:\n self.error(\"Unexpected exception: {e}\\n{t}\".format(\n e=e, t=traceback.format_exc()))", "response": "Main entry point. Runs service."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef run_processes(self,\n procdetails: List[ProcessDetails],\n subproc_run_timeout_sec: float = 1,\n stop_event_timeout_ms: int = 1000,\n kill_timeout_sec: float = 5) -> None:\n \"\"\"\n Run multiple child processes.\n\n Args:\n\n procdetails: list of :class:`ProcessDetails` objects (q.v.)\n subproc_run_timeout_sec: time (in seconds) to wait for each process\n when polling child processes to see how they're getting on\n (default ``1``)\n stop_event_timeout_ms: time to wait (in ms) while checking the\n Windows stop event for this service (default ``1000``)\n kill_timeout_sec: how long (in seconds) will we wait for the\n subprocesses to end peacefully, before we try to kill them?\n\n .. todo::\n cardinal_pythonlib.winservice.WindowsService: NOT YET IMPLEMENTED:\n Windows service autorestart\n \"\"\"\n\n # https://stackoverflow.com/questions/16333054\n def cleanup():\n self.debug(\"atexit function called: cleaning up\")\n for pmgr_ in self.process_managers:\n pmgr_.stop()\n\n atexit.register(cleanup)\n\n # Set up process info\n self.process_managers = [] # type: List[ProcessManager]\n n = len(procdetails)\n for i, details in enumerate(procdetails):\n pmgr = ProcessManager(details, i + 1, n,\n kill_timeout_sec=kill_timeout_sec,\n debugging=self.debugging)\n self.process_managers.append(pmgr)\n\n # Start processes\n for pmgr in self.process_managers:\n pmgr.start()\n self.info(\"All started\")\n\n # Run processes\n something_running = True\n stop_requested = False\n subproc_failed = False\n while something_running and not stop_requested and not subproc_failed:\n if (win32event.WaitForSingleObject(\n self.h_stop_event,\n stop_event_timeout_ms) == win32event.WAIT_OBJECT_0):\n stop_requested = True\n self.info(\"Stop requested; stopping\")\n else:\n something_running = False\n for pmgr in self.process_managers:\n if subproc_failed:\n break\n try:\n retcode = pmgr.wait(timeout_s=subproc_run_timeout_sec)\n if retcode != 0:\n subproc_failed = True\n except subprocess.TimeoutExpired:\n something_running = True\n\n # Kill any outstanding processes\n #\n # (a) Slow way\n # for pmgr in self.process_managers:\n # pmgr.stop()\n #\n # (b) Faster (slightly more parallel) way\n # for pmgr in self.process_managers:\n # pmgr.terminate()\n # for pmgr in self.process_managers:\n # pmgr.stop_having_terminated()\n #\n # ... No, it's bad if we leave things orphaned.\n # Let's go for slow, clean code.\n\n for pmgr in self.process_managers:\n pmgr.stop()\n self.info(\"All stopped\")", "response": "Run multiple child processes."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_info_to_exception(err: Exception, info: Dict) -> None:\n if not err.args:\n err.args = ('', )\n err.args += (info, )", "response": "Add an information dictionary to an exception."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef recover_info_from_exception(err: Exception) -> Dict:\n if len(err.args) < 1:\n return {}\n info = err.args[-1]\n if not isinstance(info, dict):\n return {}\n return info", "response": "Retains the information added to an exception by\n ."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nserializes a function object to a dict.", "response": "def _serialize_function(obj):\n \"\"\"\n Still needing this much try-except stuff. We should find a way to get rid of this.\n :param obj:\n :return:\n \"\"\"\n try:\n obj = inspect.getsource(obj)\n except (TypeError, IOError):\n try:\n obj = marshal.dumps(obj)\n except ValueError:\n if hasattr(obj, '__dict__'):\n obj = _serialize_dict(obj.__dict__)\n return obj"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts a module into a dict.", "response": "def _serialize_module(obj):\n \"\"\"\n Tries to serialize a module by its __dict__ attr.\n Remove the builtins attr as this one is not relevant and extremely large.\n If its value is a callable, serialize it using serialize_obj, else, use its repr,\n because in this case we most likely run into max recursion depth errors.\n :param obj:\n :return:\n \"\"\"\n obj = dict(obj.__dict__)\n if '__builtins__' in obj.keys():\n obj.pop('__builtins__')\n for k, v in obj.items():\n if callable(v):\n obj[k] = serialize_obj(v)\n else:\n obj[k] = repr(v)\n return obj"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _serialize_iterable(obj):\n if isinstance(obj, (tuple, set)):\n # make a tuple assignable by casting it to list\n obj = list(obj)\n for item in obj:\n obj[obj.index(item)] = serialize_obj(item)\n return obj", "response": "Serialize an iterable into a list of the available base classes."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndisable boolean icons for a Django ModelAdmin field.", "response": "def disable_bool_icon(\n fieldname: str,\n model) -> Callable[[Any], bool]:\n \"\"\"\n Disable boolean icons for a Django ModelAdmin field.\n The '_meta' attribute is present on Django model classes and instances.\n\n model_class: ``Union[Model, Type[Model]]``\n\n ... only the type checker in Py3.5 is broken; see ``files.py``\n \"\"\"\n # noinspection PyUnusedLocal\n def func(self, obj):\n return getattr(obj, fieldname)\n func.boolean = False\n func.admin_order_field = fieldname\n # func.short_description = \\\n # model._meta.get_field_by_name(fieldname)[0].verbose_name\n # get_field_by_name() deprecated in Django 1.9 and will go in 1.10\n # https://docs.djangoproject.com/en/1.8/ref/models/meta/\n\n # noinspection PyProtectedMember, PyUnresolvedReferences\n func.short_description = \\\n model._meta.get_field(fieldname).verbose_name\n return func"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget a Django admin site URL for an object.", "response": "def admin_view_url(admin_site: AdminSite,\n obj,\n view_type: str = \"change\",\n current_app: str = None) -> str:\n \"\"\"\n Get a Django admin site URL for an object.\n \"\"\"\n app_name = obj._meta.app_label.lower()\n model_name = obj._meta.object_name.lower()\n pk = obj.pk\n viewname = \"admin:{}_{}_{}\".format(app_name, model_name, view_type)\n if current_app is None:\n current_app = admin_site.name\n url = reverse(viewname, args=[pk], current_app=current_app)\n return url"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting a Django admin site URL for an object that s found from a foreign key in our object of interest.", "response": "def admin_view_fk_link(modeladmin: ModelAdmin,\n obj,\n fkfield: str,\n missing: str = \"(None)\",\n use_str: bool = True,\n view_type: str = \"change\",\n current_app: str = None) -> str:\n \"\"\"\n Get a Django admin site URL for an object that's found from a foreign\n key in our object of interest.\n \"\"\"\n if not hasattr(obj, fkfield):\n return missing\n linked_obj = getattr(obj, fkfield)\n app_name = linked_obj._meta.app_label.lower()\n model_name = linked_obj._meta.object_name.lower()\n viewname = \"admin:{}_{}_{}\".format(app_name, model_name, view_type)\n # https://docs.djangoproject.com/en/dev/ref/contrib/admin/#reversing-admin-urls # noqa\n if current_app is None:\n current_app = modeladmin.admin_site.name\n # ... plus a bit of home-grown magic; see Django source\n url = reverse(viewname, args=[linked_obj.pk], current_app=current_app)\n if use_str:\n label = escape(str(linked_obj))\n else:\n label = \"{} {}\".format(escape(linked_obj._meta.object_name),\n linked_obj.pk)\n return '<a href=\"{}\">{}</a>'.format(url, label)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef admin_view_reverse_fk_links(modeladmin: ModelAdmin,\n obj,\n reverse_fk_set_field: str,\n missing: str = \"(None)\",\n use_str: bool = True,\n separator: str = \"<br>\",\n view_type: str = \"change\",\n current_app: str = None) -> str:\n \"\"\"\n Get multiple Django admin site URL for multiple objects linked to our\n object of interest (where the other objects have foreign keys to our\n object).\n \"\"\"\n if not hasattr(obj, reverse_fk_set_field):\n return missing\n linked_objs = getattr(obj, reverse_fk_set_field).all()\n if not linked_objs:\n return missing\n first = linked_objs[0]\n app_name = first._meta.app_label.lower()\n model_name = first._meta.object_name.lower()\n viewname = \"admin:{}_{}_{}\".format(app_name, model_name, view_type)\n if current_app is None:\n current_app = modeladmin.admin_site.name\n links = []\n for linked_obj in linked_objs:\n # log.debug(\"linked_obj: {}\", linked_obj)\n url = reverse(viewname, args=[linked_obj.pk], current_app=current_app)\n\n if use_str:\n label = escape(str(linked_obj))\n else:\n label = \"{} {}\".format(escape(linked_obj._meta.object_name),\n linked_obj.pk)\n links.append('<a href=\"{}\">{}</a>'.format(url, label))\n # log.debug(\"links: {}\", links)\n return separator.join(links)", "response": "Get multiple Django admin site URL for multiple objects linked to our object."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\napplying a low - pass filter to the data.", "response": "def lowpass_filter(data: FLOATS_TYPE,\n sampling_freq_hz: float,\n cutoff_freq_hz: float,\n numtaps: int) -> FLOATS_TYPE:\n \"\"\"\n Apply a low-pass filter to the data.\n\n Args:\n data: time series of the data\n sampling_freq_hz: sampling frequency :math:`f_s`, in Hz\n (or other consistent units)\n cutoff_freq_hz: filter cutoff frequency in Hz\n (or other consistent units)\n numtaps: number of filter taps\n\n Returns:\n filtered data\n\n Note: number of filter taps = filter order + 1\n \"\"\"\n coeffs = firwin(\n numtaps=numtaps,\n cutoff=normalized_frequency(cutoff_freq_hz, sampling_freq_hz),\n pass_zero=True\n ) # coefficients of a finite impulse response (FIR) filter using window method # noqa\n filtered_data = lfilter(b=coeffs, a=1.0, x=data)\n return filtered_data"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\napplies a band - pass filter to the data.", "response": "def bandpass_filter(data: FLOATS_TYPE,\n sampling_freq_hz: float,\n lower_freq_hz: float,\n upper_freq_hz: float,\n numtaps: int) -> FLOATS_TYPE:\n \"\"\"\n Apply a band-pass filter to the data.\n\n Args:\n data: time series of the data\n sampling_freq_hz: sampling frequency :math:`f_s`, in Hz\n (or other consistent units)\n lower_freq_hz: filter cutoff lower frequency in Hz\n (or other consistent units)\n upper_freq_hz: filter cutoff upper frequency in Hz\n (or other consistent units)\n numtaps: number of filter taps\n\n Returns:\n filtered data\n\n Note: number of filter taps = filter order + 1\n \"\"\"\n f1 = normalized_frequency(lower_freq_hz, sampling_freq_hz)\n f2 = normalized_frequency(upper_freq_hz, sampling_freq_hz)\n coeffs = firwin(\n numtaps=numtaps,\n cutoff=[f1, f2],\n pass_zero=False\n )\n filtered_data = lfilter(b=coeffs, a=1.0, x=data)\n return filtered_data"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndesigns and use a notch filter to filter the data.", "response": "def notch_filter(data: FLOATS_TYPE,\n sampling_freq_hz: float,\n notch_freq_hz: float,\n quality_factor: float) -> FLOATS_TYPE:\n \"\"\"\n Design and use a notch (band reject) filter to filter the data.\n\n Args:\n data: time series of the data\n sampling_freq_hz: sampling frequency :math:`f_s`, in Hz\n (or other consistent units)\n notch_freq_hz: notch frequency, in Hz\n (or other consistent units)\n quality_factor: notch filter quality factor, :math:`Q`\n\n Returns:\n filtered data\n \"\"\"\n b, a = iirnotch(\n w0=normalized_frequency(notch_freq_hz, sampling_freq_hz),\n Q=quality_factor\n )\n filtered_data = lfilter(b=b, a=a, x=data)\n return filtered_data"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rotation(angle):\n return N.array([[N.cos(angle),-N.sin(angle),0],\n [N.sin(angle), N.cos(angle),0],\n [0 , 0 ,1]])", "response": "Rotation about the Z axis"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef ellipse(center,covariance_matrix,level=1, n=1000):\n # singular value decomposition\n U, s, rotation_matrix = N.linalg.svd(covariance_matrix)\n # semi-axes (largest first)\n\n saxes = N.sqrt(s)*level ## If the _area_ of a 2s ellipse is twice that of a 1s ellipse\n # If the _axes_ are supposed to be twice as long, then it should be N.sqrt(s)*width\n\n u = N.linspace(0, 2*N.pi, n)\n data = N.column_stack((saxes[0]*N.cos(u), saxes[1]*N.sin(u)))\n # rotate data\n return N.dot(data, rotation_matrix)+ center", "response": "Returns error ellipse in slope - azimuth space"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef to_mapping(self,**values):\n strike, dip, rake = self.strike_dip_rake()\n min, max = self.angular_errors()\n\n try:\n disabled = self.disabled\n except AttributeError:\n disabled = False\n\n mapping = dict(\n uid=self.hash,\n axes=self.axes.tolist(),\n hyperbolic_axes=self.hyperbolic_axes.tolist(),\n max_angular_error=max,\n min_angular_error=min,\n strike=strike,\n dip=dip,\n rake=rake,\n disabled=disabled)\n\n # Add in user-provided-values, overwriting if\n # necessary\n for k,v in values.items():\n mapping[k] = v\n return mapping", "response": "Create a JSON - serializable representation of the current object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nkill children of this process that were registered in the .", "response": "def kill_child_processes() -> None:\n \"\"\"\n Kills children of this process that were registered in the\n :data:`processes` variable.\n\n Use with ``@atexit.register``.\n \"\"\"\n timeout_sec = 5\n for p in processes:\n try:\n p.wait(timeout_sec)\n except TimeoutExpired:\n # failed to close\n p.kill()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef start_process(args: List[str],\n stdin: Any = None,\n stdout: Any = None,\n stderr: Any = None) -> Popen:\n \"\"\"\n Launch a child process and record it in our :data:`processes` variable.\n\n Args:\n args: program and its arguments, as a list\n stdin: typically None\n stdout: use None to perform no routing, which preserves console colour!\n Otherwise, specify somewhere to route stdout. See subprocess\n documentation. If either is PIPE, you'll need to deal with the\n output.\n stderr: As above. You can use stderr=STDOUT to route stderr to the same\n place as stdout.\n\n Returns:\n The process object (which is also stored in :data:`processes`).\n \"\"\"\n log.debug(\"{!r}\", args)\n global processes\n global proc_args_list\n proc = Popen(args, stdin=stdin, stdout=stdout, stderr=stderr)\n # proc = Popen(args, stdin=None, stdout=PIPE, stderr=STDOUT)\n # proc = Popen(args, stdin=None, stdout=PIPE, stderr=PIPE)\n # Can't preserve colour: http://stackoverflow.com/questions/13299550/preserve-colored-output-from-python-os-popen # noqa\n processes.append(proc)\n proc_args_list.append(args)\n return proc", "response": "Launch a child process and record it in our processes variable."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef wait_for_processes(die_on_failure: bool = True,\n timeout_sec: float = 1) -> None:\n \"\"\"\n Wait for child processes (catalogued in :data:`processes`) to finish.\n\n If ``die_on_failure`` is ``True``, then whenever a subprocess returns\n failure, all are killed.\n\n If ``timeout_sec`` is None, the function waits for its first process to\n complete, then waits for the second, etc. So a subprocess dying does not\n trigger a full quit instantly (or potentially for ages).\n\n If ``timeout_sec`` is something else, each process is tried for that time;\n if it quits within that time, well and good (successful quit -> continue\n waiting for the others; failure -> kill everything, if ``die_on_failure``);\n if it doesn't, we try the next. That is much more responsive.\n\n \"\"\"\n global processes\n global proc_args_list\n n = len(processes)\n Pool(n).map(print_lines, processes) # in case of PIPE\n something_running = True\n while something_running:\n something_running = False\n for i, p in enumerate(processes):\n try:\n retcode = p.wait(timeout=timeout_sec)\n if retcode == 0:\n log.info(\"Process #{} (of {}) exited cleanly\", i, n)\n if retcode != 0:\n log.critical(\n \"Process #{} (of {}) exited with return code {} \"\n \"(indicating failure); its args were: {!r}\",\n i, n, retcode, proc_args_list[i])\n if die_on_failure:\n log.critical(\"Exiting top-level process (will kill \"\n \"all other children)\")\n fail() # exit this process, therefore kill its children # noqa\n except TimeoutExpired:\n something_running = True\n processes.clear()\n proc_args_list.clear()", "response": "Wait for all processes to finish and kill them."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef print_lines(process: Popen) -> None:\n out, err = process.communicate()\n if out:\n for line in out.decode(\"utf-8\").splitlines():\n print(line)\n if err:\n for line in err.decode(\"utf-8\").splitlines():\n print(line)", "response": "Print lines from a subprocess."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfiring up multiple processes and wait for them to finish.", "response": "def run_multiple_processes(args_list: List[List[str]],\n die_on_failure: bool = True) -> None:\n \"\"\"\n Fire up multiple processes, and wait for them to finihs.\n\n Args:\n args_list: command arguments for each process\n die_on_failure: see :func:`wait_for_processes`\n \"\"\"\n for procargs in args_list:\n start_process(procargs)\n # Wait for them all to finish\n wait_for_processes(die_on_failure=die_on_failure)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef run(self) -> None:\n fd = self._fd\n encoding = self._encoding\n line_terminators = self._line_terminators\n queue = self._queue\n buf = \"\"\n while True:\n try:\n c = fd.read(1).decode(encoding)\n except UnicodeDecodeError as e:\n log.warning(\"Decoding error from {!r}: {!r}\", self._cmdargs, e)\n if self._suppress_decoding_errors:\n continue\n else:\n raise\n # log.critical(\"c={!r}, returncode={!r}\", c, p.returncode)\n if not c:\n # Subprocess has finished\n return\n buf += c\n # log.critical(\"buf={!r}\", buf)\n # noinspection PyTypeChecker\n for t in line_terminators:\n try:\n t_idx = buf.index(t) + len(t) # include terminator\n fragment = buf[:t_idx]\n buf = buf[t_idx:]\n queue.put(fragment)\n except ValueError:\n pass", "response": "Read lines and put them on the queue."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating national US and State Map and add it to the database.", "response": "def create_nation_fixtures(self):\n \"\"\"\n Create national US and State Map\n \"\"\"\n SHP_SLUG = \"cb_{}_us_state_500k\".format(self.YEAR)\n DOWNLOAD_PATH = os.path.join(self.DOWNLOAD_DIRECTORY, SHP_SLUG)\n\n shape = shapefile.Reader(\n os.path.join(DOWNLOAD_PATH, \"{}.shp\".format(SHP_SLUG))\n )\n fields = shape.fields[1:]\n field_names = [f[0] for f in fields]\n features = []\n for shp in shape.shapeRecords():\n state = dict(zip(field_names, shp.record))\n geodata = {\n \"type\": \"Feature\",\n \"geometry\": shp.shape.__geo_interface__,\n \"properties\": {\n \"state\": state[\"STATEFP\"],\n \"name\": state[\"NAME\"],\n },\n }\n features.append(geodata)\n Geometry.objects.update_or_create(\n division=self.NATION,\n subdivision_level=self.STATE_LEVEL,\n simplification=self.THRESHOLDS[\"nation\"],\n source=os.path.join(\n self.SHP_SOURCE_BASE.format(self.YEAR), SHP_SLUG\n )\n + \".zip\",\n series=self.YEAR,\n defaults={\n \"topojson\": self.toposimplify(\n geojson.FeatureCollection(features),\n self.THRESHOLDS[\"nation\"],\n )\n },\n )\n\n geo, created = Geometry.objects.update_or_create(\n division=self.NATION,\n subdivision_level=self.COUNTY_LEVEL,\n simplification=self.THRESHOLDS[\"nation\"],\n source=os.path.join(\n self.SHP_SOURCE_BASE.format(self.YEAR), SHP_SLUG\n )\n + \".zip\",\n series=self.YEAR,\n defaults={\"topojson\": self.get_state_county_shps(\"00\")},\n )\n tqdm.write(\"Nation\\n\")\n tqdm.write(\n self.TQDM_PREFIX\n + \"> FIPS {} @ ~{}kb \".format(\n \"00\", round(len(json.dumps(geo.topojson)) / 1000)\n )\n )\n tqdm.write(self.style.SUCCESS(\"Done.\\n\"))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nserializes an orientation object to a dict suitable for JSON", "response": "def serialize(pca, **kwargs):\n \"\"\"\n Serialize an orientation object to a dict suitable\n for JSON\n \"\"\"\n strike, dip, rake = pca.strike_dip_rake()\n hyp_axes = sampling_axes(pca)\n\n return dict(\n **kwargs,\n principal_axes = pca.axes.tolist(),\n hyperbolic_axes = hyp_axes.tolist(),\n n_samples = pca.n,\n strike=strike, dip=dip, rake=rake,\n angular_errors=[2*N.degrees(i)\n for i in angular_errors(hyp_axes)])"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning an InsertOnDuplicate object.", "response": "def insert_on_duplicate(tablename: str,\n values: Any = None,\n inline: bool = False,\n **kwargs):\n \"\"\"\n Command to produce an :class:`InsertOnDuplicate` object.\n\n Args:\n tablename: name of the table\n values: values to ``INSERT``\n inline: as per \n http://docs.sqlalchemy.org/en/latest/core/dml.html#sqlalchemy.sql.expression.insert\n kwargs: additional parameters\n\n Returns:\n an :class:`InsertOnDuplicate` object\n\n \"\"\" # noqa\n return InsertOnDuplicate(tablename, values, inline=inline, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncompiling an INSERT statement that is used for inserting on duplicate key update.", "response": "def compile_insert_on_duplicate_key_update(insert: Insert,\n compiler: SQLCompiler,\n **kw) -> str:\n \"\"\"\n Hooks into the use of the :class:`InsertOnDuplicate` class\n for the MySQL dialect. Compiles the relevant SQL for an ``INSERT...\n ON DUPLICATE KEY UPDATE`` statement.\n\n Notes:\n \n - We can't get the fieldnames directly from ``insert`` or ``compiler``.\n - We could rewrite the innards of the visit_insert statement\n (https://github.com/bedwards/sqlalchemy_mysql_ext/blob/master/duplicate.py)... \n but, like that, it will get outdated.\n - We could use a hack-in-by-hand method\n (http://stackoverflow.com/questions/6611563/sqlalchemy-on-duplicate-key-update)\n ... but a little automation would be nice.\n - So, regex to the rescue.\n - NOTE THAT COLUMNS ARE ALREADY QUOTED by this stage; no need to repeat.\n \"\"\" # noqa\n # log.critical(compiler.__dict__)\n # log.critical(compiler.dialect.__dict__)\n # log.critical(insert.__dict__)\n s = compiler.visit_insert(insert, **kw)\n # log.critical(s)\n m = RE_INSERT_FIELDNAMES.match(s)\n if m is None:\n raise ValueError(\"compile_insert_on_duplicate_key_update: no match\")\n columns = [c.strip() for c in m.group('columns').split(\",\")]\n # log.critical(columns)\n updates = \", \".join(\n [\"{c} = VALUES({c})\".format(c=c) for c in columns])\n s += ' ON DUPLICATE KEY UPDATE {}'.format(updates)\n # log.critical(s)\n return s"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_groups(orientations, *groups, **kwargs):\n grouped = []\n # Copy all datasets to be safe (this could be bad for\n # memory usage, so can be disabled).\n if kwargs.pop('copy', True):\n orientations = [copy(o) for o in orientations]\n\n for o in orientations:\n # Get rid of and recreate group membership\n o.member_of = None\n try:\n grouped += o.members\n for a in o.members:\n a.member_of = o\n except AttributeError:\n pass\n\n def find(uid):\n try:\n val = next(x for x in orientations if x.hash == uid)\n if val in grouped:\n raise GroupedPlaneError(\"{} is already in a group.\"\n .format(val.hash))\n return val\n except StopIteration:\n raise KeyError(\"No measurement of with hash {} found\"\n .format(uid))\n\n for uid_list in groups:\n vals = [find(uid) for uid in uid_list]\n o = GroupedOrientation(*vals, **kwargs)\n orientations.append(o)\n\n return orientations", "response": "Create groups of an orientation measurement dataset."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef merge_csv(filenames: List[str],\n outfile: TextIO = sys.stdout,\n input_dialect: str = 'excel',\n output_dialect: str = 'excel',\n debug: bool = False,\n headers: bool = True) -> None:\n \"\"\"\n Amalgamate multiple CSV/TSV/similar files into one.\n\n Args:\n filenames: list of filenames to process\n outfile: file-like object to write output to\n input_dialect: dialect of input files, as passed to ``csv.reader``\n output_dialect: dialect to write, as passed to ``csv.writer``\n debug: be verbose?\n headers: do the files have header lines?\n \"\"\"\n writer = csv.writer(outfile, dialect=output_dialect)\n written_header = False\n header_items = [] # type: List[str]\n for filename in filenames:\n log.info(\"Processing file \" + repr(filename))\n with open(filename, 'r') as f:\n reader = csv.reader(f, dialect=input_dialect)\n if headers:\n if not written_header:\n header_items = next(reader)\n if debug:\n log.debug(\"Header row: {!r}\", header_items)\n writer.writerow(header_items)\n written_header = True\n else:\n new_headers = next(reader)\n if new_headers != header_items:\n raise ValueError(\n \"Header line in file {filename} doesn't match - \"\n \"it was {new} but previous was {old}\".format(\n filename=repr(filename),\n new=repr(new_headers),\n old=repr(header_items),\n ))\n if debug:\n log.debug(\"Header row matches previous\")\n else:\n if debug:\n log.debug(\"No headers in use\")\n for row in reader:\n if debug:\n log.debug(\"Data row: {!r}\", row)\n writer.writerow(row)", "response": "A function that takes a list of filenames and merges them into one."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef main():\n main_only_quicksetup_rootlogger()\n\n parser = argparse.ArgumentParser()\n parser.add_argument(\n \"filenames\",\n nargs=\"+\",\n help=\"Names of CSV/TSV files to merge\"\n )\n parser.add_argument(\n \"--outfile\",\n default=\"-\",\n help=\"Specify an output filename. If omitted or '-', stdout is used.\",\n )\n parser.add_argument(\n \"--inputdialect\",\n default=\"excel\",\n help=\"The input files' CSV/TSV dialect. Default: %(default)s.\",\n choices=csv.list_dialects(),\n )\n parser.add_argument(\n \"--outputdialect\",\n default=\"excel\",\n help=\"The output file's CSV/TSV dialect. Default: %(default)s.\",\n choices=csv.list_dialects(),\n )\n parser.add_argument(\n \"--noheaders\",\n action=\"store_true\",\n help=\"By default, files are assumed to have column headers. \"\n \"Specify this option to assume no headers.\",\n )\n parser.add_argument(\n \"--debug\",\n action=\"store_true\",\n help=\"Verbose debugging output.\",\n )\n progargs = parser.parse_args()\n\n kwargs = {\n \"filenames\": progargs.filenames,\n \"input_dialect\": progargs.inputdialect,\n \"output_dialect\": progargs.outputdialect,\n \"debug\": progargs.debug,\n \"headers\": not progargs.noheaders,\n }\n if progargs.outfile == '-':\n log.info(\"Writing to stdout\")\n merge_csv(outfile=sys.stdout, **kwargs)\n else:\n log.info(\"Writing to \" + repr(progargs.outfile))\n with open(progargs.outfile, 'w') as outfile:\n # noinspection PyTypeChecker\n merge_csv(outfile=outfile, **kwargs)", "response": "Command - line processor for the merge_csv command."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef softmax(x: np.ndarray,\n b: float = 1.0) -> np.ndarray:\n r\"\"\"\n Standard softmax function:\n\n .. math::\n\n P_i = \\frac {e ^ {\\beta \\cdot x_i}} { \\sum_{i}{\\beta \\cdot x_i} }\n\n Args:\n x: vector (``numpy.array``) of values\n b: exploration parameter :math:`\\beta`, or inverse temperature\n [Daw2009], or :math:`1/t`; see below\n\n Returns:\n vector of probabilities corresponding to the input values\n\n where:\n\n - :math:`t` is temperature (towards infinity: all actions equally likely;\n towards zero: probability of action with highest value tends to 1)\n - Temperature is not used directly as optimizers may take it to zero,\n giving an infinity; use inverse temperature instead.\n - [Daw2009] Daw ND, \"Trial-by-trial data analysis using computational\n methods\", 2009/2011; in \"Decision Making, Affect, and Learning: Attention\n and Performance XXIII\"; Delgado MR, Phelps EA, Robbins TW (eds),\n Oxford University Press.\n\n \"\"\"\n constant = np.mean(x)\n products = x * b - constant\n # ... softmax is invariant to addition of a constant: Daw article and\n # http://www.faqs.org/faqs/ai-faq/neural-nets/part2/section-12.html#b\n # noinspection PyUnresolvedReferences\n if products.max() > sys.float_info.max_exp:\n # ... max_exp for base e; max_10_exp for base 10\n log.warning(\"OVERFLOW in softmax(): x = {}, b = {}, constant = {}, \"\n \"x*b - constant = {}\".format(x, b, constant, products))\n # map the maximum to 1, other things to zero\n n = len(x)\n index_of_max = np.argmax(products)\n answer = np.zeros(n)\n answer[index_of_max] = 1.0\n else:\n # noinspection PyUnresolvedReferences\n exponented = np.exp(products)\n answer = exponented / np.sum(exponented)\n return answer", "response": "r Softmax function for the base 10 of the base 10 of the base 10 of the base 10."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef inv_logistic(y: Union[float, np.ndarray],\n k: float,\n theta: float) -> Optional[float]:\n r\"\"\"\n Inverse standard logistic function:\n\n .. math::\n\n x = ( log( \\frac {1} {y} - 1) / -k ) + \\theta\n\n Args:\n y: :math:`y`\n k: :math:`k`\n theta: :math:`\\theta`\n\n Returns:\n :math:`x`\n\n \"\"\"\n if y is None or k is None or theta is None:\n return None\n # noinspection PyUnresolvedReferences\n return (np.log((1 / y) - 1) / -k) + theta", "response": "r Inverse standard logistic function."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nbuilding a RTree index from a GeoJSON file.", "response": "def _build_from_geojson(self, src):\n \"\"\" Build a RTree index to disk using bounding box of each feature \"\"\"\n geojson = json.loads(self.read(src))\n idx = index.Index()\n data_store = {}\n for i, feature in enumerate(geojson['features']):\n feature = self._build_obj(feature)\n idx.insert(i, feature['geometry'].bounds)\n data_store[i] = feature\n return data_store, idx"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get(self, point, buffer_size=0, multiple=False):\n lon, lat = point\n geohash = Geohash.encode(lat, lon, precision=self.precision)\n key = (geohash, buffer_size, multiple)\n if key in self.geohash_cache:\n # cache hit on geohash\n self.hit += 1\n #print self.hit, self.miss\n return self.geohash_cache[key]\n self.miss += 1\n # cache miss on geohash\n # project point to ESRI:102005\n lat, lon = Geohash.decode(geohash)\n proj_point = project([float(lon), float(lat)])\n args = dict(buffer_size=buffer_size, multiple=multiple)\n payload = self.get_object(proj_point, **args)\n self.geohash_cache[key] = payload\n return payload", "response": "get a single object from the Tweet"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef run(in_file_nose, out_dir_unitth):\n suites = Converter.read_nose(in_file_nose)\n Converter.write_unitth(suites, out_dir_unitth)", "response": "Convert nose - style test reports to UnitTH - style test reports by splitting modules into separate XML files and saving them to a UnitTH - style test reports."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef read_nose(in_file):\n suites = {}\n doc_xml = minidom.parse(in_file)\n suite_xml = doc_xml.getElementsByTagName(\"testsuite\")[0]\n for case_xml in suite_xml.getElementsByTagName('testcase'):\n classname = case_xml.getAttribute('classname')\n if classname not in suites:\n suites[classname] = []\n case = {\n 'name': case_xml.getAttribute('name'),\n 'time': float(case_xml.getAttribute('time')),\n }\n \n skipped_xml = case_xml.getElementsByTagName('skipped')\n if skipped_xml:\n if skipped_xml[0].hasAttribute('type'):\n type = skipped_xml[0].getAttribute('type')\n else:\n type = ''\n case['skipped'] = {\n 'type': type,\n 'message': skipped_xml[0].getAttribute('message'),\n 'text': \"\".join([child.nodeValue for child in skipped_xml[0].childNodes]),\n }\n\n failure_xml = case_xml.getElementsByTagName('failure')\n if failure_xml:\n if failure_xml[0].hasAttribute('type'):\n type = failure_xml[0].getAttribute('type')\n else:\n type = ''\n case['failure'] = {\n 'type': type,\n 'message': failure_xml[0].getAttribute('message'),\n 'text': \"\".join([child.nodeValue for child in failure_xml[0].childNodes]),\n }\n\n error_xml = case_xml.getElementsByTagName('error')\n if error_xml:\n if error_xml[0].hasAttribute('type'):\n type = error_xml[0].getAttribute('type')\n else:\n type = ''\n case['error'] = {\n 'type': type,\n 'message': error_xml[0].getAttribute('message'),\n 'text': \"\".join([child.nodeValue for child in error_xml[0].childNodes]),\n }\n\n suites[classname].append(case)\n\n return suites", "response": "Parse a nose - style test report into a dict of test suites."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef write_unitth(suites, out_dir):\n if not os.path.isdir(out_dir):\n os.mkdir(out_dir)\n\n for classname, cases in suites.items():\n doc_xml = minidom.Document()\n\n suite_xml = doc_xml.createElement('testsuite')\n suite_xml.setAttribute('name', classname)\n suite_xml.setAttribute('tests', str(len(cases)))\n suite_xml.setAttribute('errors', str(sum('error' in case for case in cases)))\n suite_xml.setAttribute('failures', str(sum('failure' in case for case in cases)))\n suite_xml.setAttribute('skipped', str(sum('skipped' in case for case in cases)))\n suite_xml.setAttribute('time', '{:.3f}'.format(sum(case['time'] for case in cases)))\n doc_xml.appendChild(suite_xml)\n\n for case in cases:\n case_xml = doc_xml.createElement('testcase')\n case_xml.setAttribute('classname', classname)\n case_xml.setAttribute('name', case['name'])\n case_xml.setAttribute('time', '{:.3f}'.format(case['time']))\n suite_xml.appendChild(case_xml)\n\n if 'skipped' in case:\n skipped_xml = doc_xml.createElement('skipped')\n skipped_xml.setAttribute('type', case['skipped']['type'])\n skipped_xml.setAttribute('message', case['skipped']['message'])\n case_xml.appendChild(skipped_xml)\n\n skipped_text_xml = doc_xml.createCDATASection(case['skipped']['text'])\n skipped_xml.appendChild(skipped_text_xml)\n\n if 'failure' in case:\n failure_xml = doc_xml.createElement('failure')\n failure_xml.setAttribute('type', case['failure']['type'])\n failure_xml.setAttribute('message', case['failure']['message'])\n case_xml.appendChild(failure_xml)\n\n failure_text_xml = doc_xml.createCDATASection(case['failure']['text'])\n failure_xml.appendChild(failure_text_xml)\n\n if 'error' in case:\n error_xml = doc_xml.createElement('error')\n error_xml.setAttribute('type', case['error']['type'])\n error_xml.setAttribute('message', case['error']['message'])\n case_xml.appendChild(error_xml)\n\n error_text_xml = doc_xml.createCDATASection(case['error']['text'])\n error_xml.appendChild(error_text_xml)\n\n with open(os.path.join(out_dir, '{}.xml'.format(classname)), 'w') as output:\n doc_xml.writexml(output, encoding='utf-8', addindent='', newl=\"\")\n doc_xml.unlink()", "response": "Writes UnitTH - style test reports for a set of test suites."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nplotting asymptotic error bounds for a single hyperbola on a stereonet.", "response": "def error_asymptotes(pca,**kwargs):\n \"\"\"\n Plots asymptotic error bounds for\n hyperbola on a stereonet.\n \"\"\"\n ax = kwargs.pop(\"ax\",current_axes())\n\n lon,lat = pca.plane_errors('upper', n=1000)\n ax.plot(lon,lat,'-')\n\n lon,lat = pca.plane_errors('lower', n=1000)\n ax.plot(lon,lat,'-')\n\n ax.plane(*pca.strike_dip())"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the rows and column names from a raw SQL query.", "response": "def get_rows_fieldnames_from_raw_sql(\n session: Union[Session, Engine, Connection],\n sql: str) -> Tuple[Sequence[Sequence[Any]], Sequence[str]]:\n \"\"\"\n Returns results and column names from a query.\n\n Args:\n session: SQLAlchemy :class:`Session`, :class:`Engine`, or\n :class:`Connection` object\n sql: raw SQL to execure\n\n Returns:\n ``(rows, fieldnames)`` where ``rows`` is the usual set of results and\n ``fieldnames`` are the name of the result columns/fields.\n\n \"\"\"\n result = session.execute(sql) # type: ResultProxy\n fieldnames = result.keys()\n rows = result.fetchall()\n return rows, fieldnames"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef count_star(session: Union[Session, Engine, Connection],\n tablename: str,\n *criteria: Any) -> int:\n \"\"\"\n Returns the result of ``COUNT(*)`` from the specified table (with\n additional ``WHERE`` criteria if desired).\n\n Args:\n session: SQLAlchemy :class:`Session`, :class:`Engine`, or\n :class:`Connection` object\n tablename: name of the table\n criteria: optional SQLAlchemy \"where\" criteria\n\n Returns:\n a scalar\n \"\"\"\n # works if you pass a connection or a session or an engine; all have\n # the execute() method\n query = select([func.count()]).select_from(table(tablename))\n for criterion in criteria:\n query = query.where(criterion)\n return session.execute(query).scalar()", "response": "Returns the result of COUNT(*) from the specified table."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef exists_in_table(session: Session, table_: Table, *criteria: Any) -> bool:\n exists_clause = exists().select_from(table_)\n # ... EXISTS (SELECT * FROM tablename)\n for criterion in criteria:\n exists_clause = exists_clause.where(criterion)\n # ... EXISTS (SELECT * FROM tablename WHERE ...)\n\n if session.get_bind().dialect.name == SqlaDialectName.MSSQL:\n query = select([literal(True)]).where(exists_clause)\n # ... SELECT 1 WHERE EXISTS (SELECT * FROM tablename WHERE ...)\n else:\n query = select([exists_clause])\n # ... SELECT EXISTS (SELECT * FROM tablename WHERE ...)\n\n result = session.execute(query).scalar()\n return bool(result)", "response": "Returns a boolean indicating if a record or records exist in the specified table."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef exists_plain(session: Session, tablename: str, *criteria: Any) -> bool:\n return exists_in_table(session, table(tablename), *criteria)", "response": "Returns a boolean array that contains a record or records exist in a table."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fetch_all_first_values(session: Session,\n select_statement: Select) -> List[Any]:\n \"\"\"\n Returns a list of the first values in each row returned by a ``SELECT``\n query.\n\n A Core version of this sort of thing:\n http://xion.io/post/code/sqlalchemy-query-values.html\n\n Args:\n session: SQLAlchemy :class:`Session` object\n select_statement: SQLAlchemy :class:`Select` object\n\n Returns:\n a list of the first value of each result row\n\n \"\"\"\n rows = session.execute(select_statement) # type: ResultProxy\n try:\n return [row[0] for row in rows]\n except ValueError as e:\n raise MultipleResultsFound(str(e))", "response": "Returns a list of the first values in each row returned by a SELECT statement."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a hyperbola that opens along y axis", "response": "def hyperbola(axes, **kwargs):\n \"\"\"\n Plots a hyperbola that opens along y axis\n \"\"\"\n opens_up = kwargs.pop('opens_up', True)\n center = kwargs.pop('center', defaults['center'])\n th = N.linspace(0,2*N.pi,kwargs.pop('n', 500))\n\n vals = [N.tan(th),1/N.cos(th)]\n if not opens_up:\n vals = vals[::-1]\n\n x = axes[0]*vals[0]+center[0]\n y = axes[1]*vals[1]+center[1]\n\n extrema = [N.argmin(x),N.argmax(x)]\n\n def remove_asymptotes(arr):\n arr[extrema] = N.nan\n return arr\n\n xy = tuple(remove_asymptotes(i) for i in (x,y))\n\n return xy"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef __reverse_ellipse(axes, scalar=1):\n ax1 = axes.copy()[::-1]*scalar\n center = ax1[1]*N.sqrt(2)*scalar\n return ax1, center", "response": "This method returns the first axis and the center of the ellipse."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndisables constraints checking for the specified table while the resource is held.", "response": "def if_sqlserver_disable_constraints(session: SqlASession,\n tablename: str) -> None:\n \"\"\"\n If we're running under SQL Server, disable constraint checking for the\n specified table while the resource is held.\n\n Args:\n session: SQLAlchemy :class:`Session`\n tablename: table name\n\n See\n https://stackoverflow.com/questions/123558/sql-server-2005-t-sql-to-temporarily-disable-a-trigger\n \"\"\" # noqa\n engine = get_engine_from_session(session)\n if is_sqlserver(engine):\n quoted_tablename = quote_identifier(tablename, engine)\n session.execute(\n \"ALTER TABLE {} NOCHECK CONSTRAINT all\".format(\n quoted_tablename))\n yield\n session.execute(\n \"ALTER TABLE {} WITH CHECK CHECK CONSTRAINT all\".format(\n quoted_tablename))\n else:\n yield"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef if_sqlserver_disable_constraints_triggers(session: SqlASession,\n tablename: str) -> None:\n \"\"\"\n If we're running under SQL Server, disable triggers AND constraints for the\n specified table while the resource is held.\n\n Args:\n session: SQLAlchemy :class:`Session`\n tablename: table name\n \"\"\"\n with if_sqlserver_disable_constraints(session, tablename):\n with if_sqlserver_disable_triggers(session, tablename):\n yield", "response": "Context manager for SQL Server disable triggers AND constraints for the\nTaxonomy specified table while the resource is held."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef expand_action(data):\n # when given a string, assume user wants to index raw json\n if isinstance(data, string_types):\n return '{\"index\": {}}', data\n\n # make sure we don't alter the action\n data = data.copy()\n op_type = data.pop(EsBulk.OP_TYPE, EsBulk.INDEX)\n\n action = ActionParser._get_relevant_action_params(data, op_type)\n\n # no data payload for delete\n if op_type == EsBulk.DELETE:\n return action, None\n\n return action, data.get(EsDocProperties.SOURCE, data)", "response": "Expand the action to bulk format"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nstream bulk consumes actions from the iterable passed in and return the results of all bulk data :func:`~elasticsearch.helpers.bulk` which is a wrapper around streaming bulk that returns summary information about the bulk operation once the entire input is consumed and sent. :arg actions: iterable containing the actions to be executed :arg chunk_size: number of docs in one chunk sent to es (default: 500) :arg max_chunk_bytes: the maximum size of the request in bytes (default: 100MB) :arg raise_on_error: raise ``BulkIndexError`` containing errors (as `.errors`) from the execution of the last chunk when some occur. By default we raise. :arg raise_on_exception: if ``False`` then don't propagate exceptions from call to ``bulk`` and just report the items that failed as failed. :arg expand_action_callback: callback executed on each action passed in, should return a tuple containing the action line and the data line (`None` if data line should be omitted).", "response": "def streaming_bulk(self, actions, chunk_size=500, max_chunk_bytes=100 * 1024 * 1024,\n raise_on_error=True, expand_action_callback=ActionParser.expand_action,\n raise_on_exception=True, **kwargs):\n \"\"\"\n Streaming bulk consumes actions from the iterable passed in and return the results of all bulk data\n :func:`~elasticsearch.helpers.bulk` which is a wrapper around streaming\n bulk that returns summary information about the bulk operation once the\n entire input is consumed and sent.\n :arg actions: iterable containing the actions to be executed\n :arg chunk_size: number of docs in one chunk sent to es (default: 500)\n :arg max_chunk_bytes: the maximum size of the request in bytes (default: 100MB)\n :arg raise_on_error: raise ``BulkIndexError`` containing errors (as `.errors`)\n from the execution of the last chunk when some occur. By default we raise.\n :arg raise_on_exception: if ``False`` then don't propagate exceptions from\n call to ``bulk`` and just report the items that failed as failed.\n :arg expand_action_callback: callback executed on each action passed in,\n should return a tuple containing the action line and the data line\n (`None` if data line should be omitted).\n \"\"\"\n actions = list(map(expand_action_callback, actions))\n\n for bulk_actions in self._chunk_actions(actions, chunk_size, max_chunk_bytes):\n yield self._process_bulk_chunk(bulk_actions, raise_on_exception, raise_on_error, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsending a bulk request to elasticsearch and process the response.", "response": "def _process_bulk_chunk(self, bulk_actions, raise_on_exception=True, raise_on_error=True, **kwargs):\n \"\"\"\n Send a bulk request to elasticsearch and process the output.\n \"\"\"\n # if raise on error is set, we need to collect errors per chunk before\n # raising them\n\n resp = None\n try:\n # send the actual request\n actions = \"{}\\n\".format('\\n'.join(bulk_actions))\n resp = yield self.client.bulk(actions, **kwargs)\n except ConnectionTimeout as e:\n # default behavior - just propagate exception\n if raise_on_exception:\n raise\n\n self._handle_transport_error(bulk_actions, e, raise_on_error)\n returnValue([])\n\n # go through request-response pairs and detect failures\n errors = []\n results = []\n for op_type, item in map(methodcaller('popitem'), resp['items']):\n ok = 200 <= item.get('status', 500) < 300\n if not ok and raise_on_error:\n errors.append({op_type: item})\n\n if ok or not errors:\n # if we are not just recording all errors to be able to raise\n # them all at once, yield items individually\n results.append((ok, {op_type: item}))\n\n if errors:\n msg_fmt = '{num} document(s) failed to index.'\n raise BulkIndexError(msg_fmt.format(num=len(errors)), errors)\n else:\n returnValue(results)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef displayhtml(site_key,\n language='',\n theme='light',\n fallback=False,\n d_type='image',\n size='normal'):\n \"\"\"\n Gets the HTML to display for reCAPTCHA\n\n site_key -- The site key\n language -- The language code for the widget.\n theme -- The color theme of the widget. `light` or `dark`\n fallback -- Old version recaptcha.\n d_type -- The type of CAPTCHA to serve. `image` or `audio`\n size -- The size of the dispalyed CAPTCHA, 'normal' or 'compact'\n\n For more detail, refer to:\n - https://developers.google.com/recaptcha/docs/display\n \"\"\"\n\n return \"\"\"\n<script\n src=\"https://www.google.com/recaptcha/api.js?hl=%(LanguageCode)s&fallback=%(Fallback)s&\"\n async=\"async\" defer=\"defer\"></script>\n<div class=\"g-recaptcha\"\n data-sitekey=\"%(SiteKey)s\"\n data-theme=\"%(Theme)s\"\n data-type=\"%(Type)s\"\n data-size=\"%(Size)s\">\n</div>\n<noscript>\n <div style=\"width: 302px; height: 480px;\">\n <div style=\"width: 302px; height: 422px; position: relative;\">\n <div style=\"width: 302px; height: 422px; position: relative;\">\n <iframe\n src=\"https://www.google.com/recaptcha/api/fallback?k=%(SiteKey)s&hl=%(LanguageCode)s\"\n frameborder=\"0\" scrolling=\"no\"\n style=\"width: 302px; height:422px; border-style: none;\">\n </iframe>\n </div>\n <div\n style=\"border-style: none; bottom: 12px; left: 25px;\n margin: 0px; padding: 0px; right: 25px;\n background: #f9f9f9; border: 1px solid #c1c1c1;\n border-radius: 3px; height: 60px; width: 300px;\">\n <textarea\n id=\"g-recaptcha-response\" name=\"g-recaptcha-response\"\n class=\"g-recaptcha-response\"\n style=\"width: 250px; height: 40px; border: 1px solid #c1c1c1;\n margin: 10px 25px; padding: 0px; resize: none;\"\n value=\"\"></textarea>\n </div>\n </div>\n </div>\n</noscript>\n\"\"\" % {\n 'LanguageCode': language,\n 'SiteKey': site_key,\n 'Theme': theme,\n 'Type': d_type,\n 'Size': size,\n 'Fallback': fallback,\n }", "response": "Displays the HTML for a reCAPTCHA widget."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsubmitting a reCAPTCHA request for verification. Returns RecaptchaResponse object for the request.", "response": "def submit(recaptcha_response_field,\n secret_key,\n remoteip,\n verify_server=VERIFY_SERVER):\n \"\"\"\n Submits a reCAPTCHA request for verification. Returns RecaptchaResponse\n for the request\n\n recaptcha_response_field -- The value from the form\n secret_key -- your reCAPTCHA secret key\n remoteip -- the user's ip address\n \"\"\"\n\n if not (recaptcha_response_field and len(recaptcha_response_field)):\n return RecaptchaResponse(\n is_valid=False,\n error_code='incorrect-captcha-sol'\n )\n\n def encode_if_necessary(s):\n if isinstance(s, unicode):\n return s.encode('utf-8')\n return s\n\n params = urllib.urlencode({\n 'secret': encode_if_necessary(secret_key),\n 'remoteip': encode_if_necessary(remoteip),\n 'response': encode_if_necessary(recaptcha_response_field),\n })\n\n request = Request(\n url=\"https://%s/recaptcha/api/siteverify\" % verify_server,\n data=params,\n headers={\n \"Content-type\": \"application/x-www-form-urlencoded\",\n \"User-agent\": \"noReCAPTCHA Python\"\n }\n )\n\n httpresp = urlopen(request)\n\n return_values = json.loads(httpresp.read())\n httpresp.close()\n\n return_code = return_values['success']\n error_codes = return_values.get('error-codes', [])\n\n if return_code:\n return RecaptchaResponse(is_valid=True)\n else:\n return RecaptchaResponse(\n is_valid=False,\n error_code=error_codes\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets the head revision of the Alembic script from the Alembic config file.", "response": "def get_head_revision_from_alembic(\n alembic_config_filename: str,\n alembic_base_dir: str = None,\n version_table: str = DEFAULT_ALEMBIC_VERSION_TABLE) -> str:\n \"\"\"\n Ask Alembic what its head revision is (i.e. where the Python code would\n like the database to be at).\n\n Arguments:\n alembic_config_filename: config filename\n alembic_base_dir: directory to start in, so relative paths in the\n config file work.\n version_table: table name for Alembic versions\n \"\"\"\n if alembic_base_dir is None:\n alembic_base_dir = os.path.dirname(alembic_config_filename)\n os.chdir(alembic_base_dir) # so the directory in the config file works\n config = Config(alembic_config_filename)\n script = ScriptDirectory.from_config(config)\n with EnvironmentContext(config,\n script,\n version_table=version_table):\n return script.get_current_head()"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nask the database what its current revision is.", "response": "def get_current_revision(\n database_url: str,\n version_table: str = DEFAULT_ALEMBIC_VERSION_TABLE) -> str:\n \"\"\"\n Ask the database what its current revision is.\n\n Arguments:\n database_url: SQLAlchemy URL for the database\n version_table: table name for Alembic versions\n \"\"\"\n engine = create_engine(database_url)\n conn = engine.connect()\n opts = {'version_table': version_table}\n mig_context = MigrationContext.configure(conn, opts=opts)\n return mig_context.get_current_revision()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_current_and_head_revision(\n database_url: str,\n alembic_config_filename: str,\n alembic_base_dir: str = None,\n version_table: str = DEFAULT_ALEMBIC_VERSION_TABLE) -> Tuple[str, str]:\n \"\"\"\n Returns a tuple of ``(current_revision, head_revision)``; see\n :func:`get_current_revision` and :func:`get_head_revision_from_alembic`.\n\n Arguments:\n database_url: SQLAlchemy URL for the database\n alembic_config_filename: config filename\n alembic_base_dir: directory to start in, so relative paths in the\n config file work.\n version_table: table name for Alembic versions\n \"\"\"\n # Where we are\n head_revision = get_head_revision_from_alembic(\n alembic_config_filename=alembic_config_filename,\n alembic_base_dir=alembic_base_dir,\n version_table=version_table\n )\n log.info(\"Intended database version: {}\", head_revision)\n\n # Where we want to be\n current_revision = get_current_revision(\n database_url=database_url,\n version_table=version_table\n )\n log.info(\"Current database version: {}\", current_revision)\n\n # Are we where we want to be?\n return current_revision, head_revision", "response": "Returns a tuple of current revision and head revision from Alembic."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef upgrade_database(\n alembic_config_filename: str,\n alembic_base_dir: str = None,\n starting_revision: str = None,\n destination_revision: str = \"head\",\n version_table: str = DEFAULT_ALEMBIC_VERSION_TABLE,\n as_sql: bool = False) -> None:\n \"\"\"\n Use Alembic to upgrade our database.\n\n See http://alembic.readthedocs.org/en/latest/api/runtime.html\n but also, in particular, ``site-packages/alembic/command.py``\n\n Arguments:\n alembic_config_filename:\n config filename\n\n alembic_base_dir:\n directory to start in, so relative paths in the config file work\n\n starting_revision:\n revision to start at (typically ``None`` to ask the database)\n\n destination_revision:\n revision to aim for (typically ``\"head\"`` to migrate to the latest\n structure)\n\n version_table: table name for Alembic versions\n\n as_sql:\n run in \"offline\" mode: print the migration SQL, rather than\n modifying the database. See\n http://alembic.zzzcomputing.com/en/latest/offline.html\n\n \"\"\"\n\n if alembic_base_dir is None:\n alembic_base_dir = os.path.dirname(alembic_config_filename)\n os.chdir(alembic_base_dir) # so the directory in the config file works\n config = Config(alembic_config_filename)\n script = ScriptDirectory.from_config(config)\n\n # noinspection PyUnusedLocal,PyProtectedMember\n def upgrade(rev, context):\n return script._upgrade_revs(destination_revision, rev)\n\n log.info(\"Upgrading database to revision {!r} using Alembic\",\n destination_revision)\n\n with EnvironmentContext(config,\n script,\n fn=upgrade,\n as_sql=as_sql,\n starting_rev=starting_revision,\n destination_rev=destination_revision,\n tag=None,\n version_table=version_table):\n script.run_env()\n\n log.info(\"Database upgrade completed\")", "response": "Upgrade the database to a new revision."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef create_database_migration_numbered_style(\n alembic_ini_file: str,\n alembic_versions_dir: str,\n message: str,\n n_sequence_chars: int = 4) -> None:\n \"\"\"\n Create a new Alembic migration script.\n \n Alembic compares the **state of the database** to the **state of the\n metadata**, and generates a migration that brings the former up to the\n latter. (It does **not** compare the most recent revision to the current\n metadata, so make sure your database is up to date with the most recent\n revision before running this!)\n \n You **must check** that the autogenerated revisions are sensible.\n \n How does it know where to look for the database?\n \n 1. This function changes into the directory of the Alembic ``.ini``\n file and calls the external program\n \n .. code-block:: bash\n \n alembic -c ALEMBIC_INI_FILE revision --autogenerate -m MESSAGE --rev-id REVISION_ID\n \n 2. The Alembic ``.ini`` file points (via the ``script_location``\n variable) to a directory containing your ``env.py``. Alembic loads\n this script.\n \n 3. That script typically works out the database URL and calls further\n into the Alembic code. \n \n See http://alembic.zzzcomputing.com/en/latest/autogenerate.html.\n\n Regarding filenames: the default ``n_sequence_chars`` of 4 is like Django\n and gives files with names like\n\n .. code-block:: none\n\n 0001_x.py, 0002_y.py, ...\n\n NOTE THAT TO USE A NON-STANDARD ALEMBIC VERSION TABLE, YOU MUST SPECIFY\n THAT IN YOUR ``env.py`` (see e.g. CamCOPS).\n\n Args:\n alembic_ini_file: filename of Alembic ``alembic.ini`` file\n alembic_versions_dir: directory in which you keep your Python scripts,\n one per Alembic revision\n message: message to be associated with this revision\n n_sequence_chars: number of numerical sequence characters to use in the\n filename/revision (see above).\n \"\"\" # noqa\n _, _, existing_version_filenames = next(os.walk(alembic_versions_dir),\n (None, None, []))\n existing_version_filenames = [\n x for x in existing_version_filenames if x != \"__init__.py\"]\n log.debug(\"Existing Alembic version script filenames: {!r}\",\n existing_version_filenames)\n current_seq_strs = [x[:n_sequence_chars]\n for x in existing_version_filenames]\n current_seq_strs.sort()\n if not current_seq_strs:\n current_seq_str = None\n new_seq_no = 1\n else:\n current_seq_str = current_seq_strs[-1]\n new_seq_no = max(int(x) for x in current_seq_strs) + 1\n new_seq_str = str(new_seq_no).zfill(n_sequence_chars)\n\n log.info(\n \"\"\"\nGenerating new revision with Alembic...\n Last revision was: {}\n New revision will be: {}\n [If it fails with \"Can't locate revision identified by...\", you might need\n to DROP the Alembic version table (by default named 'alembic_version', but\n you may have elected to change that in your env.py.]\n \"\"\",\n current_seq_str,\n new_seq_str\n )\n\n alembic_ini_dir = os.path.dirname(alembic_ini_file)\n os.chdir(alembic_ini_dir)\n cmdargs = ['alembic',\n '-c', alembic_ini_file,\n 'revision',\n '--autogenerate',\n '-m', message,\n '--rev-id', new_seq_str]\n log.info(\"From directory {!r}, calling: {!r}\", alembic_ini_dir, cmdargs)\n subprocess.call(cmdargs)", "response": "Creates a migration script that will be used to generate the numbered version of the database."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nstamping the Alembic version table with the given revision.", "response": "def stamp_allowing_unusual_version_table(\n config: Config,\n revision: str,\n sql: bool = False,\n tag: str = None,\n version_table: str = DEFAULT_ALEMBIC_VERSION_TABLE) -> None:\n \"\"\"\n Stamps the Alembic version table with the given revision; don't run any\n migrations.\n\n This function is a clone of ``alembic.command.stamp()``, but allowing\n ``version_table`` to change. See\n http://alembic.zzzcomputing.com/en/latest/api/commands.html#alembic.command.stamp\n \"\"\" # noqa\n\n script = ScriptDirectory.from_config(config)\n\n starting_rev = None\n if \":\" in revision:\n if not sql:\n raise CommandError(\"Range revision not allowed\")\n starting_rev, revision = revision.split(':', 2)\n\n # noinspection PyUnusedLocal\n def do_stamp(rev: str, context):\n # noinspection PyProtectedMember\n return script._stamp_revs(revision, rev)\n\n with EnvironmentContext(config,\n script,\n fn=do_stamp,\n as_sql=sql,\n destination_rev=revision,\n starting_rev=starting_rev,\n tag=tag,\n version_table=version_table):\n script.run_env()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsetting the url_path of the current context with the given string format.", "response": "def set_url_path_with_params(context, url_path_str_format):\n \"\"\"\n Parameters:\n\n +------+--------+\n | key | value |\n +======+========+\n | key1 | value1 |\n +------+--------+\n | key2 | value2 |\n +------+--------+\n \"\"\"\n safe_add_http_request_context_to_behave_context(context)\n table_as_json = dict(context.table)\n url_path = url_path_str_format % table_as_json\n context.http_request_context.url_path = url_path"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsets the HTTP request context s headers.", "response": "def set_headers(context):\n \"\"\"\n Parameters:\n\n +--------------+---------------+\n | header_name | header_value |\n +==============+===============+\n | header1 | value1 |\n +--------------+---------------+\n | header2 | value2 |\n +--------------+---------------+\n \"\"\"\n safe_add_http_request_context_to_behave_context(context)\n headers = dict()\n for row in context.table:\n headers[row[\"header_name\"]] = row[\"header_value\"]\n context.http_request_context.headers = headers"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef set_form_parameters(context):\n safe_add_http_request_context_to_behave_context(context)\n context.http_request_context.body_params = get_parameters(context)\n context.http_request_context.renderer = FormRenderer()", "response": "Sets the parameters of the current request to be used in the form."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef set_body_files(context):\n safe_add_http_request_context_to_behave_context(context)\n files = dict()\n for row in context.table:\n files[row[\"param_name\"]] = row[\"path_to_file\"]\n context.http_request_context.files = files", "response": "Sets the body files of the current page."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef send_http_request_with_query_parameters(context, method):\n safe_add_http_request_context_to_behave_context(context)\n set_query_parameters(context)\n send_http_request(context, method)", "response": "Send a request to the server with the query parameters set in the context."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef send_http_request_with_body_parameters(context, method):\n safe_add_http_request_context_to_behave_context(context)\n set_body_parameters(context)\n send_http_request(context, method)", "response": "Send a request with the body parameters"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsending a request to the server with the form parameters.", "response": "def send_http_request_with_form_parameters(context, method):\n \"\"\"\n Parameters:\n\n +-------------+--------------+\n | param_name | param_value |\n +=============+==============+\n | param1 | value1 |\n +-------------+--------------+\n | param2 | value2 |\n +-------------+--------------+\n \"\"\"\n safe_add_http_request_context_to_behave_context(context)\n set_form_parameters(context)\n send_http_request(context, method)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsending a HTTP request with JSON parameters.", "response": "def send_http_request_with_json(context, method):\n \"\"\"\n Parameters:\n\n .. code-block:: json\n\n {\n \"param1\": \"value1\",\n \"param2\": \"value2\",\n \"param3\": {\n \"param31\": \"value31\"\n }\n }\n \"\"\"\n safe_add_http_request_context_to_behave_context(context)\n context.http_request_context.body_params = json.loads(context.text)\n context.http_request_context.renderer = JSONRenderer()\n send_http_request(context, method)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nexecute a command - line command and returns its stdout as bytes", "response": "def get_external_command_output(command: str) -> bytes:\n \"\"\"\n Takes a command-line command, executes it, and returns its ``stdout``\n output.\n\n Args:\n command: command string\n\n Returns:\n output from the command as ``bytes``\n\n \"\"\"\n args = shlex.split(command)\n ret = subprocess.check_output(args) # this needs Python 2.7 or higher\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets the output from a piped series of commands.", "response": "def get_pipe_series_output(commands: Sequence[str],\n stdinput: BinaryIO = None) -> bytes:\n \"\"\"\n Get the output from a piped series of commands.\n\n Args:\n commands: sequence of command strings\n stdinput: optional ``stdin`` data to feed into the start of the pipe\n\n Returns:\n ``stdout`` from the end of the pipe\n\n \"\"\"\n # Python arrays indexes are zero-based, i.e. an array is indexed from\n # 0 to len(array)-1.\n # The range/xrange commands, by default, start at 0 and go to one less\n # than the maximum specified.\n\n # print commands\n processes = [] # type: List[subprocess.Popen]\n for i in range(len(commands)):\n if i == 0: # first processes\n processes.append(\n subprocess.Popen(\n shlex.split(commands[i]),\n stdin=subprocess.PIPE,\n stdout=subprocess.PIPE\n )\n )\n else: # subsequent ones\n processes.append(\n subprocess.Popen(\n shlex.split(commands[i]),\n stdin=processes[i - 1].stdout,\n stdout=subprocess.PIPE\n )\n )\n return processes[len(processes) - 1].communicate(stdinput)[0]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef launch_external_file(filename: str, raise_if_fails: bool = False) -> None:\n log.info(\"Launching external file: {!r}\", filename)\n try:\n if sys.platform.startswith('linux'):\n cmdargs = [\"xdg-open\", filename]\n # log.debug(\"... command: {!r}\", cmdargs)\n subprocess.call(cmdargs)\n else:\n # log.debug(\"... with os.startfile()\")\n # noinspection PyUnresolvedReferences\n os.startfile(filename)\n except Exception as e:\n log.critical(\"Error launching {!r}: error was {}.\\n\\n{}\",\n filename, str(e), traceback.format_exc())\n if raise_if_fails:\n raise", "response": "Launches a file using the operating system s standard launcher."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef kill_proc_tree(pid: int,\n including_parent: bool = True,\n timeout_s: float = 5) \\\n -> Tuple[Set[psutil.Process], Set[psutil.Process]]:\n \"\"\"\n Kills a tree of processes, starting with the parent. Slightly modified from\n https://stackoverflow.com/questions/1230669/subprocess-deleting-child-processes-in-windows.\n \n Args:\n pid: process ID of the parent\n including_parent: kill the parent too?\n timeout_s: timeout to wait for processes to close\n\n Returns:\n tuple: ``(gone, still_alive)``, where both are sets of\n :class:`psutil.Process` objects\n\n \"\"\" # noqa\n parent = psutil.Process(pid)\n to_kill = parent.children(recursive=True) # type: List[psutil.Process]\n if including_parent:\n to_kill.append(parent)\n for proc in to_kill:\n proc.kill() # SIGKILL\n gone, still_alive = psutil.wait_procs(to_kill, timeout=timeout_s)\n return gone, still_alive", "response": "Kills a tree of processes starting with the parent."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef make_mysql_url(username: str, password: str, dbname: str,\n driver: str = \"mysqldb\", host: str = \"localhost\",\n port: int = 3306, charset: str = \"utf8\") -> str:\n \"\"\"\n Makes an SQLAlchemy URL for a MySQL database.\n \"\"\"\n return \"mysql+{driver}://{u}:{p}@{host}:{port}/{db}?charset={cs}\".format(\n driver=driver,\n host=host,\n port=port,\n db=dbname,\n u=username,\n p=password,\n cs=charset,\n )", "response": "Makes an SQLAlchemy URL for a MySQL database."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nmake an SQLAlchemy URL for a SQLite database.", "response": "def make_sqlite_url(filename: str) -> str:\n \"\"\"\n Makes an SQLAlchemy URL for a SQLite database.\n \"\"\"\n absfile = os.path.abspath(filename)\n return \"sqlite://{host}/{path}\".format(host=\"\", path=absfile)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the SQLAlchemy Engine from a SQLAlchemy Session.", "response": "def get_engine_from_session(dbsession: Session) -> Engine:\n \"\"\"\n Gets the SQLAlchemy :class:`Engine` from a SQLAlchemy :class:`Session`.\n \"\"\"\n engine = dbsession.bind\n assert isinstance(engine, Engine)\n return engine"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_safe_url_from_engine(engine: Engine) -> str:\n raw_url = engine.url # type: str\n url_obj = make_url(raw_url) # type: URL\n return repr(url_obj)", "response": "Gets a URL from an Engine obscuring the password."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef atoi(text: str) -> Union[int, str]:\n return int(text) if text.isdigit() else text", "response": "Converts a string to integers if it s composed of digits ; otherwise returns the string unchanged."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef natural_keys(text: str) -> List[Union[int, str]]:\n return [atoi(c) for c in re.split(r'(\\d+)', text)]", "response": "Sort text into string and number parts for natural sorting."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_pretty_body(headers, body):\n try:\n if CONTENT_TYPE_HEADER_NAME in headers:\n if XMLRenderer.DEFAULT_CONTENT_TYPE == headers[CONTENT_TYPE_HEADER_NAME]:\n xml_parsed = parseString(body)\n pretty_xml_as_string = xml_parsed.toprettyxml()\n return pretty_xml_as_string\n elif JSONRenderer.DEFAULT_CONTENT_TYPE == headers[CONTENT_TYPE_HEADER_NAME]:\n decoded_body = body.decode('utf-8')\n parsed = json.loads(decoded_body)\n return json.dumps(parsed, sort_keys=True, indent=4)\n except:\n pass\n finally:\n return body", "response": "Return a pretty printed body using the Content - Type header information."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef log_print_request(method, url, query_params=None, headers=None, body=None):\n\n log_msg = '\\n>>>>>>>>>>>>>>>>>>>>> Request >>>>>>>>>>>>>>>>>>> \\n'\n log_msg += '\\t> Method: %s\\n' % method\n log_msg += '\\t> Url: %s\\n' % url\n if query_params is not None:\n log_msg += '\\t> Query params: {}\\n'.format(str(query_params))\n if headers is not None:\n log_msg += '\\t> Headers:\\n{}\\n'.format(json.dumps(dict(headers), sort_keys=True, indent=4))\n if body is not None:\n try:\n log_msg += '\\t> Payload sent:\\n{}\\n'.format(_get_pretty_body(headers, body))\n except:\n log_msg += \"\\t> Payload could't be formatted\"\n\n logger.debug(log_msg)", "response": "Log an HTTP request in a user - friendly representation."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nlog an HTTP response in a user - friendly representation.", "response": "def log_print_response(status_code, response, headers=None):\n \"\"\"\n Log an HTTP response data in a user-friendly representation.\n\n :param status_code: HTTP Status Code\n :param response: Raw response content (string)\n :param headers: Headers in the response (dict)\n :return: None\n \"\"\"\n\n log_msg = '\\n<<<<<<<<<<<<<<<<<<<<<< Response <<<<<<<<<<<<<<<<<<\\n'\n log_msg += '\\t< Response code: {}\\n'.format(str(status_code))\n if headers is not None:\n log_msg += '\\t< Headers:\\n{}\\n'.format(json.dumps(dict(headers), sort_keys=True, indent=4))\n try:\n log_msg += '\\t< Payload received:\\n{}'.format(_get_pretty_body(headers, response))\n except:\n log_msg += '\\t< Payload received:\\n{}'.format(response)\n logger.debug(log_msg)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert a set of args and kwargs to an InitDict.", "response": "def args_kwargs_to_initdict(args: ArgsList, kwargs: KwargsDict) -> InitDict:\n \"\"\"\n Converts a set of ``args`` and ``kwargs`` to an ``InitDict``.\n \"\"\"\n return {ARGS_LABEL: args,\n KWARGS_LABEL: kwargs}"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef strip_leading_underscores_from_keys(d: Dict) -> Dict:\n newdict = {}\n for k, v in d.items():\n if k.startswith('_'):\n k = k[1:]\n if k in newdict:\n raise ValueError(\"Attribute conflict: _{k}, {k}\".format(k=k))\n newdict[k] = v\n return newdict", "response": "Returns a dictionary with keys removed leading underscores from key names."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef verify_initdict(initdict: InitDict) -> None:\n if (not isinstance(initdict, dict) or\n ARGS_LABEL not in initdict or\n KWARGS_LABEL not in initdict):\n raise ValueError(\"Not an InitDict dictionary\")", "response": "Ensures that the given InitDict is a proper InitDict."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef initdict_to_instance(d: InitDict, cls: ClassType) -> Any:\n args = d.get(ARGS_LABEL, [])\n kwargs = d.get(KWARGS_LABEL, {})\n # noinspection PyArgumentList\n return cls(*args, **kwargs)", "response": "Convert a dictionary to an instance of the given class."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nwrap a function producing a KwargsDict making it into an InitDict.", "response": "def wrap_kwargs_to_initdict(init_kwargs_fn: InitKwargsFnType,\n typename: str,\n check_result: bool = True) \\\n -> InstanceToInitDictFnType:\n \"\"\"\n Wraps a function producing a ``KwargsDict``, making it into a function\n producing an ``InitDict``.\n \"\"\"\n def wrapper(obj: Instance) -> InitDict:\n result = init_kwargs_fn(obj)\n if check_result:\n if not isinstance(result, dict):\n raise ValueError(\n \"Class {} failed to provide a kwargs dict and \"\n \"provided instead: {}\".format(typename, repr(result)))\n return kwargs_to_initdict(init_kwargs_fn(obj))\n\n return wrapper"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef wrap_args_kwargs_to_initdict(init_args_kwargs_fn: InitArgsKwargsFnType,\n typename: str,\n check_result: bool = True) \\\n -> InstanceToInitDictFnType:\n \"\"\"\n Wraps a function producing a ``KwargsDict``, making it into a function\n producing an ``InitDict``.\n \"\"\"\n def wrapper(obj: Instance) -> InitDict:\n result = init_args_kwargs_fn(obj)\n if check_result:\n if (not isinstance(result, tuple) or\n not len(result) == 2 or\n not isinstance(result[0], list) or\n not isinstance(result[1], dict)):\n raise ValueError(\n \"Class {} failed to provide an (args, kwargs) tuple and \"\n \"provided instead: {}\".format(typename, repr(result)))\n return args_kwargs_to_initdict(*result)\n\n return wrapper", "response": "Wraps a function producing a KwargsDict making it into a function\n ."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef make_instance_to_initdict(attributes: List[str]) -> InstanceToDictFnType:\n def custom_instance_to_initdict(x: Instance) -> InitDict:\n kwargs = {}\n for a in attributes:\n kwargs[a] = getattr(x, a)\n return kwargs_to_initdict(kwargs)\n\n return custom_instance_to_initdict", "response": "Returns a function that takes an object and produces an instanceToDictFnType enabling its re - creation."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef register_class_for_json(\n cls: ClassType,\n method: str = METHOD_SIMPLE,\n obj_to_dict_fn: InstanceToDictFnType = None,\n dict_to_obj_fn: DictToInstanceFnType = initdict_to_instance,\n default_factory: DefaultFactoryFnType = None) -> None:\n \"\"\"\n Registers the class cls for JSON serialization.\n\n - If both ``obj_to_dict_fn`` and dict_to_obj_fn are registered, the\n framework uses these to convert instances of the class to/from Python\n dictionaries, which are in turn serialized to JSON.\n\n - Otherwise:\n\n .. code-block:: python\n\n if method == 'simple':\n # ... uses simple_to_dict and simple_from_dict (q.v.)\n\n if method == 'strip_underscore':\n # ... uses strip_underscore_to_dict and simple_from_dict (q.v.)\n \"\"\"\n typename = cls.__qualname__ # preferable to __name__\n # ... __name__ looks like \"Thing\" and is ambiguous\n # ... __qualname__ looks like \"my.module.Thing\" and is not\n if obj_to_dict_fn and dict_to_obj_fn:\n descriptor = JsonDescriptor(\n typename=typename,\n obj_to_dict_fn=obj_to_dict_fn,\n dict_to_obj_fn=dict_to_obj_fn,\n cls=cls,\n default_factory=default_factory)\n elif method == METHOD_SIMPLE:\n descriptor = JsonDescriptor(\n typename=typename,\n obj_to_dict_fn=instance_to_initdict_simple,\n dict_to_obj_fn=initdict_to_instance,\n cls=cls,\n default_factory=default_factory)\n elif method == METHOD_STRIP_UNDERSCORE:\n descriptor = JsonDescriptor(\n typename=typename,\n obj_to_dict_fn=instance_to_initdict_stripping_underscores,\n dict_to_obj_fn=initdict_to_instance,\n cls=cls,\n default_factory=default_factory)\n else:\n raise ValueError(\"Unknown method, and functions not fully specified\")\n global TYPE_MAP\n TYPE_MAP[typename] = descriptor", "response": "Registers the class cls for JSON serialization."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nclassing decorator to register classes with our JSON system. - If method is ``'provides_init_args_kwargs'``, the class provides a function .. code-block:: python def init_args_kwargs(self) -> Tuple[List[Any], Dict[str, Any]] that returns an ``(args, kwargs)`` tuple, suitable for passing to its ``__init__()`` function as ``__init__(*args, **kwargs)``. - If method is ``'provides_init_kwargs'``, the class provides a function .. code-block:: python def init_kwargs(self) -> Dict that returns a dictionary ``kwargs`` suitable for passing to its ``__init__()`` function as ``__init__(**kwargs)``. - Otherwise, the method argument is as for ``register_class_for_json()``. Usage looks like: .. code-block:: python @register_for_json(method=METHOD_STRIP_UNDERSCORE) class TableId(object): def __init__(self, db: str = '', schema: str = '', table: str = '') -> None: self._db = db self._schema = schema self._table = table", "response": "def register_for_json(*args, **kwargs) -> Any:\n \"\"\"\n Class decorator to register classes with our JSON system.\n\n - If method is ``'provides_init_args_kwargs'``, the class provides a\n function\n\n .. code-block:: python\n\n def init_args_kwargs(self) -> Tuple[List[Any], Dict[str, Any]]\n\n that returns an ``(args, kwargs)`` tuple, suitable for passing to its\n ``__init__()`` function as ``__init__(*args, **kwargs)``.\n\n - If method is ``'provides_init_kwargs'``, the class provides a function\n\n .. code-block:: python\n\n def init_kwargs(self) -> Dict\n\n that returns a dictionary ``kwargs`` suitable for passing to its\n ``__init__()`` function as ``__init__(**kwargs)``.\n\n - Otherwise, the method argument is as for ``register_class_for_json()``.\n\n Usage looks like:\n\n .. code-block:: python\n\n @register_for_json(method=METHOD_STRIP_UNDERSCORE)\n class TableId(object):\n def __init__(self, db: str = '', schema: str = '',\n table: str = '') -> None:\n self._db = db\n self._schema = schema\n self._table = table\n\n \"\"\"\n if DEBUG:\n print(\"register_for_json: args = {}\".format(repr(args)))\n print(\"register_for_json: kwargs = {}\".format(repr(kwargs)))\n\n # http://stackoverflow.com/questions/653368/how-to-create-a-python-decorator-that-can-be-used-either-with-or-without-paramet # noqa\n # In brief,\n # @decorator\n # x\n #\n # means\n # x = decorator(x)\n #\n # so\n # @decorator(args)\n # x\n #\n # means\n # x = decorator(args)(x)\n\n if len(args) == 1 and len(kwargs) == 0 and callable(args[0]):\n if DEBUG:\n print(\"... called as @register_for_json\")\n # called as @decorator\n # ... the single argument is the class itself, e.g. Thing in:\n # @decorator\n # class Thing(object):\n # # ...\n # ... e.g.:\n # args = (<class '__main__.unit_tests.<locals>.SimpleThing'>,)\n # kwargs = {}\n cls = args[0] # type: ClassType\n register_class_for_json(cls, method=METHOD_SIMPLE)\n return cls\n\n # Otherwise:\n if DEBUG:\n print(\"... called as @register_for_json(*args, **kwargs)\")\n # called as @decorator(*args, **kwargs)\n # ... e.g.:\n # args = ()\n # kwargs = {'method': 'provides_to_init_args_kwargs_dict'}\n method = kwargs.pop('method', METHOD_SIMPLE) # type: str\n obj_to_dict_fn = kwargs.pop('obj_to_dict_fn', None) # type: InstanceToDictFnType # noqa\n dict_to_obj_fn = kwargs.pop('dict_to_obj_fn', initdict_to_instance) # type: DictToInstanceFnType # noqa\n default_factory = kwargs.pop('default_factory', None) # type: DefaultFactoryFnType # noqa\n check_result = kwargs.pop('check_results', True) # type: bool\n\n def register_json_class(cls_: ClassType) -> ClassType:\n odf = obj_to_dict_fn\n dof = dict_to_obj_fn\n if method == METHOD_PROVIDES_INIT_ARGS_KWARGS:\n if hasattr(cls_, INIT_ARGS_KWARGS_FN_NAME):\n odf = wrap_args_kwargs_to_initdict(\n getattr(cls_, INIT_ARGS_KWARGS_FN_NAME),\n typename=cls_.__qualname__,\n check_result=check_result\n )\n else:\n raise ValueError(\n \"Class type {} does not provide function {}\".format(\n cls_, INIT_ARGS_KWARGS_FN_NAME))\n elif method == METHOD_PROVIDES_INIT_KWARGS:\n if hasattr(cls_, INIT_KWARGS_FN_NAME):\n odf = wrap_kwargs_to_initdict(\n getattr(cls_, INIT_KWARGS_FN_NAME),\n typename=cls_.__qualname__,\n check_result=check_result\n )\n else:\n raise ValueError(\n \"Class type {} does not provide function {}\".format(\n cls_, INIT_KWARGS_FN_NAME))\n elif method == METHOD_NO_ARGS:\n odf = obj_with_no_args_to_init_dict\n register_class_for_json(cls_,\n method=method,\n obj_to_dict_fn=odf,\n dict_to_obj_fn=dof,\n default_factory=default_factory)\n return cls_\n\n return register_json_class"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nprints the JSON registered types map to the specified file.", "response": "def dump_map(file: TextIO = sys.stdout) -> None:\n \"\"\"\n Prints the JSON \"registered types\" map to the specified file.\n \"\"\"\n pp = pprint.PrettyPrinter(indent=4, stream=file)\n print(\"Type map: \", file=file)\n pp.pprint(TYPE_MAP)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nprovide a JSON decoder that converts dictionaries to Python objects if suitable methods are found in our TYPE_MAP.", "response": "def json_class_decoder_hook(d: Dict) -> Any:\n \"\"\"\n Provides a JSON decoder that converts dictionaries to Python objects if\n suitable methods are found in our ``TYPE_MAP``.\n \"\"\"\n if TYPE_LABEL in d:\n typename = d.get(TYPE_LABEL)\n if typename in TYPE_MAP:\n if DEBUG:\n log.debug(\"Deserializing: {!r}\", d)\n d.pop(TYPE_LABEL)\n descriptor = TYPE_MAP[typename]\n obj = descriptor.to_obj(d)\n if DEBUG:\n log.debug(\"... to: {!r}\", obj)\n return obj\n return d"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nencode an object to JSON using our custom encoder.", "response": "def json_encode(obj: Instance, **kwargs) -> str:\n \"\"\"\n Encodes an object to JSON using our custom encoder.\n\n The ``**kwargs`` can be used to pass things like ``'indent'``, for\n formatting.\n \"\"\"\n return json.dumps(obj, cls=JsonClassEncoder, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndecoding an object from JSON using our custom decoder.", "response": "def json_decode(s: str) -> Any:\n \"\"\"\n Decodes an object from JSON using our custom decoder.\n \"\"\"\n try:\n return json.JSONDecoder(object_hook=json_class_decoder_hook).decode(s)\n except json.JSONDecodeError:\n log.warning(\"Failed to decode JSON (returning None): {!r}\", s)\n return None"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert an dict to an Enum.", "response": "def dict_to_enum_fn(d: Dict[str, Any], enum_class: Type[Enum]) -> Enum:\n \"\"\"\n Converts an ``dict`` to a ``Enum``.\n \"\"\"\n return enum_class[d['name']]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nclassing decorator to register Enum - derived classes with our JSON system.", "response": "def register_enum_for_json(*args, **kwargs) -> Any:\n \"\"\"\n Class decorator to register ``Enum``-derived classes with our JSON system.\n See comments/help for ``@register_for_json``, above.\n \"\"\"\n if len(args) == 1 and len(kwargs) == 0 and callable(args[0]):\n # called as @register_enum_for_json\n cls = args[0] # type: ClassType\n register_class_for_json(\n cls,\n obj_to_dict_fn=enum_to_dict_fn,\n dict_to_obj_fn=dict_to_enum_fn\n )\n return cls\n else:\n # called as @register_enum_for_json(*args, **kwargs)\n raise AssertionError(\"Use as plain @register_enum_for_json, \"\n \"without arguments\")"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts a dict object back to a Pendulum.", "response": "def dict_to_pendulum(d: Dict[str, Any],\n pendulum_class: ClassType) -> DateTime:\n \"\"\"\n Converts a ``dict`` object back to a ``Pendulum``.\n \"\"\"\n return pendulum.parse(d['iso'])"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef dict_to_pendulumdate(d: Dict[str, Any],\n pendulumdate_class: ClassType) -> Date:\n \"\"\"\n Converts a ``dict`` object back to a ``pendulum.Date``.\n \"\"\"\n # noinspection PyTypeChecker\n return pendulum.parse(d['iso']).date()", "response": "Converts a dict object back to a pendulum. Date."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ntest if two objects are equal based on a comparison of the specified attrs.", "response": "def simple_eq(one: Instance, two: Instance, attrs: List[str]) -> bool:\n \"\"\"\n Test if two objects are equal, based on a comparison of the specified\n attributes ``attrs``.\n \"\"\"\n return all(getattr(one, a) == getattr(two, a) for a in attrs)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef smart_open(filename: str, mode: str = 'Ur', buffering: int = -1,\n encoding: str = None, errors: str = None, newline: str = None,\n closefd: bool = True) -> IO:\n \"\"\"\n Context manager (for use with ``with``) that opens a filename and provides\n a :class:`IO` object. If the filename is ``'-'``, however, then\n ``sys.stdin`` is used for reading and ``sys.stdout`` is used for writing.\n \"\"\"\n # https://stackoverflow.com/questions/17602878/how-to-handle-both-with-open-and-sys-stdout-nicely # noqa\n # https://stackoverflow.com/questions/1744989/read-from-file-or-stdin/29824059#29824059 # noqa\n if filename == '-':\n if mode is None or mode == '' or 'r' in mode:\n fh = sys.stdin\n else:\n fh = sys.stdout\n else:\n fh = open(filename, mode=mode,\n buffering=buffering, encoding=encoding, errors=errors,\n newline=newline, closefd=closefd)\n try:\n yield fh\n finally:\n if filename is not '-':\n fh.close()", "response": "Context manager that opens a file and provides a sequence of IO objects."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef writelines_nl(fileobj: TextIO, lines: Iterable[str]) -> None:\n fileobj.write('\\n'.join(lines) + '\\n')", "response": "Writes lines plus terminating newline characters to the file."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nwrites text to a file.", "response": "def write_text(filename: str, text: str) -> None:\n \"\"\"\n Writes text to a file.\n \"\"\"\n with open(filename, 'w') as f: # type: TextIO\n print(text, file=f)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nwriting text to a file compressed with gzip.", "response": "def write_gzipped_text(basefilename: str, text: str) -> None:\n \"\"\"\n Writes text to a file compressed with ``gzip`` (a ``.gz`` file).\n The filename is used directly for the \"inner\" file and the extension\n ``.gz`` is appended to the \"outer\" (zipped) file's name.\n \n This function exists primarily because Lintian wants non-timestamped gzip\n files, or it complains:\n - https://lintian.debian.org/tags/package-contains-timestamped-gzip.html\n - See http://stackoverflow.com/questions/25728472/python-gzip-omit-the-original-filename-and-timestamp\n \"\"\" # noqa\n zipfilename = basefilename + '.gz'\n compresslevel = 9\n mtime = 0\n with open(zipfilename, 'wb') as f:\n with gzip.GzipFile(basefilename, 'wb', compresslevel, f, mtime) as gz:\n with io.TextIOWrapper(gz) as tw:\n tw.write(text)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreading a file and returns all lines without comments.", "response": "def get_lines_without_comments(filename: str) -> List[str]:\n \"\"\"\n Reads a file, and returns all lines as a list, left- and right-stripping\n the lines and removing everything on a line after the first ``#``.\n NOTE: does not cope well with quoted ``#`` symbols!\n \"\"\"\n lines = []\n with open(filename) as f:\n for line in f:\n line = line.partition('#')[0] # the part before the first #\n line = line.rstrip()\n line = line.lstrip()\n if line:\n lines.append(line)\n return lines"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef gen_textfiles_from_filenames(\n filenames: Iterable[str]) -> Generator[TextIO, None, None]:\n \"\"\"\n Generates file-like objects from a list of filenames.\n\n Args:\n filenames: iterable of filenames\n\n Yields:\n each file as a :class:`TextIO` object\n\n \"\"\"\n for filename in filenames:\n with open(filename) as f:\n yield f", "response": "Generates file - like objects from a list of filenames."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngenerate lines from file - like objects.", "response": "def gen_lines_from_textfiles(\n files: Iterable[TextIO]) -> Generator[str, None, None]:\n \"\"\"\n Generates lines from file-like objects.\n\n Args:\n files: iterable of :class:`TextIO` objects\n\n Yields:\n each line of all the files\n\n \"\"\"\n for file in files:\n for line in file:\n yield line"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns an iterable of strings that are lower cased.", "response": "def gen_lower(x: Iterable[str]) -> Generator[str, None, None]:\n \"\"\"\n Args:\n x: iterable of strings\n\n Yields:\n each string in lower case\n \"\"\"\n for string in x:\n yield string.lower()"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngenerates lines from binary files.", "response": "def gen_lines_from_binary_files(\n files: Iterable[BinaryIO],\n encoding: str = UTF8) -> Generator[str, None, None]:\n \"\"\"\n Generates lines from binary files.\n Strips out newlines.\n\n Args:\n files: iterable of :class:`BinaryIO` file-like objects\n encoding: encoding to use\n\n Yields:\n each line of all the files\n\n \"\"\"\n for file in files:\n for byteline in file:\n line = byteline.decode(encoding).strip()\n yield line"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsplitting lines with splitter and yields a specified part by index.", "response": "def gen_part_from_line(lines: Iterable[str],\n part_index: int,\n splitter: str = None) -> Generator[str, None, None]:\n \"\"\"\n Splits lines with ``splitter`` and yields a specified part by index.\n\n Args:\n lines: iterable of strings\n part_index: index of part to yield\n splitter: string to split the lines on\n\n Yields:\n the specified part for each line\n\n \"\"\"\n for line in lines:\n parts = line.split(splitter)\n yield parts[part_index]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef gen_rows_from_csv_binfiles(\n csv_files: Iterable[BinaryIO],\n encoding: str = UTF8,\n skip_header: bool = False,\n **csv_reader_kwargs) -> Generator[Iterable[str], None, None]:\n \"\"\"\n Iterate through binary file-like objects that are CSV files in a specified\n encoding. Yield each row.\n\n Args:\n csv_files: iterable of :class:`BinaryIO` objects\n encoding: encoding to use\n skip_header: skip the header (first) row of each file?\n csv_reader_kwargs: arguments to pass to :func:`csv.reader`\n\n Yields:\n rows from the files\n\n \"\"\"\n dialect = csv_reader_kwargs.pop('dialect', None)\n for csv_file_bin in csv_files:\n # noinspection PyTypeChecker\n csv_file = io.TextIOWrapper(csv_file_bin, encoding=encoding)\n thisfile_dialect = dialect\n if thisfile_dialect is None:\n thisfile_dialect = csv.Sniffer().sniff(csv_file.read(1024))\n csv_file.seek(0)\n reader = csv.reader(csv_file, dialect=thisfile_dialect,\n **csv_reader_kwargs)\n first = True\n for row in reader:\n if first:\n first = False\n if skip_header:\n continue\n yield row", "response": "Generator that yields each row of the CSV file - like object in a specified encoding."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nrewrite a file from srcfilename to destfilename HTML - escaping it IVA in the process.", "response": "def webify_file(srcfilename: str, destfilename: str) -> None:\n \"\"\"\n Rewrites a file from ``srcfilename`` to ``destfilename``, HTML-escaping it\n in the process.\n \"\"\"\n with open(srcfilename) as infile, open(destfilename, 'w') as ofile:\n for line_ in infile:\n ofile.write(escape(line_))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef remove_gzip_timestamp(filename: str,\n gunzip_executable: str = \"gunzip\",\n gzip_executable: str = \"gzip\",\n gzip_args: List[str] = None) -> None:\n \"\"\"\n Uses external ``gunzip``/``gzip`` tools to remove a ``gzip`` timestamp.\n Necessary for Lintian.\n \"\"\"\n gzip_args = gzip_args or [\n \"-9\", # maximum compression (or Lintian moans)\n \"-n\",\n ]\n # gzip/gunzip operate on SINGLE files\n with tempfile.TemporaryDirectory() as dir_:\n basezipfilename = os.path.basename(filename)\n newzip = os.path.join(dir_, basezipfilename)\n with open(newzip, 'wb') as z:\n log.info(\n \"Removing gzip timestamp: \"\n \"{} -> gunzip -c -> gzip -n -> {}\",\n basezipfilename, newzip)\n p1 = subprocess.Popen([gunzip_executable, \"-c\", filename],\n stdout=subprocess.PIPE)\n p2 = subprocess.Popen([gzip_executable] + gzip_args,\n stdin=p1.stdout, stdout=z)\n p2.communicate()\n shutil.copyfile(newzip, filename)", "response": "Removes a gzip timestamp from a single file."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef replace_in_file(filename: str, text_from: str, text_to: str) -> None:\n log.info(\"Amending {}: {} -> {}\",\n filename, repr(text_from), repr(text_to))\n with open(filename) as infile:\n contents = infile.read()\n contents = contents.replace(text_from, text_to)\n with open(filename, 'w') as outfile:\n outfile.write(contents)", "response": "Replaces text in a file."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreplace multiple from and to strings pairs within a single file.", "response": "def replace_multiple_in_file(filename: str,\n replacements: List[Tuple[str, str]]) -> None:\n \"\"\"\n Replaces multiple from/to string pairs within a single file.\n\n Args:\n filename: filename to process (modifying it in place)\n replacements: list of ``(from_text, to_text)`` tuples\n \"\"\"\n with open(filename) as infile:\n contents = infile.read()\n for text_from, text_to in replacements:\n log.info(\"Amending {}: {} -> {}\",\n filename, repr(text_from), repr(text_to))\n contents = contents.replace(text_from, text_to)\n with open(filename, 'w') as outfile:\n outfile.write(contents)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting a file from UNIX to Windows line endings or the alphabetical order.", "response": "def convert_line_endings(filename: str, to_unix: bool = False,\n to_windows: bool = False) -> None:\n \"\"\"\n Converts a file (in place) from UNIX to Windows line endings, or the\n reverse.\n\n Args:\n filename: filename to modify (in place)\n to_unix: convert Windows (CR LF) to UNIX (LF)\n to_windows: convert UNIX (LF) to Windows (CR LF)\n \"\"\"\n assert to_unix != to_windows\n with open(filename, \"rb\") as f:\n contents = f.read()\n windows_eol = b\"\\r\\n\" # CR LF\n unix_eol = b\"\\n\" # LF\n if to_unix:\n log.info(\"Converting from Windows to UNIX line endings: {!r}\",\n filename)\n src = windows_eol\n dst = unix_eol\n else: # to_windows\n log.info(\"Converting from UNIX to Windows line endings: {!r}\",\n filename)\n src = unix_eol\n dst = windows_eol\n if windows_eol in contents:\n log.info(\"... already contains at least one Windows line ending; \"\n \"probably converted before; skipping\")\n return\n contents = contents.replace(src, dst)\n with open(filename, \"wb\") as f:\n f.write(contents)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef is_line_in_file(filename: str, line: str) -> bool:\n assert \"\\n\" not in line\n with open(filename, \"r\") as file:\n for fileline in file:\n if fileline == line:\n return True\n return False", "response": "Detects whether a line is present within a file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef add_line_if_absent(filename: str, line: str) -> None:\n assert \"\\n\" not in line\n if not is_line_in_file(filename, line):\n log.info(\"Appending line {!r} to file {!r}\", line, filename)\n with open(filename, \"a\") as file:\n file.writelines([line])", "response": "Adds a line to the file if it s not already in the file."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nchecks if the get query returned a result and return a boolean", "response": "def is_get_query_with_results(results):\n \"\"\"\n :param results: the response from Elasticsearch\n :return: true if the get query returned a result, false otherwise\n \"\"\"\n return results and EsConst.FOUND in results and results[EsConst.FOUND] and EsConst.FIELDS in results"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef validate_scan_result(results):\n if results[EsConst.SHARDS][EsConst.FAILED] and results[EsConst.SHARDS][EsConst.FAILED] > 0:\n raise ScanError(\n 'Scroll request has failed on %d shards out of %d.' %\n (results[EsConst.SHARDS][EsConst.FAILED], results[EsConst.SHARDS][EsConst.TOTAL])\n )", "response": "Check if there s a failed shard in the scan query and if so raise an exception."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef to_key_val_list(value, sort=False, insensitive=False):\n if value is None:\n return None\n\n if isinstance(value, (str, bytes, bool, int)):\n raise ValueError('cannot encode objects that are not 2-tuples')\n\n if isinstance(value, collections.Mapping):\n value = value.items()\n\n if sort and not insensitive:\n values = sorted(value)\n elif sort:\n values = sorted(value, key=lambda t: t[0].lower())\n else:\n values = value\n return list(values)", "response": "Takes an object and test to see if it can be represented as a\n dictionary. If it can be return a list of tuples. If it can be return a list of tuples."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert a dictionary of values into a key - value dictionary.", "response": "def to_key_val_dict(values):\n \"\"\"\n Take an object and test to see if it can be represented as a\n dictionary. If it can be, return a list of tuples, e.g.,\n ::\n >>> to_key_val_dict([('key', 'val')])\n {'key': 'val'}\n >>> to_key_val_dict({'key': 'val'})\n {'key': 'val'}\n >>> to_key_val_dict('string')\n ValueError: dictionary update sequence element.\n \"\"\"\n if values is None:\n return {}\n\n if isinstance(values, (str, bytes, bool, int)):\n raise ValueError('cannot encode objects that are not 2-tuples')\n\n if isinstance(values, collections.Mapping):\n values = values.items()\n\n dict_to_return = dict()\n for k, v in values:\n if k in dict_to_return and isinstance(dict_to_return[k], list) and isinstance(v, list):\n dict_to_return[k].extend(v)\n elif k in dict_to_return and isinstance(dict_to_return[k], list):\n dict_to_return[k].append(v)\n elif k in dict_to_return:\n dict_to_return[k] = [dict_to_return[k], v]\n else:\n dict_to_return[k] = v\n\n return dict_to_return"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef valid_choice(strvalue: str, choices: Iterable[Tuple[str, str]]) -> bool:\n return strvalue in [str(x[0]) for x in choices]", "response": "Checks that value is one of the valid option in choices"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef choice_explanation(value: str, choices: Iterable[Tuple[str, str]]) -> str:\n for k, v in choices:\n if k == value:\n return v\n return ''", "response": "Returns the explanation associated with a Django choice tuple - list."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nrunning a command and captures its output to one or more TEED files.", "response": "def teed_call(cmd_args,\n stdout_targets: List[TextIO] = None,\n stderr_targets: List[TextIO] = None,\n encoding: str = sys.getdefaultencoding(),\n **kwargs):\n \"\"\"\n Runs a command and captures its output via :func:`tee` to one or more \n destinations. The output is always captured (otherwise we would lose \n control of the output and ability to ``tee`` it); if no destination is \n specified, we add a null handler.\n\n We insist on ``TextIO`` output files to match ``sys.stdout`` (etc.).\n\n A variation on:\n https://stackoverflow.com/questions/4984428/python-subprocess-get-childrens-output-to-file-and-terminal\n\n Args:\n cmd_args: arguments for the command to run\n stdout_targets: file-like objects to write ``stdout`` to\n stderr_targets: file-like objects to write ``stderr`` to \n encoding: encoding to apply to ``stdout`` and ``stderr``\n kwargs: additional arguments for :class:`subprocess.Popen`\n \n \"\"\" # noqa\n # Make a copy so we can append without damaging the original:\n stdout_targets = stdout_targets.copy() if stdout_targets else [] # type: List[TextIO] # noqa\n stderr_targets = stderr_targets.copy() if stderr_targets else [] # type: List[TextIO] # noqa\n p = Popen(cmd_args, stdout=PIPE, stderr=PIPE, **kwargs)\n threads = [] # type: List[Thread]\n with open(os.devnull, \"w\") as null: # type: TextIO\n if not stdout_targets:\n stdout_targets.append(null)\n if not stderr_targets:\n stderr_targets.append(null)\n # Now, by default, because we're not using \"universal_newlines\", both\n # p.stdout and p.stderr are binary.\n stdout_txt = TextIOWrapper(p.stdout, encoding=encoding) # type: TextIO # noqa\n stderr_txt = TextIOWrapper(p.stderr, encoding=encoding) # type: TextIO # noqa\n threads.append(tee(stdout_txt, *stdout_targets))\n threads.append(tee(stderr_txt, *stderr_targets))\n for t in threads:\n t.join() # wait for IO completion\n return p.wait()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nwriting a message to the underlying stream.", "response": "def write(self, message: str) -> None:\n \"\"\"\n To act as a file.\n \"\"\"\n self.underlying_stream.write(message)\n self.file.write(message)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef flush(self) -> None:\n self.underlying_stream.flush()\n self.file.flush()\n os.fsync(self.file.fileno())", "response": "Flushes the underlying stream and file."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nclose the file and the underlying stream.", "response": "def close(self) -> None:\n \"\"\"\n To act as a file.\n \"\"\"\n if self.underlying_stream:\n if self.using_stdout:\n sys.stdout = self.underlying_stream\n else:\n sys.stderr = self.underlying_stream\n self.underlying_stream = None\n if self.file:\n # Do NOT close the file; we don't own it.\n self.file = None\n log.debug(\"Finished copying {} to {}\",\n self.output_description, self.filename)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nprovide a bootstrap resampling of an array.", "response": "def bootstrap(array):\n \"\"\"\n Provides a bootstrap resampling of an array. Provides another\n statistical method to estimate the variance of a dataset.\n\n For a `PCA` object in this library, it should be applied to\n `Orientation.array` method.\n \"\"\"\n reg_func = lambda a: N.linalg.svd(a,full_matrices=False)[2][2]\n beta_boots = bootstrap(array, func=reg_func)\n return yhat, yhat_boots"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ntransforming text into its ANSI SQL - quoted version e. g. in Python 3. 6.", "response": "def sql_string_literal(text: str) -> str:\n \"\"\"\n Transforms text into its ANSI SQL-quoted version, e.g. (in Python ``repr()``\n format):\n\n .. code-block:: none\n\n \"some string\" -> \"'some string'\"\n \"Jack's dog\" -> \"'Jack''s dog'\"\n \"\"\"\n # ANSI SQL: http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt\n # <character string literal>\n return SQUOTE + text.replace(SQUOTE, DOUBLE_SQUOTE) + SQUOTE"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef sql_datetime_literal(dt: DateTimeLikeType,\n subsecond: bool = False) -> str:\n \"\"\"\n Transforms a Python object that is of duck type ``datetime.datetime`` into\n an ANSI SQL literal string, like ``'2000-12-31 23:59:59'``, or if\n ``subsecond=True``, into the (non-ANSI) format\n ``'2000-12-31 23:59:59.123456'`` or similar.\n \"\"\"\n # ANSI SQL: http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt\n # <timestamp string>\n # ... the subsecond part is non-ANSI\n fmt = \"'%Y-%m-%d %H:%M:%S{}'\".format(\".%f\" if subsecond else \"\")\n return dt.strftime(fmt)", "response": "Transforms a Python object that is of duck type datetime. datetime into an ANSI SQL literal string like 2000 - 12 - 31 23 - 59 - 59.123456."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ntransforms a single - line string into an ANSI SQL comment.", "response": "def sql_comment(comment: str) -> str:\n \"\"\"\n Transforms a single- or multi-line string into an ANSI SQL comment,\n prefixed by ``--``.\n \"\"\"\n \"\"\"Using -- as a comment marker is ANSI SQL.\"\"\"\n if not comment:\n return \"\"\n return \"\\n\".join(\"-- {}\".format(x) for x in comment.splitlines())"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nremoving double quotes from a SQL string.", "response": "def sql_dequote_string(s: str) -> str:\n \"\"\"\n Reverses :func:`sql_quote_string`.\n \"\"\"\n if len(s) < 2 or s[0] != SQUOTE or s[-1] != SQUOTE:\n raise ValueError(\"Not an SQL string literal\")\n s = s[1:-1] # strip off the surrounding quotes\n return s.replace(DOUBLE_SQUOTE, SQUOTE)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef gen_items_from_sql_csv(s: str) -> Generator[str, None, None]:\n # csv.reader will not both process the quotes and return the quotes;\n # we need them to distinguish e.g. NULL from 'NULL'.\n # log.warning('gen_items_from_sql_csv: s = {0!r}', s)\n if not s:\n return\n n = len(s)\n startpos = 0\n pos = 0\n in_quotes = False\n while pos < n:\n if not in_quotes:\n if s[pos] == COMMA:\n # end of chunk\n chunk = s[startpos:pos] # does not include s[pos]\n result = chunk.strip()\n # log.warning('yielding: {0!r}', result)\n yield result\n startpos = pos + 1\n elif s[pos] == SQUOTE:\n # start of quote\n in_quotes = True\n else:\n if pos < n - 1 and s[pos] == SQUOTE and s[pos + 1] == SQUOTE:\n # double quote, '', is an escaped quote, not end of quote\n pos += 1 # skip one more than we otherwise would\n elif s[pos] == SQUOTE:\n # end of quote\n in_quotes = False\n pos += 1\n # Last chunk\n result = s[startpos:].strip()\n # log.warning('yielding last: {0!r}', result)\n yield result", "response": "Yields the items from a SQL string."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_case_insensitive_dict_key(d: Dict, k: str) -> Optional[str]:\n for key in d.keys():\n if k.lower() == key.lower():\n return key\n return None", "response": "Given a dictionary d and a key k return the key that matches k."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef merge_two_dicts(x: Dict, y: Dict) -> Dict:\n z = x.copy()\n z.update(y)\n return z", "response": "Given two dicts merge them into a new dict as a shallow copy"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nrename a key in dictionary d from old to new in place.", "response": "def rename_key(d: Dict[str, Any], old: str, new: str) -> None:\n \"\"\"\n Rename a key in dictionary ``d`` from ``old`` to ``new``, in place.\n \"\"\"\n d[new] = d.pop(old)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a copy of the dictionary d with the keys renamed according to mapping.", "response": "def rename_keys(d: Dict[str, Any], mapping: Dict[str, str]) -> Dict[str, Any]:\n \"\"\"\n Returns a copy of the dictionary ``d`` with its keys renamed according to\n ``mapping``.\n\n Args:\n d: the starting dictionary\n mapping: a dictionary of the format ``{old_key_name: new_key_name}``\n\n Returns:\n a new dictionary\n\n Keys that are not in ``mapping`` are left unchanged.\n The input parameters are not modified.\n \"\"\"\n result = {} # type: Dict[str, Any]\n for k, v in d.items():\n if k in mapping:\n k = mapping[k]\n result[k] = v\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nrenames the keys in the dictionary d according to the mapping renames.", "response": "def rename_keys_in_dict(d: Dict[str, Any], renames: Dict[str, str]) -> None:\n \"\"\"\n Renames, IN PLACE, the keys in ``d`` according to the mapping in\n ``renames``.\n \n Args:\n d: a dictionary to modify \n renames: a dictionary of the format ``{old_key_name: new_key_name}``\n \n See\n https://stackoverflow.com/questions/4406501/change-the-name-of-a-key-in-dictionary.\n \"\"\" # noqa\n for old_key, new_key in renames.items():\n if new_key == old_key:\n continue\n if old_key in d:\n if new_key in d:\n raise ValueError(\n \"rename_keys_in_dict: renaming {} -> {} but new key \"\n \"already exists\".format(repr(old_key), repr(new_key)))\n d[new_key] = d.pop(old_key)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a copy of d but with prefix prepended to its keys.", "response": "def prefix_dict_keys(d: Dict[str, Any], prefix: str) -> Dict[str, Any]:\n \"\"\"\n Returns a dictionary that's a copy of as ``d`` but with ``prefix``\n prepended to its keys.\n \"\"\"\n result = {} # type: Dict[str, Any]\n for k, v in d.items():\n result[prefix + k] = v\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ntake a k - > v mapping and returns a k - > v mapping.", "response": "def reversedict(d: Dict[Any, Any]) -> Dict[Any, Any]:\n \"\"\"\n Takes a ``k -> v`` mapping and returns a ``v -> k`` mapping.\n \"\"\"\n return {v: k for k, v in d.items()}"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef set_null_values_in_dict(d: Dict[str, Any],\n null_literals: List[Any]) -> None:\n \"\"\"\n Within ``d`` (in place), replace any values found in ``null_literals`` with\n ``None``.\n \"\"\"\n if not null_literals:\n return\n # DO NOT add/delete values to/from a dictionary during iteration, but it\n # is OK to modify existing keys:\n # https://stackoverflow.com/questions/6777485\n # https://stackoverflow.com/questions/2315520\n # https://docs.python.org/3/library/stdtypes.html#dict-views\n for k, v in d.items():\n if v in null_literals:\n d[k] = None", "response": "Sets any null values found in null_literals with None."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nmap keys in a list of objects to their corresponding values.", "response": "def map_keys_to_values(l: List[Any], d: Dict[Any, Any], default: Any = None,\n raise_if_missing: bool = False,\n omit_if_missing: bool = False) -> List[Any]:\n \"\"\"\n The ``d`` dictionary contains a ``key -> value`` mapping.\n\n We start with a list of potential keys in ``l``, and return a list of\n corresponding values -- substituting ``default`` if any are missing,\n or raising :exc:`KeyError` if ``raise_if_missing`` is true, or omitting the\n entry if ``omit_if_missing`` is true.\n \"\"\"\n result = []\n for k in l:\n if raise_if_missing and k not in d:\n raise ValueError(\"Missing key: \" + repr(k))\n if omit_if_missing and k not in d:\n continue\n result.append(d.get(k, default))\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a dictionary that contains the changes made to d1 to d2.", "response": "def dict_diff(d1: Dict[Any, Any], d2: Dict[Any, Any],\n deleted_value: Any = None) -> Dict[Any, Any]:\n \"\"\"\n Returns a representation of the changes that need to be made to ``d1`` to\n create ``d2``.\n\n Args:\n d1: a dictionary\n d2: another dictionary\n deleted_value: value to use for deleted keys; see below\n\n Returns:\n dict: a dictionary of the format ``{k: v}`` where the ``k``/``v`` pairs\n are key/value pairs that are absent from ``d1`` and present in ``d2``,\n or present in both but with different values (in which case the ``d2``\n value is shown). If a key ``k`` is present in ``d1`` but absent in\n ``d2``, the result dictionary has the entry ``{k: deleted_value}``.\n\n \"\"\"\n changes = {k: v for k, v in d2.items()\n if k not in d1 or d2[k] != d1[k]}\n for k in d1.keys():\n if k not in d2:\n changes[k] = deleted_value\n return changes"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndeletes keys from a dictionary in place.", "response": "def delete_keys(d: Dict[Any, Any],\n keys_to_delete: List[Any],\n keys_to_keep: List[Any]) -> None:\n \"\"\"\n Deletes keys from a dictionary, in place.\n\n Args:\n d:\n dictonary to modify\n keys_to_delete:\n if any keys are present in this list, they are deleted...\n keys_to_keep:\n ... unless they are present in this list.\n \"\"\"\n for k in keys_to_delete:\n if k in d and k not in keys_to_keep:\n del d[k]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef process_bind_param(self, value: Optional[List[str]],\n dialect: Dialect) -> str:\n \"\"\"Convert things on the way from Python to the database.\"\"\"\n retval = self._strlist_to_dbstr(value)\n return retval", "response": "Convert things on the way from Python to database."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef process_result_value(self, value: Optional[str],\n dialect: Dialect) -> List[str]:\n \"\"\"Convert things on the way from the database to Python.\"\"\"\n retval = self._dbstr_to_strlist(value)\n return retval", "response": "Convert things on the way from the database to Python."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef process_literal_param(self, value: Optional[List[int]],\n dialect: Dialect) -> str:\n \"\"\"Convert things on the way from Python to the database.\"\"\"\n retval = self._intlist_to_dbstr(value)\n return retval", "response": "Convert things on the way from Python to the database."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef process_result_value(self, value: Optional[str],\n dialect: Dialect) -> List[int]:\n \"\"\"Convert things on the way from the database to Python.\"\"\"\n retval = self._dbstr_to_intlist(value)\n return retval", "response": "Convert things on the way from the database to Python."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef set_timing(self, timing: bool, reset: bool = False) -> None:\n self._timing = timing\n if reset:\n self.reset()", "response": "Set the timing parameter."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef start(self, name: str, increment_count: bool = True) -> None:\n if not self._timing:\n return\n now = get_now_utc_pendulum()\n\n # If we were already timing something else, pause that.\n if self._stack:\n last = self._stack[-1]\n self._totaldurations[last] += now - self._starttimes[last]\n\n # Start timing our new thing\n if name not in self._starttimes:\n self._totaldurations[name] = datetime.timedelta()\n self._count[name] = 0\n self._starttimes[name] = now\n if increment_count:\n self._count[name] += 1\n self._stack.append(name)", "response": "Start a named timer."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nstop a named timer.", "response": "def stop(self, name: str) -> None:\n \"\"\"\n Stop a named timer.\n\n Args:\n name: timer to stop\n \"\"\"\n if not self._timing:\n return\n now = get_now_utc_pendulum()\n\n # Validity check\n if not self._stack:\n raise AssertionError(\"MultiTimer.stop() when nothing running\")\n if self._stack[-1] != name:\n raise AssertionError(\n \"MultiTimer.stop({}) when {} is running\".format(\n repr(name), repr(self._stack[-1])))\n\n # Finish what we were asked to\n self._totaldurations[name] += now - self._starttimes[name]\n self._stack.pop()\n\n # Now, if we were timing something else before we started \"name\",\n # resume...\n if self._stack:\n last = self._stack[-1]\n self._starttimes[last] = now"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nplots an attitude measurement on an object.", "response": "def girdle_error(ax, fit, **kwargs):\n \"\"\"\n Plot an attitude measurement on an `mplstereonet` axes object.\n \"\"\"\n vertices = []\n codes = []\n for sheet in ('upper','lower'):\n err = plane_errors(fit.axes, fit.covariance_matrix,\n sheet=sheet)\n lonlat = N.array(err)\n lonlat *= -1\n n = len(lonlat)\n if sheet == 'lower':\n lonlat = lonlat[::-1]\n vertices += list(lonlat)\n codes.append(Path.MOVETO)\n codes += [Path.LINETO]*(n-1)\n\n plot_patch(ax, vertices, codes, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nplots the error to the pole to a plane on a matplotlibstereonet object.", "response": "def pole_error(ax, fit, **kwargs):\n \"\"\"\n Plot the error to the pole to a plane on a `mplstereonet`\n axis object.\n \"\"\"\n ell = normal_errors(fit.axes, fit.covariance_matrix)\n lonlat = -N.array(ell)\n n = len(lonlat)\n codes = [Path.MOVETO]\n codes += [Path.LINETO]*(n-1)\n vertices = list(lonlat)\n\n plot_patch(ax, vertices, codes, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nping a host using OS tools.", "response": "def ping(hostname: str, timeout_s: int = 5) -> bool:\n \"\"\"\n Pings a host, using OS tools.\n\n Args:\n hostname: host name or IP address\n timeout_s: timeout in seconds\n\n Returns:\n was the ping successful?\n\n \"\"\"\n if sys.platform == \"win32\":\n timeout_ms = timeout_s * 1000\n args = [\n \"ping\",\n hostname,\n \"-n\", \"1\", # ping count\n \"-w\", str(timeout_ms), # timeout\n ]\n elif sys.platform.startswith('linux'):\n args = [\n \"ping\",\n hostname,\n \"-c\", \"1\", # ping count\n \"-w\", str(timeout_s), # timeout\n ]\n else:\n raise AssertionError(\"Don't know how to ping on this operating system\")\n proc = subprocess.Popen(args,\n stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n proc.communicate()\n retcode = proc.returncode\n return retcode == 0"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndownload a URL to a file.", "response": "def download(url: str, filename: str,\n skip_cert_verify: bool = True) -> None:\n \"\"\"\n Downloads a URL to a file.\n\n Args:\n url: URL to download from\n filename: file to save to\n skip_cert_verify: skip SSL certificate check?\n \"\"\"\n log.info(\"Downloading from {} to {}\", url, filename)\n\n # urllib.request.urlretrieve(url, filename)\n # ... sometimes fails (e.g. downloading\n # https://www.openssl.org/source/openssl-1.1.0g.tar.gz under Windows) with:\n # ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777) # noqa\n # ... due to this certificate root problem (probably because OpenSSL\n # [used by Python] doesn't play entirely by the same rules as others?):\n # https://stackoverflow.com/questions/27804710\n # So:\n\n ctx = ssl.create_default_context() # type: ssl.SSLContext\n if skip_cert_verify:\n log.debug(\"Skipping SSL certificate check for \" + url)\n ctx.check_hostname = False\n ctx.verify_mode = ssl.CERT_NONE\n with urllib.request.urlopen(url, context=ctx) as u, open(filename,\n 'wb') as f: # noqa\n f.write(u.read())"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef gen_binary_files_from_urls(\n urls: Iterable[str],\n on_disk: bool = False,\n show_info: bool = True) -> Generator[BinaryIO, None, None]:\n \"\"\"\n Generate binary files from a series of URLs (one per URL).\n\n Args:\n urls: iterable of URLs\n on_disk: if ``True``, yields files that are on disk (permitting\n random access); if ``False``, yields in-memory files (which will\n not permit random access)\n show_info: show progress to the log?\n\n Yields:\n files, each of type :class:`BinaryIO`\n\n \"\"\"\n for url in urls:\n if on_disk:\n # Necessary for e.g. zip processing (random access)\n with tempfile.TemporaryDirectory() as tmpdir:\n filename = os.path.join(tmpdir, \"tempfile\")\n download(url=url, filename=filename)\n with open(filename, 'rb') as f:\n yield f\n else:\n if show_info:\n log.info(\"Reading from URL: {}\", url)\n with urllib.request.urlopen(url) as f:\n yield f\n if show_info:\n log.info(\"... finished reading from URL: {}\", url)", "response": "Generates binary files from a series of URLs."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef find_nth(s: str, x: str, n: int = 0, overlap: bool = False) -> int:\n length_of_fragment = 1 if overlap else len(x)\n i = -length_of_fragment\n for _ in range(n + 1):\n i = s.find(x, i + length_of_fragment)\n if i < 0:\n break\n return i", "response": "Finds the position of nth occurrence of x in s or - 1 if there isn t one."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef split_string(x: str, n: int) -> List[str]:\n # https://stackoverflow.com/questions/9475241/split-string-every-nth-character # noqa\n return [x[i:i+n] for i in range(0, len(x), n)]", "response": "Split string into chunks of length n"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef multiple_replace(text: str, rep: Dict[str, str]) -> str:\n rep = dict((re.escape(k), v) for k, v in rep.items())\n pattern = re.compile(\"|\".join(rep.keys()))\n return pattern.sub(lambda m: rep[re.escape(m.group(0))], text)", "response": "Returns a version of text with the keys of rep replaced by their values."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a list of strings produced by applying multiple_replace to every string in stringlist.", "response": "def replace_in_list(stringlist: Iterable[str],\n replacedict: Dict[str, str]) -> List[str]:\n \"\"\"\n Returns a list produced by applying :func:`multiple_replace` to every\n string in ``stringlist``.\n\n Args:\n stringlist: list of source strings\n replacedict: dictionary mapping \"original\" to \"replacement\" strings\n\n Returns:\n list of final strings\n\n \"\"\"\n newlist = []\n for fromstring in stringlist:\n newlist.append(multiple_replace(fromstring, replacedict))\n return newlist"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nmangling unicode to ASCII losing accents etc. in the process.", "response": "def mangle_unicode_to_ascii(s: Any) -> str:\n \"\"\"\n Mangle unicode to ASCII, losing accents etc. in the process.\n \"\"\"\n # http://stackoverflow.com/questions/1207457\n if s is None:\n return \"\"\n if not isinstance(s, str):\n s = str(s)\n return (\n unicodedata.normalize('NFKD', s)\n .encode('ascii', 'ignore') # gets rid of accents\n .decode('ascii') # back to a string\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef strnum(prefix: str, num: int, suffix: str = \"\") -> str:\n return \"{}{}{}\".format(prefix, num, suffix)", "response": "Makes a string of the format <prefix><number><suffix >."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nmake a string of the format <prefix><number><suffix > for every number in numbers and returns them as a list.", "response": "def strnumlist(prefix: str, numbers: List[int], suffix: str = \"\") -> List[str]:\n \"\"\"\n Makes a string of the format ``<prefix><number><suffix>`` for every number\n in ``numbers``, and returns them as a list.\n \"\"\"\n return [\"{}{}{}\".format(prefix, num, suffix) for num in numbers]"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nmake a string of the format <prefix><number><suffix > for every number from first to last inclusive.", "response": "def strseq(prefix: str, first: int, last: int, suffix: str = \"\") -> List[str]:\n \"\"\"\n Makes a string of the format ``<prefix><number><suffix>`` for every number\n from ``first`` to ``last`` inclusive, and returns them as a list.\n \"\"\"\n return [strnum(prefix, n, suffix) for n in range(first, last + 1)]"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncalculates the variance of a bootstrap .", "response": "def aligned_residuals(pca):\n \"\"\"\n Plots error components along with bootstrap\n resampled error surface. Provides another\n statistical method to estimate the variance\n of a dataset.\n \"\"\"\n A = pca.rotated()\n fig, axes = P.subplots(2,1,\n sharex=True, frameon=False)\n fig.subplots_adjust(hspace=0, wspace=0.1)\n kw = dict(c=\"#555555\", s=40, alpha=0.5)\n\n #lengths = attitude.pca.singular_values[::-1]\n lengths = (A[:,i].max()-A[:,i].min() for i in range(3))\n\n titles = (\n \"Long cross-section (axis 3 vs. axis 1)\",\n \"Short cross-section (axis 3 vs. axis 2)\")\n\n for title,ax,(a,b) in zip(titles,axes,\n [(0,2),(1,2)]):\n\n seaborn.regplot(A[:,a], A[:,b], ax=ax)\n ax.text(0,1,title,\n verticalalignment='top',\n transform=ax.transAxes)\n ax.autoscale(tight=True)\n for spine in ax.spines.itervalues():\n spine.set_visible(False)\n ax.set_xlabel(\"Meters\")\n return fig"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the singleton instance of the class.", "response": "def get_instance(self):\n \"\"\"\n Returns the singleton instance. Upon its first call, it creates a\n new instance of the decorated class and calls its `__init__` method.\n On all subsequent calls, the already created instance is returned.\n \"\"\"\n try:\n return self._instance\n except AttributeError:\n self._instance = self._decorated()\n return self._instance"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef mapper_init(self):\n self.lookup = CachedMetroLookup(precision=GEOHASH_PRECISION)\n self.extractor = WordExtractor()", "response": "build local spatial index of US metro areas"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nestablishing connection to MongoDB instance and initialize the object", "response": "def reducer_init_output(self):\n \"\"\" establish connection to MongoDB \"\"\"\n try:\n self.mongo = MongoGeo(db=DB, collection=COLLECTION, timeout=MONGO_TIMEOUT)\n except ServerSelectionTimeoutError:\n # failed to connect to running MongoDB instance\n self.mongo = None"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef save(self, *args, **kwargs):\n slug = \"{}:{}\".format(self.level.uid, self.code)\n if self.parent:\n self.uid = \"{}_{}\".format(self.parent.uid, slug)\n else:\n self.uid = slug\n self.slug = uuslug(\n self.name, instance=self, max_length=100, separator=\"-\", start_no=2\n )\n super(Division, self).save(*args, **kwargs)", "response": "Save the instance to the database."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef add_intersecting(self, division, intersection=None, symm=True):\n relationship, created = IntersectRelationship.objects.update_or_create(\n from_division=self,\n to_division=division,\n defaults={\"intersection\": intersection},\n )\n if symm:\n division.add_intersecting(self, None, False)\n return relationship", "response": "Adds paired relationships between intersecting divisions."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nremoves paired relationships between intersecting divisions", "response": "def remove_intersecting(self, division, symm=True):\n \"\"\"Removes paired relationships between intersecting divisions\"\"\"\n IntersectRelationship.objects.filter(\n from_division=self, to_division=division\n ).delete()\n if symm:\n division.remove_intersecting(self, False)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set_intersection(self, division, intersection):\n IntersectRelationship.objects.filter(\n from_division=self, to_division=division\n ).update(intersection=intersection)", "response": "Set intersection percentage of intersecting divisions."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting intersection percentage of intersecting divisions.", "response": "def get_intersection(self, division):\n \"\"\"Get intersection percentage of intersecting divisions.\"\"\"\n try:\n return IntersectRelationship.objects.get(\n from_division=self, to_division=division\n ).intersection\n except ObjectDoesNotExist:\n raise Exception(\"No intersecting relationship with that division.\")"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nlaunches a job into the SLURM environment.", "response": "def launch_slurm(jobname: str,\n cmd: str,\n memory_mb: int,\n project: str,\n qos: str,\n email: str,\n duration: timedelta,\n tasks_per_node: int,\n cpus_per_task: int,\n partition: str = \"\",\n modules: List[str] = None,\n directory: str = os.getcwd(),\n encoding: str = \"ascii\") -> None:\n \"\"\"\n Launch a job into the SLURM environment.\n\n Args:\n jobname: name of the job\n cmd: command to be executed\n memory_mb: maximum memory requirement per process (Mb)\n project: project name\n qos: quality-of-service name\n email: user's e-mail address\n duration: maximum duration per job\n tasks_per_node: tasks per (cluster) node\n cpus_per_task: CPUs per task\n partition: cluster partition name\n modules: SLURM modules to load\n directory: directory to change to\n encoding: encoding to apply to launch script as sent to ``sbatch``\n \"\"\"\n if partition:\n partition_cmd = \"#SBATCH -p {}\".format(partition)\n else:\n partition_cmd = \"\"\n if modules is None:\n modules = [\"default-wbic\"]\n\n log.info(\"Launching SLURM job: {}\", jobname)\n script = \"\"\"#!/bin/bash\n\n#! Name of the job:\n#SBATCH -J {jobname}\n\n#! Which project should jobs run under:\n#SBATCH -A {project}\n\n#! What QoS [Quality of Service] should the job run in?\n#SBATCH --qos={qos}\n\n#! How much resource should be allocated?\n#SBATCH --tasks-per-node={tasks_per_node}\n#SBATCH --cpus-per-task={cpus_per_task}\n\n#! Memory requirements\n#SBATCH --mem={memory_mb}\n\n#! How much wall-clock time will be required?\n#SBATCH --time={duration}\n\n#! What e-mail address to use for notifications?\n#SBATCH --mail-user={email}\n\n#! What types of email messages do you wish to receive?\n#SBATCH --mail-type=ALL\n\n#! Uncomment this to prevent the job from being requeued (e.g. if\n#! interrupted by node failure or system downtime):\n#! SBATCH --no-requeue\n\n#! Partition\n{partition_cmd}\n\n#! sbatch directives end here (put any additional directives above this line)\n\n#! ############################################################\n#! Modify the settings below to specify the application's environment, location\n#! and launch method:\n\n#! Optionally modify the environment seen by the application\n#! (note that SLURM reproduces the environment at submission irrespective of ~/.bashrc):\n. /etc/profile.d/modules.sh # Leave this line (enables the module command)\nmodule purge # Removes all modules still loaded\nmodule load {modules} # Basic one, e.g. default-wbic, is REQUIRED - loads the basic environment\n\n#! Insert additional module load commands after this line if needed:\n\n#! Full path to your application executable:\napplication=\"hostname\"\n\n#! Run options for the application:\noptions=\"\"\n\n#! Work directory (i.e. where the job will run):\nworkdir=\"$SLURM_SUBMIT_DIR\" # The value of SLURM_SUBMIT_DIR sets workdir to the directory\n # in which sbatch is run.\n\n#! Are you using OpenMP (NB this is **unrelated to OpenMPI**)? If so increase this\n#! safe value to no more than 24:\nexport OMP_NUM_THREADS=24\n\n# Command line to be submited by SLURM:\nCMD=\"{cmd}\"\n\n###############################################################\n### You should not have to change anything below this line ####\n###############################################################\n\ncd $workdir\necho -e \"Changed directory to `pwd`.\\n\"\n\nJOBID=$SLURM_JOB_ID\n\necho -e \"JobID: $JOBID\\n======\"\necho \"Time: `date`\"\necho \"Running on master node: `hostname`\"\necho \"Current directory: `pwd`\"\n\nif [ \"$SLURM_JOB_NODELIST\" ]; then\n #! Create a machine file:\n export NODEFILE=`/usr/bin/generate_pbs_nodefile`\n cat $NODEFILE | uniq > machine.file.$JOBID\n echo -e \"\\nNodes allocated:\\n================\"\n echo `cat machine.file.$JOBID | sed -e 's/\\..*$//g'`\nfi\n\necho -e \"\\nExecuting command:\\n==================\\n$CMD\\n\"\n\neval $CMD\n \"\"\".format( # noqa\n cmd=cmd,\n cpus_per_task=cpus_per_task,\n duration=strfdelta(duration, SLURM_TIMEDELTA_FMT),\n email=email,\n jobname=jobname,\n memory_mb=memory_mb,\n modules=\" \".join(modules),\n partition_cmd=partition_cmd,\n project=project,\n qos=qos,\n tasks_per_node=tasks_per_node,\n )\n cmdargs = [\"sbatch\"]\n with pushd(directory):\n p = Popen(cmdargs, stdin=PIPE)\n p.communicate(input=script.encode(encoding))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef launch_cambridge_hphi(\n jobname: str,\n cmd: str,\n memory_mb: int,\n qos: str,\n email: str,\n duration: timedelta,\n cpus_per_task: int,\n project: str = \"hphi\",\n tasks_per_node: int = 1,\n partition: str = \"wbic-cs\", # 2018-02: was \"wbic\", now \"wbic-cs\"\n modules: List[str] = None,\n directory: str = os.getcwd(),\n encoding: str = \"ascii\") -> None:\n \"\"\"\n Specialization of :func:`launch_slurm` (q.v.) with defaults for the\n University of Cambridge WBIC HPHI.\n \"\"\"\n if modules is None:\n modules = [\"default-wbic\"]\n launch_slurm(\n cmd=cmd,\n cpus_per_task=cpus_per_task,\n directory=directory,\n duration=duration,\n email=email,\n encoding=encoding,\n jobname=jobname,\n memory_mb=memory_mb,\n modules=modules,\n partition=partition,\n project=project,\n qos=qos,\n tasks_per_node=tasks_per_node,\n )", "response": "Launch a Cambridge HPHI."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert a drug name to a : class :. Drug class.", "response": "def get_drug(drug_name: str,\n name_is_generic: bool = False,\n include_categories: bool = False) -> Optional[Drug]:\n \"\"\"\n Converts a drug name to a :class:`.Drug` class.\n\n If you already have the generic name, you can get the Drug more\n efficiently by setting ``name_is_generic=True``.\n\n Set ``include_categories=True`` to include drug categories (such as\n tricyclics) as well as individual drugs.\n \"\"\"\n drug_name = drug_name.strip().lower()\n if name_is_generic:\n drug = DRUGS_BY_GENERIC_NAME.get(drug_name) # type: Optional[Drug]\n if drug is not None and drug.category_not_drug and not include_categories: # noqa\n return None\n return drug\n else:\n for d in DRUGS:\n if d.name_matches(drug_name):\n return d\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting a drug name to its generic equivalent.", "response": "def drug_name_to_generic(drug_name: str,\n unknown_to_default: bool = False,\n default: str = None,\n include_categories: bool = False) -> str:\n \"\"\"\n Converts a drug name to the name of its generic equivalent.\n \"\"\"\n drug = get_drug(drug_name, include_categories=include_categories)\n if drug is not None:\n return drug.generic_name\n return default if unknown_to_default else drug_name"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert a list of drug names to their generic equivalents.", "response": "def drug_names_to_generic(drugs: List[str],\n unknown_to_default: bool = False,\n default: str = None,\n include_categories: bool = False) -> List[str]:\n \"\"\"\n Converts a list of drug names to their generic equivalents.\n\n The arguments are as for :func:`drug_name_to_generic` but this function\n handles a list of drug names rather than a single one.\n\n Note in passing the following conversion of blank-type representations from\n R via ``reticulate``, when using e.g. the ``default`` parameter and storing\n results in a ``data.table()`` character column:\n\n .. code-block:: none\n\n ------------------------------ ----------------\n To Python Back from Python\n ------------------------------ ----------------\n [not passed, so Python None] \"NULL\"\n NULL \"NULL\"\n NA_character_ \"NA\"\n NA TRUE (logical)\n ------------------------------ ----------------\n\n \"\"\"\n return [\n drug_name_to_generic(drug,\n unknown_to_default=unknown_to_default,\n default=default,\n include_categories=include_categories)\n for drug in drugs\n ]"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndetermines whether a drug instance matches the specified criteria.", "response": "def drug_matches_criteria(drug: Drug, **criteria: Dict[str, bool]) -> bool:\n \"\"\"\n Determines whether a drug, passed as an instance of :class:`.Drug`, matches\n the specified criteria.\n\n Args:\n drug: a :class:`.Drug` instance\n criteria: ``name=value`` pairs to match against the attributes of\n the :class:`Drug` class. For example, you can include keyword\n arguments like ``antidepressant=True``.\n \"\"\"\n for attribute, value in criteria.items():\n if getattr(drug, attribute) != value:\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef all_drugs_where(sort=True,\n include_categories: bool = False,\n **criteria: Dict[str, bool]) -> List[Drug]:\n \"\"\"\n Find all drugs matching the specified criteria (see\n :func:`drug_matches_criteria`). If ``include_categories`` is true, then\n drug categories (like \"tricyclics\") are included as well as individual\n drugs.\n\n Pass keyword arguments such as\n\n .. code-block:: python\n\n from cardinal_pythonlib.psychiatry.drugs import *\n non_ssri_antidep = all_drugs_where(antidepressant=True, ssri=False)\n print([d.generic_name for d in non_ssri_antidep])\n conventional_antidep = all_drugs_where(conventional_antidepressant=True)\n print([d.generic_name for d in conventional_antidep])\n \"\"\"\n matching_drugs = [] # type: List[Drug]\n for drug in DRUGS:\n if drug.category_not_drug and not include_categories:\n continue\n if drug_matches_criteria(drug, **criteria):\n matching_drugs.append(drug)\n if sort:\n matching_drugs.sort(key=lambda d: d.generic_name)\n return matching_drugs", "response": "Returns a list of all the drugs that match the criteria."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef drug_name_matches_criteria(drug_name: str,\n name_is_generic: bool = False,\n include_categories: bool = False,\n **criteria: Dict[str, bool]) -> bool:\n \"\"\"\n Establish whether a single drug, passed by name, matches the specified\n criteria. See :func:`drug_matches_criteria`.\n \"\"\"\n drug = get_drug(drug_name, name_is_generic)\n if drug is None:\n return False\n if drug.category_not_drug and not include_categories:\n return False\n return drug_matches_criteria(drug, **criteria)", "response": "Determines whether a single drug passed by name matches the specified criteria."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nestablish whether multiple drugs passed as a list of drug names each racket matches the specified criteria.", "response": "def drug_names_match_criteria(drug_names: List[str],\n names_are_generic: bool = False,\n include_categories: bool = False,\n **criteria: Dict[str, bool]) -> List[bool]:\n \"\"\"\n Establish whether multiple drugs, passed as a list of drug names, each\n matches the specified criteria. See :func:`drug_matches_criteria`.\n \"\"\"\n return [\n drug_name_matches_criteria(\n dn,\n name_is_generic=names_are_generic,\n include_categories=include_categories,\n **criteria)\n for dn in drug_names\n ]"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the regex text for this drug.", "response": "def regex_text(self) -> str:\n \"\"\"\n Return regex text (yet to be compiled) for this drug.\n \"\"\"\n if self._regex_text is None:\n possibilities = [] # type: List[str]\n for p in list(set(self.all_generics + self.alternatives)):\n if self.add_preceding_word_boundary and not p.startswith(WB):\n p = WB + p\n if self.add_preceding_wildcards and not p.startswith(WILDCARD):\n p = WILDCARD + p\n if self.add_following_wildcards and not p.endswith(WILDCARD):\n p = p + WILDCARD\n possibilities.append(p)\n self._regex_text = \"|\".join(\"(?:\" + x + \")\" for x in possibilities)\n return self._regex_text"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef regex(self) -> Pattern:\n if self._regex is None:\n self._regex = re.compile(self.regex_text,\n re.IGNORECASE | re.DOTALL)\n return self._regex", "response": "Returns a compiled regex for this drug."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting a regular expression text to a reasonably close fragment of SQL string.", "response": "def regex_to_sql_like(regex_text: str,\n single_wildcard: str = \"_\",\n zero_or_more_wildcard: str = \"%\") -> List[str]:\n \"\"\"\n Converts regular expression text to a reasonably close fragment\n for the SQL ``LIKE`` operator.\n\n NOT PERFECT, but works for current built-in regular expressions.\n\n Args:\n regex_text: regular expression text to work with\n single_wildcard: SQL single wildcard, typically an underscore\n zero_or_more_wildcard: SQL \"zero/one/many\" wildcard, probably always\n a percent symbol\n\n Returns:\n string for an SQL string literal\n\n Raises:\n :exc:`ValueError` for some regex text that it doesn't understand\n properly\n \"\"\"\n def append_to_all(new_content: str) -> None:\n nonlocal results\n results = [r + new_content for r in results]\n\n def split_and_append(new_options: List[str]) -> None:\n nonlocal results\n newresults = [] # type: List[str]\n for option in new_options:\n newresults.extend([r + option for r in results])\n results = newresults\n\n def deduplicate_wildcards(text: str) -> str:\n while zero_or_more_wildcard + zero_or_more_wildcard in text:\n text = text.replace(\n zero_or_more_wildcard + zero_or_more_wildcard,\n zero_or_more_wildcard)\n return text\n\n # Basic processing\n working = regex_text # strings are immutable\n results = [zero_or_more_wildcard] # start with a wildcard\n\n while working:\n if working.startswith(\".*\"):\n # e.g. \".*ozapi\"\n append_to_all(zero_or_more_wildcard)\n working = working[2:]\n elif working.startswith(\"[\"):\n # e.g. \"[io]peridol\"\n close_bracket = working.index(\"]\") # may raise\n bracketed = working[1:close_bracket]\n option_groups = bracketed.split(\"|\")\n options = [c for group in option_groups for c in group]\n split_and_append(options)\n working = working[close_bracket + 1:]\n elif len(working) > 1 and working[1] == \"?\":\n # e.g. \"r?azole\"\n split_and_append([\"\", working[0]])\n # ... regex \"optional character\"\n # ... SQL: some results with a single wildcard, some without\n working = working[2:]\n elif working.startswith(\".\"):\n # single character wildcard\n append_to_all(single_wildcard)\n working = working[1:]\n else:\n append_to_all(working[0])\n working = working[1:]\n append_to_all(zero_or_more_wildcard) # end with a wildcard\n\n # Remove any duplicate (consecutive) % wildcards:\n results = [deduplicate_wildcards(r) for r in results]\n\n # Done\n return results"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns all the SQL - like fragments that are used to compare the database column names with the SQL - LIKE operator.", "response": "def sql_like_fragments(self) -> List[str]:\n \"\"\"\n Returns all the string literals to which a database column should be\n compared using the SQL ``LIKE`` operator, to match this drug.\n\n This isn't as accurate as the regex, but ``LIKE`` can do less.\n\n ``LIKE`` uses the wildcards ``?`` and ``%``.\n \"\"\"\n if self._sql_like_fragments is None:\n self._sql_like_fragments = []\n for p in list(set(self.all_generics + self.alternatives)):\n self._sql_like_fragments.extend(self.regex_to_sql_like(p))\n return self._sql_like_fragments"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef name_matches(self, name: str) -> bool:\n return bool(self.regex.match(name))", "response": "Determines whether the given name matches our knowledge of any of the related objects."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sql_column_like_drug(self, column_name: str) -> str:\n clauses = [\n \"{col} LIKE {fragment}\".format(\n col=column_name,\n fragment=sql_string_literal(f))\n for f in self.sql_like_fragments\n ]\n return \"({})\".format(\" OR \".join(clauses))", "response": "Returns the SQL like clause for the drug column_name."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef quaternion(vector, angle):\n return N.cos(angle/2)+vector*N.sin(angle/2)", "response": "Returns the unit quaternion for a vector and an angle."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a parameterized set of vectors defining an ellipse.", "response": "def ellipse(n=1000, adaptive=False):\n \"\"\"\n Get a parameterized set of vectors defining\n ellipse for a major and minor axis length.\n Resulting vector bundle has major axes\n along axes given.\n \"\"\"\n u = N.linspace(0,2*N.pi,n)\n # Get a bundle of vectors defining\n # a full rotation around the unit circle\n return N.array([N.cos(u),N.sin(u)]).T"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn major axes of error ellipse or rescaled using chi2 test statistic", "response": "def scale_errors(cov_axes, confidence_level=0.95):\n \"\"\"\n Returns major axes of error ellipse or\n hyperbola, rescaled using chi2 test statistic\n \"\"\"\n dof = len(cov_axes)\n x2t = chi2.ppf(confidence_level,dof)\n return N.sqrt(x2t*cov_axes)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef normal_errors(axes, covariance_matrix, **kwargs):\n level = kwargs.pop('level',1)\n traditional_layout = kwargs.pop('traditional_layout',True)\n d = N.diagonal(covariance_matrix)\n ell = ellipse(**kwargs)\n\n if axes[2,2] < 0:\n axes *= -1\n\n # Not sure where this factor comes from but it\n # seems to make things work better\n c1 = 2\n axis_lengths = d[:2]\n f = N.linalg.norm(\n ell*axis_lengths,axis=1)\n\n e0 = -ell.T*d[2]*c1\n e = N.vstack((e0,f))\n\n _ = dot(e.T,axes).T\n\n if traditional_layout:\n lon,lat = stereonet_math.cart2sph(_[2],_[0],-_[1])\n else:\n lon,lat = stereonet_math.cart2sph(-_[1],_[0],_[2])\n return list(zip(lon,lat))", "response": "Returns a list of lon lat values for a set of normal axes."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef plane_errors(axes, covariance_matrix, sheet='upper',**kwargs):\n\n level = kwargs.pop('level',1)\n traditional_layout = kwargs.pop('traditional_layout',True)\n\n d = N.sqrt(covariance_matrix)\n\n ell = ellipse(**kwargs)\n bundle = dot(ell, d[:2])\n res = d[2]*level\n\n # Switch hemispheres if PCA is upside-down\n # Normal vector is always correctly fit\n #if traditional_layout:\n #if axes[2,2] > 0:\n\n if axes[2,2] > 0:\n res *= -1\n\n if sheet == 'upper':\n bundle += res\n elif sheet == 'lower':\n bundle -= res\n\n _ = dot(bundle,axes).T\n\n if traditional_layout:\n lon,lat = stereonet_math.cart2sph(_[2],_[0],_[1])\n else:\n lon,lat = stereonet_math.cart2sph(-_[1],_[0],_[2])\n\n return list(zip(lon,lat))", "response": "Returns a list of lon lat values for each plane in the diagram."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef iterative_plane_errors(axes,covariance_matrix, **kwargs):\n sheet = kwargs.pop('sheet','upper')\n level = kwargs.pop('level',1)\n n = kwargs.pop('n',100)\n\n cov = N.sqrt(N.diagonal(covariance_matrix))\n u = N.linspace(0, 2*N.pi, n)\n\n scales = dict(upper=1,lower=-1,nominal=0)\n c1 = scales[sheet]\n c1 *= -1 # We assume upper hemisphere\n if axes[2,2] < 0:\n c1 *= -1\n\n def sdot(a,b):\n return sum([i*j for i,j in zip(a,b)])\n\n def step_func(a):\n e = [\n N.cos(a)*cov[0],\n N.sin(a)*cov[1],\n c1*cov[2]]\n d = [sdot(e,i)\n for i in axes.T]\n x,y,z = d[2],d[0],d[1]\n r = N.sqrt(x**2 + y**2 + z**2)\n lat = N.arcsin(z/r)\n lon = N.arctan2(y, x)\n return lon,lat\n\n # Get a bundle of vectors defining\n # a full rotation around the unit circle\n return N.array([step_func(i)\n for i in u])", "response": "This function computes an iterative version of pca. plane_errors which computes an error surface for a plane."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef quote_identifier_if_required(cls, identifier: str,\n debug_force_quote: bool = False) -> str:\n \"\"\"\n Transforms a SQL identifier (such as a column name) into a version\n that is quoted if required (or, with ``debug_force_quote=True``, is\n quoted regardless).\n \"\"\"\n if debug_force_quote:\n return cls.quote_identifier(identifier)\n if cls.is_quoted(identifier):\n return identifier\n if cls.requires_quoting(identifier):\n return cls.quote_identifier(identifier)\n return identifier", "response": "Transforms a SQL identifier into a version\n that is quoted if required."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd points and converts to topojson string.", "response": "def to_topojson(self):\n \"\"\"Adds points and converts to topojson string.\"\"\"\n topojson = self.topojson\n topojson[\"objects\"][\"points\"] = {\n \"type\": \"GeometryCollection\",\n \"geometries\": [point.to_topojson() for point in self.points.all()],\n }\n return json.dumps(topojson)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nposting an element to Netuitive.", "response": "def post(self, element):\n \"\"\"\n :param element: Element to post to Netuitive\n :type element: object\n \"\"\"\n\n try:\n\n if self.disabled is True:\n element.clear_samples()\n logging.error('Posting has been disabled. '\n 'See previous errors for details.')\n return(False)\n\n if element.id is None:\n raise Exception('element id is not set')\n\n element.merge_metrics()\n payload = json.dumps(\n [element], default=lambda o: o.__dict__, sort_keys=True)\n logging.debug(payload)\n\n headers = {'Content-Type': 'application/json',\n 'User-Agent': self.agent}\n request = urllib2.Request(\n self.dataurl, data=payload, headers=headers)\n resp = urllib2.urlopen(request)\n logging.debug(\"Response code: %d\", resp.getcode())\n\n resp.close()\n\n self.post_error_count = 0\n\n return(True)\n\n except urllib2.HTTPError as e:\n logging.debug(\"Response code: %d\", e.code)\n\n if e.code in self.kill_codes:\n self.disabled = True\n\n logging.exception('Posting has been disabled.'\n 'See previous errors for details.')\n else:\n self.post_error_count += 1\n if self.post_error_count > self.max_post_errors:\n element.clear_samples()\n\n logging.exception(\n 'error posting payload to api ingest endpoint (%s): %s',\n self.dataurl, e)\n\n except Exception as e:\n self.post_error_count += 1\n if self.post_error_count > self.max_post_errors:\n element.clear_samples() # pragma: no cover\n\n logging.exception(\n 'error posting payload to api ingest endpoint (%s): %s',\n self.dataurl, e)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\npost an event to Netuitive .", "response": "def post_event(self, event):\n \"\"\"\n :param event: Event to post to Netuitive\n :type event: object\n \"\"\"\n\n if self.disabled is True:\n logging.error('Posting has been disabled. '\n 'See previous errors for details.')\n return(False)\n\n payload = json.dumps(\n [event], default=lambda o: o.__dict__, sort_keys=True)\n logging.debug(payload)\n try:\n headers = {'Content-Type': 'application/json',\n 'User-Agent': self.agent}\n request = urllib2.Request(\n self.eventurl, data=payload, headers=headers)\n resp = urllib2.urlopen(request)\n logging.debug(\"Response code: %d\", resp.getcode())\n resp.close()\n\n return(True)\n\n except urllib2.HTTPError as e:\n logging.debug(\"Response code: %d\", e.code)\n\n if e.code in self.kill_codes:\n self.disabled = True\n logging.exception('Posting has been disabled.'\n 'See previous errors for details.')\n else:\n logging.exception(\n 'error posting payload to api ingest endpoint (%s): %s',\n self.eventurl, e)\n\n except Exception as e:\n logging.exception(\n 'error posting payload to api ingest endpoint (%s): %s',\n self.eventurl, e)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\npost a check to the Metricly API.", "response": "def post_check(self, check):\n \"\"\"\n :param check: Check to post to Metricly\n :type check: object\n \"\"\"\n\n if self.disabled is True:\n logging.error('Posting has been disabled. '\n 'See previous errors for details.')\n return(False)\n\n url = self.checkurl + '/' \\\n + check.name + '/' \\\n + check.elementId + '/' \\\n + str(check.ttl)\n headers = {'User-Agent': self.agent}\n try:\n request = urllib2.Request(\n url, data='', headers=headers)\n resp = self._repeat_request(request, self.connection_timeout)\n logging.debug(\"Response code: %d\", resp.getcode())\n resp.close()\n\n return(True)\n\n except urllib2.HTTPError as e:\n logging.debug(\"Response code: %d\", e.code)\n\n if e.code in self.kill_codes:\n self.disabled = True\n logging.exception('Posting has been disabled.'\n 'See previous errors for details.')\n else:\n logging.exception(\n 'HTTPError posting payload to api ingest endpoint'\n + ' (%s): %s',\n url, e)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef urlsplit(url):\n p = '((?P<scheme>.*)?.*://)?(?P<host>[^:/ ]+).?(?P<port>[0-9]*).*'\n m = re.search(p, url)\n scheme = m.group('scheme')\n host = m.group('host')\n port = m.group('port')\n\n _scheme, _netloc, path, query, _fragment = tuple(py_urlsplit(url))\n\n return scheme, host, port, path, query", "response": "Split url into scheme host port path query fragment"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngenerate URI from parameters.", "response": "def generate_url(scheme=None, host=None, port=None, path=None, query=None):\n \"\"\"\n Generate URI from parameters.\n\n :param str scheme:\n :param str host:\n :param int port:\n :param str path:\n :param dict query:\n :return:\n \"\"\"\n url = \"\"\n if scheme is not None:\n url += \"%s://\" % scheme\n if host is not None:\n url += host\n if port is not None:\n url += \":%s\" % str(port)\n if path is not None:\n url += ensure_url_path_starts_with_slash(path)\n if query is not None:\n url += \"?%s\" % (urlencode(query))\n return url"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndownload a file from the given url to the given directory.", "response": "def download(self, output_dir, url, overwrite):\n \"\"\" Dowload file to /tmp \"\"\"\n tmp = self.url2tmp(output_dir, url)\n if os.path.isfile(tmp) and not overwrite:\n logging.info(\"File {0} already exists. Skipping download.\".format(tmp))\n return tmp\n f = open(tmp, 'wb')\n logging.info(\"Downloading {0}\".format(url))\n res = requests.get(url, stream=True)\n if res.status_code != 200:\n # failed to download, cleanup and raise exception\n f.close()\n os.remove(tmp)\n error = \"{0}\\n\\nFailed to download < {0} >\".format(res.content, url)\n raise IOError(error)\n for block in res.iter_content(1024):\n f.write(block)\n f.close()\n return tmp"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nextracting POI nodes from osm pbf extract", "response": "def extract(self, pbf, output):\n \"\"\" extract POI nodes from osm pbf extract \"\"\"\n logging.info(\"Extracting POI nodes from {0} to {1}\".format(pbf, output))\n with open(output, 'w') as f:\n # define callback for each node that is processed\n def nodes_callback(nodes):\n for node in nodes:\n node_id, tags, coordinates = node\n # if any tags have a matching key then write record\n if any([t in tags for t in POI_TAGS]):\n f.write(json.dumps(dict(tags=tags, coordinates=coordinates)))\n f.write('\\n')\n parser = OSMParser(concurrency=4, nodes_callback=nodes_callback)\n parser.parse(pbf)\n return output"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting url to tmp filename", "response": "def url2tmp(self, root, url):\n \"\"\" convert url path to filename \"\"\"\n filename = url.rsplit('/', 1)[-1]\n return os.path.join(root, filename)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nprompt the user with a default. Returns user input from stdin.", "response": "def ask_user(prompt: str, default: str = None) -> Optional[str]:\n \"\"\"\n Prompts the user, with a default. Returns user input from ``stdin``.\n \"\"\"\n if default is None:\n prompt += \": \"\n else:\n prompt += \" [\" + default + \"]: \"\n result = input(prompt)\n return result if len(result) > 0 else default"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_save_as_filename(defaultfilename: str,\n defaultextension: str,\n title: str = \"Save As\") -> str:\n \"\"\"\n Provides a GUI \"Save As\" dialogue (via ``tkinter``) and returns the\n filename.\n \"\"\"\n root = tkinter.Tk() # create and get Tk topmost window\n # (don't do this too early; the command prompt loses focus)\n root.withdraw() # won't need this; this gets rid of a blank Tk window\n root.attributes('-topmost', True) # makes the tk window topmost\n filename = filedialog.asksaveasfilename(\n initialfile=defaultfilename,\n defaultextension=defaultextension,\n parent=root,\n title=title\n )\n root.attributes('-topmost', False) # stop the tk window being topmost\n return filename", "response": "Provides a GUI Save As dialogue and returns the filename."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nprovide a GUI Open dialogue and returns the filename.", "response": "def get_open_filename(defaultfilename: str,\n defaultextension: str,\n title: str = \"Open\") -> str:\n \"\"\"\n Provides a GUI \"Open\" dialogue (via ``tkinter``) and returns the filename.\n \"\"\"\n root = tkinter.Tk() # create and get Tk topmost window\n # (don't do this too early; the command prompt loses focus)\n root.withdraw() # won't need this; this gets rid of a blank Tk window\n root.attributes('-topmost', True) # makes the tk window topmost\n filename = filedialog.askopenfilename(\n initialfile=defaultfilename,\n defaultextension=defaultextension,\n parent=root,\n title=title\n )\n root.attributes('-topmost', False) # stop the tk window being topmost\n return filename"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef str2bool(v: str) -> bool:\n lv = v.lower()\n if lv in ('yes', 'true', 't', 'y', '1'):\n return True\n elif lv in ('no', 'false', 'f', 'n', '0'):\n return False\n else:\n raise ArgumentTypeError('Boolean value expected.')", "response": "Convert a string to a boolean value."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncall to abort the request on timeout", "response": "def abort_request(self, request):\n \"\"\"Called to abort request on timeout\"\"\"\n self.timedout = True\n try:\n request.cancel()\n except error.AlreadyCancelled:\n return"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nmake an HTTP request and return the body", "response": "def getBody(cls, url, method='GET', headers={}, data=None, socket=None, timeout=120):\n \"\"\"Make an HTTP request and return the body\n \"\"\"\n if not 'User-Agent' in headers:\n headers['User-Agent'] = ['Tensor HTTP checker']\n\n return cls().request(url, method, headers, data, socket, timeout)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef getJson(cls, url, method='GET', headers={}, data=None, socket=None, timeout=120):\n if not 'Content-Type' in headers:\n headers['Content-Type'] = ['application/json']\n\n body = yield cls().getBody(url, method, headers, data, socket, timeout)\n\n defer.returnValue(json.loads(body))", "response": "Fetch a JSON result via HTTP HTTP\n "} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the covariance matrix aligned to the fit.", "response": "def aligned_covariance(fit, type='noise'):\n \"\"\"\n Covariance rescaled so that eigenvectors sum to 1\n and rotated into data coordinates from PCA space\n \"\"\"\n cov = fit._covariance_matrix(type)\n # Rescale eigenvectors to sum to 1\n cov /= N.linalg.norm(cov)\n return dot(fit.axes,cov)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfinding the angle between the nominal vectors fit1 and fit2.", "response": "def fit_angle(fit1, fit2, degrees=True):\n \"\"\"\n Finds the angle between the nominal vectors\n \"\"\"\n return N.degrees(angle(fit1.normal,fit2.normal))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncalculate the similarity between two fit objects.", "response": "def fit_similarity(fit1, fit2):\n \"\"\"\n Distance apart for vectors, given in\n standard deviations\n \"\"\"\n cov1 = aligned_covariance(fit1)\n cov2 = aligned_covariance(fit2)\n if fit2.axes[2,2] < 0:\n cov2 *= -1\n v0 = fit1.normal-fit2.normal\n cov0 = cov1+cov2 # Axes are aligned, so no covariances\n\n # Find distance of point from center\n\n # Decompose covariance matrix\n U,s,V = N.linalg.svd(cov0)\n rotated = dot(V,v0) # rotate vector into PCA space\n val = rotated**2/N.sqrt(s)\n return N.sqrt(val.sum())"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef axis_aligned_transforms():\n I = N.eye(3)\n xy = I[:2]\n xz = N.vstack((I[0],I[2]))\n yz = I[1:]\n return xy,xz,yz", "response": "Generates three transformation matrices to map three - dimensional data on the xy xz and yz planes respectively."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef png_img_html_from_pyplot_figure(fig: \"Figure\",\n dpi: int = 100,\n extra_html_class: str = None) -> str:\n \"\"\"\n Converts a ``pyplot`` figure to an HTML IMG tag with encapsulated PNG.\n \"\"\"\n if fig is None:\n return \"\"\n # Make a file-like object\n memfile = io.BytesIO()\n # In general, can do\n # fig.savefig(filename/file-like-object/backend, format=...)\n # or\n # backend.savefig(fig):\n # see e.g. http://matplotlib.org/api/backend_pdf_api.html\n fig.savefig(memfile, format=\"png\", dpi=dpi)\n memfile.seek(0)\n pngblob = memoryview(memfile.read())\n return rnc_web.get_png_img_html(pngblob, extra_html_class)", "response": "Converts a pyplot figure to an HTML IMG tag with encapsulated PNG."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts a pyplot figure to an SVG tag.", "response": "def svg_html_from_pyplot_figure(fig: \"Figure\") -> str:\n \"\"\"\n Converts a ``pyplot`` figure to an SVG tag.\n \"\"\"\n if fig is None:\n return \"\"\n memfile = io.BytesIO() # StringIO doesn't like mixing str/unicode\n fig.savefig(memfile, format=\"svg\")\n return memfile.getvalue().decode(\"utf-8\")"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nset the current font size within the matplotlib library.", "response": "def set_matplotlib_fontsize(matplotlib: ModuleType,\n fontsize: Union[int, float] = 12) -> None:\n \"\"\"\n Sets the current font size within the ``matplotlib`` library.\n\n **WARNING:** not an appropriate method for multithreaded environments, as\n it writes (indirectly) to ``matplotlib`` global objects. See CamCOPS for\n alternative methods.\n \"\"\"\n font = {\n # http://stackoverflow.com/questions/3899980\n # http://matplotlib.org/users/customizing.html\n 'family': 'sans-serif',\n # ... serif, sans-serif, cursive, fantasy, monospace\n 'style': 'normal', # normal (roman), italic, oblique\n 'variant': 'normal', # normal, small-caps\n 'weight': 'normal',\n # ... normal [=400], bold [=700], bolder [relative to current],\n # lighter [relative], 100, 200, 300, ..., 900\n 'size': fontsize # in pt (default 12)\n }\n # noinspection PyUnresolvedReferences\n matplotlib.rc('font', **font)\n legend = {\n # http://stackoverflow.com/questions/7125009\n 'fontsize': fontsize\n }\n # noinspection PyUnresolvedReferences\n matplotlib.rc('legend', **legend)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef massage_keys(keys):\n m_keys = []\n for key in keys:\n if key.startswith('keybase:'):\n m_keys.append(cryptorito.key_from_keybase(key[8:])['fingerprint'])\n else:\n m_keys.append(key)\n\n return m_keys", "response": "Takes a list of GPG keys and returns a list of GPG keys."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef encrypt_file(src, dest, csv_keys):\n keys = massage_keys(csv_keys.split(','))\n cryptorito.encrypt(src, dest, keys)", "response": "Encrypt a file with the specific GPG keys and write out\n to the specified path"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nencrypting what comes in from stdin and return base64 encrypted against the specified keys returning on stdout", "response": "def encrypt_var(csv_keys):\n \"\"\"Encrypt what comes in from stdin and return base64\n encrypted against the specified keys, returning on stdout\"\"\"\n keys = massage_keys(csv_keys.split(','))\n data = sys.stdin.read()\n encrypted = cryptorito.encrypt_var(data, keys)\n print(cryptorito.portable_b64encode(encrypted))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nimporting a public GPG key from Keybase", "response": "def import_keybase(useropt):\n \"\"\"Imports a public GPG key from Keybase\"\"\"\n public_key = None\n u_bits = useropt.split(':')\n username = u_bits[0]\n if len(u_bits) == 1:\n public_key = cryptorito.key_from_keybase(username)\n else:\n fingerprint = u_bits[1]\n public_key = cryptorito.key_from_keybase(username, fingerprint)\n\n if cryptorito.has_gpg_key(public_key['fingerprint']):\n sys.exit(2)\n\n cryptorito.import_gpg_key(public_key['bundle'].encode('ascii'))\n sys.exit(0)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef do_thing():\n if len(sys.argv) == 5 and sys.argv[1] == \"encrypt_file\":\n encrypt_file(sys.argv[2], sys.argv[3], sys.argv[4])\n elif len(sys.argv) == 4 and sys.argv[1] == \"decrypt_file\":\n decrypt_file(sys.argv[2], sys.argv[3])\n elif len(sys.argv) == 3 and sys.argv[1] == \"encrypt\":\n encrypt_var(sys.argv[2])\n elif len(sys.argv) == 2 and sys.argv[1] == \"decrypt\":\n decrypt_var()\n elif len(sys.argv) == 3 and sys.argv[1] == \"decrypt\":\n decrypt_var(passphrase=sys.argv[2])\n elif len(sys.argv) == 3 and sys.argv[1] == \"has_key\":\n has_key(sys.argv[2])\n elif len(sys.argv) == 3 and sys.argv[1] == \"import_keybase\":\n import_keybase(sys.argv[2])\n elif len(sys.argv) == 3 and sys.argv[1] == \"export\":\n export_key(sys.argv[2])\n else:\n print(\"Cryptorito testing wrapper. Not suitable for routine use.\",\n file=sys.stderr)\n sys.exit(1)", "response": "Execute command line cryptorito actions"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_monochrome_handler(\n extranames: List[str] = None,\n with_process_id: bool = False,\n with_thread_id: bool = False,\n stream: TextIO = None) -> logging.StreamHandler:\n \"\"\"\n Gets a monochrome log handler using a standard format.\n\n Args:\n extranames: additional names to append to the logger's name\n with_process_id: include the process ID in the logger's name?\n with_thread_id: include the thread ID in the logger's name?\n stream: ``TextIO`` stream to send log output to\n\n Returns:\n the :class:`logging.StreamHandler`\n\n \"\"\"\n fmt = \"%(asctime)s.%(msecs)03d\"\n if with_process_id or with_thread_id:\n procinfo = [] # type: List[str]\n if with_process_id:\n procinfo.append(\"p%(process)d\")\n if with_thread_id:\n procinfo.append(\"t%(thread)d\")\n fmt += \" [{}]\".format(\".\".join(procinfo))\n extras = \":\" + \":\".join(extranames) if extranames else \"\"\n fmt += \" %(name)s{extras}:%(levelname)s: \".format(extras=extras)\n fmt += \"%(message)s\"\n f = logging.Formatter(fmt, datefmt=LOG_DATEFMT, style='%')\n h = logging.StreamHandler(stream)\n h.setFormatter(f)\n return h", "response": "Returns a Monochrome log handler using a standard format."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a colour log handler using a standard format.", "response": "def get_colour_handler(extranames: List[str] = None,\n with_process_id: bool = False,\n with_thread_id: bool = False,\n stream: TextIO = None) -> logging.StreamHandler:\n \"\"\"\n Gets a colour log handler using a standard format.\n\n Args:\n extranames: additional names to append to the logger's name\n with_process_id: include the process ID in the logger's name?\n with_thread_id: include the thread ID in the logger's name?\n stream: ``TextIO`` stream to send log output to\n\n Returns:\n the :class:`logging.StreamHandler`\n\n \"\"\"\n fmt = \"%(white)s%(asctime)s.%(msecs)03d\" # this is dim white = grey\n if with_process_id or with_thread_id:\n procinfo = [] # type: List[str]\n if with_process_id:\n procinfo.append(\"p%(process)d\")\n if with_thread_id:\n procinfo.append(\"t%(thread)d\")\n fmt += \" [{}]\".format(\".\".join(procinfo))\n extras = \":\" + \":\".join(extranames) if extranames else \"\"\n fmt += \" %(name)s{extras}:%(levelname)s: \".format(extras=extras)\n fmt += \"%(reset)s%(log_color)s%(message)s\"\n cf = ColoredFormatter(fmt,\n datefmt=LOG_DATEFMT,\n reset=True,\n log_colors=LOG_COLORS,\n secondary_log_colors={},\n style='%')\n ch = logging.StreamHandler(stream)\n ch.setFormatter(cf)\n return ch"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconfigure a logger for a colour.", "response": "def configure_logger_for_colour(logger: logging.Logger,\n level: int = logging.INFO,\n remove_existing: bool = False,\n extranames: List[str] = None,\n with_process_id: bool = False,\n with_thread_id: bool = False) -> None:\n \"\"\"\n Applies a preconfigured datetime/colour scheme to a logger.\n\n Should ONLY be called from the ``if __name__ == 'main'`` script;\n see https://docs.python.org/3.4/howto/logging.html#library-config.\n\n Args:\n logger: logger to modify\n level: log level to set\n remove_existing: remove existing handlers from logger first?\n extranames: additional names to append to the logger's name\n with_process_id: include the process ID in the logger's name?\n with_thread_id: include the thread ID in the logger's name?\n \"\"\"\n if remove_existing:\n logger.handlers = [] # http://stackoverflow.com/questions/7484454\n handler = get_colour_handler(extranames,\n with_process_id=with_process_id,\n with_thread_id=with_thread_id)\n handler.setLevel(level)\n logger.addHandler(handler)\n logger.setLevel(level)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef main_only_quicksetup_rootlogger(level: int = logging.DEBUG,\n with_process_id: bool = False,\n with_thread_id: bool = False) -> None:\n \"\"\"\n Quick function to set up the root logger for colour.\n\n Should ONLY be called from the ``if __name__ == 'main'`` script;\n see https://docs.python.org/3.4/howto/logging.html#library-config.\n\n Args:\n level: log level to set\n with_process_id: include the process ID in the logger's name?\n with_thread_id: include the thread ID in the logger's name?\n \"\"\"\n # Nasty. Only call from \"if __name__ == '__main__'\" clauses!\n rootlogger = logging.getLogger()\n configure_logger_for_colour(rootlogger, level, remove_existing=True,\n with_process_id=with_process_id,\n with_thread_id=with_thread_id)", "response": "Quick function to set up the root logger for colour."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef remove_all_logger_handlers(logger: logging.Logger) -> None:\n while logger.handlers:\n h = logger.handlers[0]\n logger.removeHandler(h)", "response": "Removes all handlers from a logger."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nresetting the log format of the given logger.", "response": "def reset_logformat(logger: logging.Logger,\n fmt: str,\n datefmt: str = '%Y-%m-%d %H:%M:%S') -> None:\n \"\"\"\n Create a new formatter and apply it to the logger.\n\n :func:`logging.basicConfig` won't reset the formatter if another module\n has called it, so always set the formatter like this.\n\n Args:\n logger: logger to modify\n fmt: passed to the ``fmt=`` argument of :class:`logging.Formatter`\n datefmt: passed to the ``datefmt=`` argument of\n :class:`logging.Formatter`\n \"\"\"\n handler = logging.StreamHandler()\n formatter = logging.Formatter(fmt=fmt, datefmt=datefmt)\n handler.setFormatter(formatter)\n remove_all_logger_handlers(logger)\n logger.addHandler(handler)\n logger.propagate = False"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef reset_logformat_timestamped(logger: logging.Logger,\n extraname: str = \"\",\n level: int = logging.INFO) -> None:\n \"\"\"\n Apply a simple time-stamped log format to an existing logger, and set\n its loglevel to either ``logging.DEBUG`` or ``logging.INFO``.\n\n Args:\n logger: logger to modify\n extraname: additional name to append to the logger's name\n level: log level to set\n \"\"\"\n namebit = extraname + \":\" if extraname else \"\"\n fmt = (\"%(asctime)s.%(msecs)03d:%(levelname)s:%(name)s:\" + namebit +\n \"%(message)s\")\n # logger.info(fmt)\n reset_logformat(logger, fmt=fmt)\n # logger.info(fmt)\n logger.setLevel(level)", "response": "Reset a logger s logformat to a simple time - stamped log format."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\napply a preconfigured datetime / colour scheme to ALL loggers.", "response": "def configure_all_loggers_for_colour(remove_existing: bool = True) -> None:\n \"\"\"\n Applies a preconfigured datetime/colour scheme to ALL logger.\n\n Should ONLY be called from the ``if __name__ == 'main'`` script;\n see https://docs.python.org/3.4/howto/logging.html#library-config.\n\n Generally MORE SENSIBLE just to apply a handler to the root logger.\n\n Args:\n remove_existing: remove existing handlers from logger first?\n\n \"\"\"\n handler = get_colour_handler()\n apply_handler_to_all_logs(handler, remove_existing=remove_existing)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\napplies a handler to all logs in the root logger.", "response": "def apply_handler_to_root_log(handler: logging.Handler,\n remove_existing: bool = False) -> None:\n \"\"\"\n Applies a handler to all logs, optionally removing existing handlers.\n\n Should ONLY be called from the ``if __name__ == 'main'`` script;\n see https://docs.python.org/3.4/howto/logging.html#library-config.\n\n Generally MORE SENSIBLE just to apply a handler to the root logger.\n\n Args:\n handler: the handler to apply\n remove_existing: remove existing handlers from logger first?\n \"\"\"\n rootlog = logging.getLogger()\n if remove_existing:\n rootlog.handlers = []\n rootlog.addHandler(handler)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\napplying a handler to all logs.", "response": "def apply_handler_to_all_logs(handler: logging.Handler,\n remove_existing: bool = False) -> None:\n \"\"\"\n Applies a handler to all logs, optionally removing existing handlers.\n\n Should ONLY be called from the ``if __name__ == 'main'`` script;\n see https://docs.python.org/3.4/howto/logging.html#library-config.\n\n Generally MORE SENSIBLE just to apply a handler to the root logger.\n\n Args:\n handler: the handler to apply\n remove_existing: remove existing handlers from logger first?\n \"\"\"\n # noinspection PyUnresolvedReferences\n for name, obj in logging.Logger.manager.loggerDict.items():\n if remove_existing:\n obj.handlers = [] # http://stackoverflow.com/questions/7484454\n obj.addHandler(handler)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncopying all currently configured logs to a file.", "response": "def copy_root_log_to_file(filename: str,\n fmt: str = LOG_FORMAT,\n datefmt: str = LOG_DATEFMT) -> None:\n \"\"\"\n Copy all currently configured logs to the specified file.\n\n Should ONLY be called from the ``if __name__ == 'main'`` script;\n see https://docs.python.org/3.4/howto/logging.html#library-config.\n \"\"\"\n fh = logging.FileHandler(filename)\n # default file mode is 'a' for append\n formatter = logging.Formatter(fmt=fmt, datefmt=datefmt)\n fh.setFormatter(formatter)\n apply_handler_to_root_log(fh)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncopy all currently configured logs to a file.", "response": "def copy_all_logs_to_file(filename: str,\n fmt: str = LOG_FORMAT,\n datefmt: str = LOG_DATEFMT) -> None:\n \"\"\"\n Copy all currently configured logs to the specified file.\n\n Should ONLY be called from the ``if __name__ == 'main'`` script;\n see https://docs.python.org/3.4/howto/logging.html#library-config.\n\n Args:\n filename: file to send log output to\n fmt: passed to the ``fmt=`` argument of :class:`logging.Formatter`\n datefmt: passed to the ``datefmt=`` argument of\n :class:`logging.Formatter`\n \"\"\"\n fh = logging.FileHandler(filename)\n # default file mode is 'a' for append\n formatter = logging.Formatter(fmt=fmt, datefmt=datefmt)\n fh.setFormatter(formatter)\n apply_handler_to_all_logs(fh)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning information on a log formatter as a dictionary.", "response": "def get_formatter_report(f: logging.Formatter) -> Optional[Dict[str, str]]:\n \"\"\"\n Returns information on a log formatter, as a dictionary.\n For debugging.\n \"\"\"\n if f is None:\n return None\n return {\n '_fmt': f._fmt,\n 'datefmt': f.datefmt,\n '_style': str(f._style),\n }"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning information on a log handler as a dictionary.", "response": "def get_handler_report(h: logging.Handler) -> Dict[str, Any]:\n \"\"\"\n Returns information on a log handler, as a dictionary.\n For debugging.\n \"\"\"\n return {\n 'get_name()': h.get_name(),\n 'level': h.level,\n 'formatter': get_formatter_report(h.formatter),\n 'filters': h.filters,\n }"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn information on a log as a dictionary. For debugging.", "response": "def get_log_report(log: Union[logging.Logger,\n logging.PlaceHolder]) -> Dict[str, Any]:\n \"\"\"\n Returns information on a log, as a dictionary. For debugging.\n \"\"\"\n if isinstance(log, logging.Logger):\n # suppress invalid error for Logger.manager:\n # noinspection PyUnresolvedReferences\n return {\n '(object)': str(log),\n 'level': log.level,\n 'disabled': log.disabled,\n 'propagate': log.propagate,\n 'parent': str(log.parent),\n 'manager': str(log.manager),\n 'handlers': [get_handler_report(h) for h in log.handlers],\n }\n elif isinstance(log, logging.PlaceHolder):\n return {\n \"(object)\": str(log),\n }\n else:\n raise ValueError(\"Unknown object type: {!r}\".format(log))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef print_report_on_all_logs() -> None:\n d = {}\n # noinspection PyUnresolvedReferences\n for name, obj in logging.Logger.manager.loggerDict.items():\n d[name] = get_log_report(obj)\n rootlogger = logging.getLogger()\n d['(root logger)'] = get_log_report(rootlogger)\n print(json.dumps(d, sort_keys=True, indent=4, separators=(',', ': ')))", "response": "Use this function to print a report on all logs."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsetting a log level for a log and all its handlers.", "response": "def set_level_for_logger_and_its_handlers(log: logging.Logger,\n level: int) -> None:\n \"\"\"\n Set a log level for a log and all its handlers.\n\n Args:\n log: log to modify\n level: log level to set\n \"\"\"\n log.setLevel(level)\n for h in log.handlers: # type: logging.Handler\n h.setLevel(level)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_brace_style_log_with_null_handler(name: str) -> BraceStyleAdapter:\n log = logging.getLogger(name)\n log.addHandler(logging.NullHandler())\n return BraceStyleAdapter(log)", "response": "Returns a log with the specified name that has a null handler attached."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef format(self, record: logging.LogRecord) -> str:\n\n # message = super().format(record)\n super().format(record)\n # Since fmt does not contain asctime, the Formatter.format()\n # will not write asctime (since its usesTime()) function will be\n # false. Therefore:\n record.asctime = self.formatTime(record, self.datefmt)\n bg_col = self.log_background_colors[record.levelno]\n msg = escape(record.getMessage())\n # escape() won't replace \\n but will replace & etc.\n if self.replace_nl_with_br:\n msg = msg.replace(\"\\n\", \"<br>\")\n html = (\n '<span style=\"color:#008B8B\">{time}.{ms:03d} {name}:{lvname}: '\n '</span><span style=\"color:{color}{bg}\">{msg}</font>{br}'.format(\n time=record.asctime,\n ms=int(record.msecs),\n name=record.name,\n lvname=record.levelname,\n color=self.log_colors[record.levelno],\n msg=msg,\n bg=\";background-color:{}\".format(bg_col) if bg_col else \"\",\n br=\"<br>\" if self.append_br else \"\",\n )\n )\n # print(\"record.__dict__: {}\".format(record.__dict__))\n # print(\"html: {}\".format(html))\n return html", "response": "Format the log record as HTML."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef emit(self, record: logging.LogRecord) -> None:\n # noinspection PyBroadException\n try:\n html = self.format(record)\n self.logfunction(html)\n except: # nopep8\n self.handleError(record)", "response": "Internal function to process a logging. LogRecord."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a list of database table names from the given engine.", "response": "def get_table_names(engine: Engine) -> List[str]:\n \"\"\"\n Returns a list of database table names from the :class:`Engine`.\n \"\"\"\n insp = Inspector.from_engine(engine)\n return insp.get_table_names()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a list of database view names from the Engine.", "response": "def get_view_names(engine: Engine) -> List[str]:\n \"\"\"\n Returns a list of database view names from the :class:`Engine`.\n \"\"\"\n insp = Inspector.from_engine(engine)\n return insp.get_view_names()"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns whether the named table or view exists in the database?", "response": "def table_or_view_exists(engine: Engine, table_or_view_name: str) -> bool:\n \"\"\"\n Does the named table/view exist (either as a table or as a view) in the\n database?\n \"\"\"\n tables_and_views = get_table_names(engine) + get_view_names(engine)\n return table_or_view_name in tables_and_views"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef gen_columns_info(engine: Engine,\n tablename: str) -> Generator[SqlaColumnInspectionInfo,\n None, None]:\n \"\"\"\n For the specified table, generate column information as\n :class:`SqlaColumnInspectionInfo` objects.\n \"\"\"\n # Dictionary structure: see\n # http://docs.sqlalchemy.org/en/latest/core/reflection.html#sqlalchemy.engine.reflection.Inspector.get_columns # noqa\n insp = Inspector.from_engine(engine)\n for d in insp.get_columns(tablename):\n yield SqlaColumnInspectionInfo(d)", "response": "Generate column info for the specified table."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_column_info(engine: Engine, tablename: str,\n columnname: str) -> Optional[SqlaColumnInspectionInfo]:\n \"\"\"\n For the specified column in the specified table, get column information\n as a :class:`SqlaColumnInspectionInfo` object (or ``None`` if such a\n column can't be found).\n \"\"\"\n for info in gen_columns_info(engine, tablename):\n if info.name == columnname:\n return info\n return None", "response": "Get the SQLAlchemy column information for the specified column in the specified table."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget the SQLAlchemy column type class for the specified column in the specified table.", "response": "def get_column_type(engine: Engine, tablename: str,\n columnname: str) -> Optional[TypeEngine]:\n \"\"\"\n For the specified column in the specified table, get its type as an\n instance of an SQLAlchemy column type class (or ``None`` if such a column\n can't be found).\n\n For more on :class:`TypeEngine`, see\n :func:`cardinal_pythonlib.orm_inspect.coltype_as_typeengine`.\n \"\"\"\n for info in gen_columns_info(engine, tablename):\n if info.name == columnname:\n return info.type\n return None"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_column_names(engine: Engine, tablename: str) -> List[str]:\n return [info.name for info in gen_columns_info(engine, tablename)]", "response": "Get all the database column names for the specified table."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_pk_colnames(table_: Table) -> List[str]:\n pk_names = [] # type: List[str]\n for col in table_.columns:\n if col.primary_key:\n pk_names.append(col.name)\n return pk_names", "response": "Returns a list of database column names for a table."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the database column name for a single - field integer PK.", "response": "def get_single_int_pk_colname(table_: Table) -> Optional[str]:\n \"\"\"\n If a table has a single-field (non-composite) integer PK, this will\n return its database column name; otherwise, None.\n\n Note that it is legitimate for a database table to have both a composite\n primary key and a separate ``IDENTITY`` (``AUTOINCREMENT``) integer field.\n This function won't find such columns.\n \"\"\"\n n_pks = 0\n int_pk_names = []\n for col in table_.columns:\n if col.primary_key:\n n_pks += 1\n if is_sqlatype_integer(col.type):\n int_pk_names.append(col.name)\n if n_pks == 1 and len(int_pk_names) == 1:\n return int_pk_names[0]\n return None"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the name of the column that is used for autoincrementing the given table.", "response": "def get_single_int_autoincrement_colname(table_: Table) -> Optional[str]:\n \"\"\"\n If a table has a single integer ``AUTOINCREMENT`` column, this will\n return its name; otherwise, ``None``.\n\n - It's unlikely that a database has >1 ``AUTOINCREMENT`` field anyway, but\n we should check.\n - SQL Server's ``IDENTITY`` keyword is equivalent to MySQL's\n ``AUTOINCREMENT``.\n - Verify against SQL Server:\n\n .. code-block:: sql\n\n SELECT table_name, column_name\n FROM information_schema.columns\n WHERE COLUMNPROPERTY(OBJECT_ID(table_schema + '.' + table_name),\n column_name,\n 'IsIdentity') = 1\n ORDER BY table_name;\n\n ... http://stackoverflow.com/questions/87747\n\n - Also:\n\n .. code-block:: sql\n\n sp_columns 'tablename';\n\n ... which is what SQLAlchemy does (``dialects/mssql/base.py``, in\n :func:`get_columns`).\n \"\"\"\n n_autoinc = 0\n int_autoinc_names = []\n for col in table_.columns:\n if col.autoincrement:\n n_autoinc += 1\n if is_sqlatype_integer(col.type):\n int_autoinc_names.append(col.name)\n if n_autoinc > 1:\n log.warning(\"Table {!r} has {} autoincrement columns\",\n table_.name, n_autoinc)\n if n_autoinc == 1 and len(int_autoinc_names) == 1:\n return int_autoinc_names[0]\n return None"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn True if the specified index exists for the specified table.", "response": "def index_exists(engine: Engine, tablename: str, indexname: str) -> bool:\n \"\"\"\n Does the specified index exist for the specified table?\n \"\"\"\n insp = Inspector.from_engine(engine)\n return any(i['name'] == indexname for i in insp.get_indexes(tablename))"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfetches the name of the PK index for the specified table in the specified schema.", "response": "def mssql_get_pk_index_name(engine: Engine,\n tablename: str,\n schemaname: str = MSSQL_DEFAULT_SCHEMA) -> str:\n \"\"\"\n For Microsoft SQL Server specifically: fetch the name of the PK index\n for the specified table (in the specified schema), or ``''`` if none is\n found.\n \"\"\"\n # http://docs.sqlalchemy.org/en/latest/core/connections.html#sqlalchemy.engine.Connection.execute # noqa\n # http://docs.sqlalchemy.org/en/latest/core/sqlelement.html#sqlalchemy.sql.expression.text # noqa\n # http://docs.sqlalchemy.org/en/latest/core/sqlelement.html#sqlalchemy.sql.expression.TextClause.bindparams # noqa\n # http://docs.sqlalchemy.org/en/latest/core/connections.html#sqlalchemy.engine.ResultProxy # noqa\n query = text(\"\"\"\nSELECT\n kc.name AS index_name\nFROM\n sys.key_constraints AS kc\n INNER JOIN sys.tables AS ta ON ta.object_id = kc.parent_object_id\n INNER JOIN sys.schemas AS s ON ta.schema_id = s.schema_id\nWHERE\n kc.[type] = 'PK'\n AND ta.name = :tablename\n AND s.name = :schemaname\n \"\"\").bindparams(\n tablename=tablename,\n schemaname=schemaname,\n )\n with contextlib.closing(\n engine.execute(query)) as result: # type: ResultProxy # noqa\n row = result.fetchone()\n return row[0] if row else ''"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the number of transactions in the database.", "response": "def mssql_transaction_count(engine_or_conn: Union[Connection, Engine]) -> int:\n \"\"\"\n For Microsoft SQL Server specifically: fetch the value of the ``TRANCOUNT``\n variable (see e.g.\n https://docs.microsoft.com/en-us/sql/t-sql/functions/trancount-transact-sql?view=sql-server-2017).\n Returns ``None`` if it can't be found (unlikely?).\n \"\"\"\n sql = \"SELECT @@TRANCOUNT\"\n with contextlib.closing(\n engine_or_conn.execute(sql)) as result: # type: ResultProxy # noqa\n row = result.fetchone()\n return row[0] if row else None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef add_index(engine: Engine,\n sqla_column: Column = None,\n multiple_sqla_columns: List[Column] = None,\n unique: bool = False,\n fulltext: bool = False,\n length: int = None) -> None:\n \"\"\"\n Adds an index to a database column (or, in restricted circumstances,\n several columns).\n\n The table name is worked out from the :class:`Column` object.\n\n Args:\n engine: SQLAlchemy :class:`Engine` object\n sqla_column: single column to index\n multiple_sqla_columns: multiple columns to index (see below)\n unique: make a ``UNIQUE`` index?\n fulltext: make a ``FULLTEXT`` index?\n length: index length to use (default ``None``)\n\n Restrictions:\n\n - Specify either ``sqla_column`` or ``multiple_sqla_columns``, not both.\n - The normal method is ``sqla_column``.\n - ``multiple_sqla_columns`` is only used for Microsoft SQL Server full-text\n indexing (as this database permits only one full-text index per table,\n though that index can be on multiple columns).\n\n \"\"\"\n # We used to process a table as a unit; this makes index creation faster\n # (using ALTER TABLE).\n # http://dev.mysql.com/doc/innodb/1.1/en/innodb-create-index-examples.html # noqa\n # ... ignored in transition to SQLAlchemy\n\n def quote(identifier: str) -> str:\n return quote_identifier(identifier, engine)\n\n is_mssql = engine.dialect.name == SqlaDialectName.MSSQL\n is_mysql = engine.dialect.name == SqlaDialectName.MYSQL\n\n multiple_sqla_columns = multiple_sqla_columns or [] # type: List[Column]\n if multiple_sqla_columns and not (fulltext and is_mssql):\n raise ValueError(\"add_index: Use multiple_sqla_columns only for mssql \"\n \"(Microsoft SQL Server) full-text indexing\")\n if bool(multiple_sqla_columns) == (sqla_column is not None):\n raise ValueError(\n \"add_index: Use either sqla_column or multiple_sqla_columns, not \"\n \"both (sqla_column = {}, multiple_sqla_columns = {}\".format(\n repr(sqla_column), repr(multiple_sqla_columns)))\n if sqla_column is not None:\n colnames = [sqla_column.name]\n sqla_table = sqla_column.table\n tablename = sqla_table.name\n else:\n colnames = [c.name for c in multiple_sqla_columns]\n sqla_table = multiple_sqla_columns[0].table\n tablename = sqla_table.name\n if any(c.table.name != tablename for c in multiple_sqla_columns[1:]):\n raise ValueError(\n \"add_index: tablenames are inconsistent in \"\n \"multiple_sqla_columns = {}\".format(\n repr(multiple_sqla_columns)))\n\n if fulltext:\n if is_mssql:\n idxname = '' # they are unnamed\n else:\n idxname = \"_idxft_{}\".format(\"_\".join(colnames))\n else:\n idxname = \"_idx_{}\".format(\"_\".join(colnames))\n if idxname and index_exists(engine, tablename, idxname):\n log.info(\"Skipping creation of index {} on table {}; already \"\n \"exists\".format(idxname, tablename))\n return\n # because it will crash if you add it again!\n log.info(\n \"Creating{ft} index {i} on table {t}, column(s) {c}\",\n ft=\" full-text\" if fulltext else \"\",\n i=idxname or \"<unnamed>\",\n t=tablename,\n c=\", \".join(colnames),\n )\n\n if fulltext:\n if is_mysql:\n log.info('OK to ignore this warning, if it follows next: '\n '\"InnoDB rebuilding table to add column FTS_DOC_ID\"')\n # https://dev.mysql.com/doc/refman/5.6/en/innodb-fulltext-index.html\n sql = (\n \"ALTER TABLE {tablename} \"\n \"ADD FULLTEXT INDEX {idxname} ({colnames})\".format(\n tablename=quote(tablename),\n idxname=quote(idxname),\n colnames=\", \".join(quote(c) for c in colnames),\n )\n )\n # DDL(sql, bind=engine).execute_if(dialect=SqlaDialectName.MYSQL)\n DDL(sql, bind=engine).execute()\n\n elif is_mssql: # Microsoft SQL Server\n # https://msdn.microsoft.com/library/ms187317(SQL.130).aspx\n # Argh! Complex.\n # Note that the database must also have had a\n # CREATE FULLTEXT CATALOG somename AS DEFAULT;\n # statement executed on it beforehand.\n schemaname = engine.schema_for_object(\n sqla_table) or MSSQL_DEFAULT_SCHEMA # noqa\n if mssql_table_has_ft_index(engine=engine,\n tablename=tablename,\n schemaname=schemaname):\n log.info(\n \"... skipping creation of full-text index on table {}; a \"\n \"full-text index already exists for that table; you can \"\n \"have only one full-text index per table, though it can \"\n \"be on multiple columns\".format(tablename))\n return\n pk_index_name = mssql_get_pk_index_name(\n engine=engine, tablename=tablename, schemaname=schemaname)\n if not pk_index_name:\n raise ValueError(\n \"To make a FULLTEXT index under SQL Server, we need to \"\n \"know the name of the PK index, but couldn't find one \"\n \"from get_pk_index_name() for table {}\".format(\n repr(tablename)))\n # We don't name the FULLTEXT index itself, but it has to relate\n # to an existing unique index.\n sql = (\n \"CREATE FULLTEXT INDEX ON {tablename} ({colnames}) \"\n \"KEY INDEX {keyidxname} \".format(\n tablename=quote(tablename),\n keyidxname=quote(pk_index_name),\n colnames=\", \".join(quote(c) for c in colnames),\n )\n )\n # SQL Server won't let you do this inside a transaction:\n # \"CREATE FULLTEXT INDEX statement cannot be used inside a user\n # transaction.\"\n # https://msdn.microsoft.com/nl-nl/library/ms191544(v=sql.105).aspx\n # So let's ensure any preceding transactions are completed, and\n # run the SQL in a raw way:\n # engine.execute(sql).execution_options(autocommit=False)\n # http://docs.sqlalchemy.org/en/latest/core/connections.html#understanding-autocommit\n #\n # ... lots of faff with this (see test code in no_transactions.py)\n # ... ended up using explicit \"autocommit=True\" parameter (for\n # pyodbc); see create_indexes()\n transaction_count = mssql_transaction_count(engine)\n if transaction_count != 0:\n log.critical(\"SQL Server transaction count (should be 0): \"\n \"{}\".format(transaction_count))\n # Executing serial COMMITs or a ROLLBACK won't help here if\n # this transaction is due to Python DBAPI default behaviour.\n DDL(sql, bind=engine).execute()\n\n # The reversal procedure is DROP FULLTEXT INDEX ON tablename;\n\n else:\n log.error(\"Don't know how to make full text index on dialect \"\n \"{}\".format(engine.dialect.name))\n\n else:\n index = Index(idxname, sqla_column, unique=unique, mysql_length=length)\n index.create(engine)", "response": "Adds an index to a database column or multiple columns."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn an instance of Column representing a BIGINT autoincrement column.", "response": "def make_bigint_autoincrement_column(column_name: str,\n dialect: Dialect) -> Column:\n \"\"\"\n Returns an instance of :class:`Column` representing a :class:`BigInteger`\n ``AUTOINCREMENT`` column in the specified :class:`Dialect`.\n \"\"\"\n # noinspection PyUnresolvedReferences\n if dialect.name == SqlaDialectName.MSSQL:\n return Column(column_name, BigInteger,\n Sequence('dummy_name', start=1, increment=1))\n else:\n # return Column(column_name, BigInteger, autoincrement=True)\n # noinspection PyUnresolvedReferences\n raise AssertionError(\n \"SQLAlchemy doesn't support non-PK autoincrement fields yet for \"\n \"dialect {}\".format(repr(dialect.name)))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a DDL to create a column using the specified dialect.", "response": "def column_creation_ddl(sqla_column: Column, dialect: Dialect) -> str:\n \"\"\"\n Returns DDL to create a column, using the specified dialect.\n\n The column should already be bound to a table (because e.g. the SQL Server\n dialect requires this for DDL generation).\n\n Manual testing:\n \n .. code-block:: python\n\n from sqlalchemy.schema import Column, CreateColumn, MetaData, Sequence, Table\n from sqlalchemy.sql.sqltypes import BigInteger\n from sqlalchemy.dialects.mssql.base import MSDialect\n dialect = MSDialect()\n col1 = Column('hello', BigInteger, nullable=True)\n col2 = Column('world', BigInteger, autoincrement=True) # does NOT generate IDENTITY\n col3 = Column('you', BigInteger, Sequence('dummy_name', start=1, increment=1))\n metadata = MetaData()\n t = Table('mytable', metadata)\n t.append_column(col1)\n t.append_column(col2)\n t.append_column(col3)\n print(str(CreateColumn(col1).compile(dialect=dialect))) # hello BIGINT NULL\n print(str(CreateColumn(col2).compile(dialect=dialect))) # world BIGINT NULL\n print(str(CreateColumn(col3).compile(dialect=dialect))) # you BIGINT NOT NULL IDENTITY(1,1)\n\n If you don't append the column to a Table object, the DDL generation step\n gives:\n \n .. code-block:: none\n\n sqlalchemy.exc.CompileError: mssql requires Table-bound columns in order to generate DDL\n \"\"\" # noqa\n return str(CreateColumn(sqla_column).compile(dialect=dialect))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef giant_text_sqltype(dialect: Dialect) -> str:\n if dialect.name == SqlaDialectName.SQLSERVER:\n return 'NVARCHAR(MAX)'\n elif dialect.name == SqlaDialectName.MYSQL:\n return 'LONGTEXT'\n else:\n raise ValueError(\"Unknown dialect: {}\".format(dialect.name))", "response": "Returns the SQL column type used to make very large text columns for a SQLAlchemy dialect."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _get_sqla_coltype_class_from_str(coltype: str,\n dialect: Dialect) -> Type[TypeEngine]:\n \"\"\"\n Returns the SQLAlchemy class corresponding to a particular SQL column\n type in a given dialect.\n\n Performs an upper- and lower-case search.\n For example, the SQLite dialect uses upper case, and the\n MySQL dialect uses lower case.\n \"\"\"\n # noinspection PyUnresolvedReferences\n ischema_names = dialect.ischema_names\n try:\n return ischema_names[coltype.upper()]\n except KeyError:\n return ischema_names[coltype.lower()]", "response": "Returns the SQLAlchemy class corresponding to a particular SQL column type in a given dialect."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_list_of_sql_string_literals_from_quoted_csv(x: str) -> List[str]:\n f = io.StringIO(x)\n reader = csv.reader(f, delimiter=',', quotechar=\"'\", quoting=csv.QUOTE_ALL,\n skipinitialspace=True)\n for line in reader: # should only be one\n return [x for x in line]", "response": "This function takes a quoted CSV file and returns a list of SQL string literals."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_sqla_coltype_from_dialect_str(coltype: str,\n dialect: Dialect) -> TypeEngine:\n \"\"\"\n Returns an SQLAlchemy column type, given a column type name (a string) and\n an SQLAlchemy dialect. For example, this might convert the string\n ``INTEGER(11)`` to an SQLAlchemy ``Integer(length=11)``.\n\n Args:\n dialect: a SQLAlchemy :class:`Dialect` class\n\n coltype: a ``str()`` representation, e.g. from ``str(c['type'])`` where\n ``c`` is an instance of :class:`sqlalchemy.sql.schema.Column`.\n\n Returns:\n a Python object that is a subclass of\n :class:`sqlalchemy.types.TypeEngine`\n\n Example:\n\n .. code-block:: python\n\n get_sqla_coltype_from_string('INTEGER(11)', engine.dialect)\n # gives: Integer(length=11)\n\n Notes:\n\n - :class:`sqlalchemy.engine.default.DefaultDialect` is the dialect base\n class\n\n - a dialect contains these things of interest:\n\n - ``ischema_names``: string-to-class dictionary\n - ``type_compiler``: instance of e.g.\n :class:`sqlalchemy.sql.compiler.GenericTypeCompiler`. This has a\n ``process()`` method, but that operates on :class:`TypeEngine` objects.\n - ``get_columns``: takes a table name, inspects the database\n\n - example of the dangers of ``eval``:\n http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html\n\n - An example of a function doing the reflection/inspection within\n SQLAlchemy is\n :func:`sqlalchemy.dialects.mssql.base.MSDialect.get_columns`,\n which has this lookup: ``coltype = self.ischema_names.get(type, None)``\n\n Caveats:\n\n - the parameters, e.g. ``DATETIME(6)``, do NOT necessarily either work at\n all or work correctly. For example, SQLAlchemy will happily spit out\n ``'INTEGER(11)'`` but its :class:`sqlalchemy.sql.sqltypes.INTEGER` class\n takes no parameters, so you get the error ``TypeError: object() takes no\n parameters``. Similarly, MySQL's ``DATETIME(6)`` uses the 6 to refer to\n precision, but the ``DATETIME`` class in SQLAlchemy takes only a boolean\n parameter (timezone).\n - However, sometimes we have to have parameters, e.g. ``VARCHAR`` length.\n - Thus, this is a bit useless.\n - Fixed, with a few special cases.\n \"\"\"\n size = None # type: Optional[int]\n dp = None # type: Optional[int]\n args = [] # type: List[Any]\n kwargs = {} # type: Dict[str, Any]\n basetype = ''\n\n # noinspection PyPep8,PyBroadException\n try:\n # Split e.g. \"VARCHAR(32) COLLATE blah\" into \"VARCHAR(32)\", \"who cares\"\n m = RE_COLTYPE_WITH_COLLATE.match(coltype)\n if m is not None:\n coltype = m.group('maintype')\n\n found = False\n\n if not found:\n # Deal with ENUM('a', 'b', 'c', ...)\n m = RE_MYSQL_ENUM_COLTYPE.match(coltype)\n if m is not None:\n # Convert to VARCHAR with max size being that of largest enum\n basetype = 'VARCHAR'\n values = get_list_of_sql_string_literals_from_quoted_csv(\n m.group('valuelist'))\n length = max(len(x) for x in values)\n kwargs = {'length': length}\n found = True\n\n if not found:\n # Split e.g. \"DECIMAL(10, 2)\" into DECIMAL, 10, 2\n m = RE_COLTYPE_WITH_TWO_PARAMS.match(coltype)\n if m is not None:\n basetype = m.group('type').upper()\n size = ast.literal_eval(m.group('size'))\n dp = ast.literal_eval(m.group('dp'))\n found = True\n\n if not found:\n # Split e.g. \"VARCHAR(32)\" into VARCHAR, 32\n m = RE_COLTYPE_WITH_ONE_PARAM.match(coltype)\n if m is not None:\n basetype = m.group('type').upper()\n size_text = m.group('size').strip().upper()\n if size_text != 'MAX':\n size = ast.literal_eval(size_text)\n found = True\n\n if not found:\n basetype = coltype.upper()\n\n # Special cases: pre-processing\n # noinspection PyUnresolvedReferences\n if (dialect.name == SqlaDialectName.MSSQL and\n basetype.lower() == 'integer'):\n basetype = 'int'\n\n cls = _get_sqla_coltype_class_from_str(basetype, dialect)\n\n # Special cases: post-processing\n if basetype == 'DATETIME' and size:\n # First argument to DATETIME() is timezone, so...\n # noinspection PyUnresolvedReferences\n if dialect.name == SqlaDialectName.MYSQL:\n kwargs = {'fsp': size}\n else:\n pass\n else:\n args = [x for x in (size, dp) if x is not None]\n\n try:\n return cls(*args, **kwargs)\n except TypeError:\n return cls()\n\n except:\n # noinspection PyUnresolvedReferences\n raise ValueError(\"Failed to convert SQL type {} in dialect {} to an \"\n \"SQLAlchemy type\".format(repr(coltype),\n repr(dialect.name)))", "response": "Returns an SQLAlchemy column type given a column type name and an SQLAlchemy dialect."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a copy of the specific column type with any COLLATION removed.", "response": "def remove_collation(coltype: TypeEngine) -> TypeEngine:\n \"\"\"\n Returns a copy of the specific column type with any ``COLLATION`` removed.\n \"\"\"\n if not getattr(coltype, 'collation', None):\n return coltype\n newcoltype = copy.copy(coltype)\n newcoltype.collation = None\n return newcoltype"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef convert_sqla_type_for_dialect(\n coltype: TypeEngine,\n dialect: Dialect,\n strip_collation: bool = True,\n convert_mssql_timestamp: bool = True,\n expand_for_scrubbing: bool = False) -> TypeEngine:\n \"\"\"\n Converts an SQLAlchemy column type from one SQL dialect to another.\n\n Args:\n coltype: SQLAlchemy column type in the source dialect\n\n dialect: destination :class:`Dialect`\n\n strip_collation: remove any ``COLLATION`` information?\n\n convert_mssql_timestamp:\n since you cannot write to a SQL Server ``TIMESTAMP`` field, setting\n this option to ``True`` (the default) converts such types to\n something equivalent but writable.\n\n expand_for_scrubbing:\n The purpose of expand_for_scrubbing is that, for example, a\n ``VARCHAR(200)`` field containing one or more instances of\n ``Jones``, where ``Jones`` is to be replaced with ``[XXXXXX]``,\n will get longer (by an unpredictable amount). So, better to expand\n to unlimited length.\n\n Returns:\n an SQLAlchemy column type instance, in the destination dialect\n\n \"\"\"\n assert coltype is not None\n\n # noinspection PyUnresolvedReferences\n to_mysql = dialect.name == SqlaDialectName.MYSQL\n # noinspection PyUnresolvedReferences\n to_mssql = dialect.name == SqlaDialectName.MSSQL\n typeclass = type(coltype)\n\n # -------------------------------------------------------------------------\n # Text\n # -------------------------------------------------------------------------\n if isinstance(coltype, sqltypes.Enum):\n return sqltypes.String(length=coltype.length)\n if isinstance(coltype, sqltypes.UnicodeText):\n # Unbounded Unicode text.\n # Includes derived classes such as mssql.base.NTEXT.\n return sqltypes.UnicodeText()\n if isinstance(coltype, sqltypes.Text):\n # Unbounded text, more generally. (UnicodeText inherits from Text.)\n # Includes sqltypes.TEXT.\n return sqltypes.Text()\n # Everything inheriting from String has a length property, but can be None.\n # There are types that can be unlimited in SQL Server, e.g. VARCHAR(MAX)\n # and NVARCHAR(MAX), that MySQL needs a length for. (Failure to convert\n # gives e.g.: 'NVARCHAR requires a length on dialect mysql'.)\n if isinstance(coltype, sqltypes.Unicode):\n # Includes NVARCHAR(MAX) in SQL -> NVARCHAR() in SQLAlchemy.\n if (coltype.length is None and to_mysql) or expand_for_scrubbing:\n return sqltypes.UnicodeText()\n # The most general case; will pick up any other string types.\n if isinstance(coltype, sqltypes.String):\n # Includes VARCHAR(MAX) in SQL -> VARCHAR() in SQLAlchemy\n if (coltype.length is None and to_mysql) or expand_for_scrubbing:\n return sqltypes.Text()\n if strip_collation:\n return remove_collation(coltype)\n return coltype\n\n # -------------------------------------------------------------------------\n # Binary\n # -------------------------------------------------------------------------\n\n # -------------------------------------------------------------------------\n # BIT\n # -------------------------------------------------------------------------\n if typeclass == mssql.base.BIT and to_mysql:\n # MySQL BIT objects have a length attribute.\n return mysql.base.BIT()\n\n # -------------------------------------------------------------------------\n # TIMESTAMP\n # -------------------------------------------------------------------------\n is_mssql_timestamp = isinstance(coltype, MSSQL_TIMESTAMP)\n if is_mssql_timestamp and to_mssql and convert_mssql_timestamp:\n # You cannot write explicitly to a TIMESTAMP field in SQL Server; it's\n # used for autogenerated values only.\n # - http://stackoverflow.com/questions/10262426/sql-server-cannot-insert-an-explicit-value-into-a-timestamp-column # noqa\n # - https://social.msdn.microsoft.com/Forums/sqlserver/en-US/5167204b-ef32-4662-8e01-00c9f0f362c2/how-to-tranfer-a-column-with-timestamp-datatype?forum=transactsql # noqa\n # ... suggesting BINARY(8) to store the value.\n # MySQL is more helpful:\n # - http://stackoverflow.com/questions/409286/should-i-use-field-datetime-or-timestamp # noqa\n return mssql.base.BINARY(8)\n\n # -------------------------------------------------------------------------\n # Some other type\n # -------------------------------------------------------------------------\n return coltype", "response": "Converts an SQLAlchemy column type instance from one SQL dialect to another."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts a column type to a typeengine.", "response": "def _coltype_to_typeengine(coltype: Union[TypeEngine,\n VisitableType]) -> TypeEngine:\n \"\"\"\n An example is simplest: if you pass in ``Integer()`` (an instance of\n :class:`TypeEngine`), you'll get ``Integer()`` back. If you pass in\n ``Integer`` (an instance of :class:`VisitableType`), you'll also get\n ``Integer()`` back. The function asserts that its return type is an\n instance of :class:`TypeEngine`.\n \"\"\"\n if isinstance(coltype, VisitableType):\n coltype = coltype()\n assert isinstance(coltype, TypeEngine)\n return coltype"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef is_sqlatype_binary(coltype: Union[TypeEngine, VisitableType]) -> bool:\n # Several binary types inherit internally from _Binary, making that the\n # easiest to check.\n coltype = _coltype_to_typeengine(coltype)\n # noinspection PyProtectedMember\n return isinstance(coltype, sqltypes._Binary)", "response": "Returns True if the SQLAlchemy column type is a binary type."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef is_sqlatype_date(coltype: TypeEngine) -> bool:\n coltype = _coltype_to_typeengine(coltype)\n # No longer valid in SQLAlchemy 1.2.11:\n # return isinstance(coltype, sqltypes._DateAffinity)\n return (\n isinstance(coltype, sqltypes.DateTime) or\n isinstance(coltype, sqltypes.Date)\n )", "response": "Is the SQLAlchemy column type a date type?"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef is_sqlatype_integer(coltype: Union[TypeEngine, VisitableType]) -> bool:\n coltype = _coltype_to_typeengine(coltype)\n return isinstance(coltype, sqltypes.Integer)", "response": "Returns true if the SQLAlchemy column type is an integer type?"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning true if the SQLAlchemy column type is a numeric type.", "response": "def is_sqlatype_numeric(coltype: Union[TypeEngine, VisitableType]) -> bool:\n \"\"\"\n Is the SQLAlchemy column type one that inherits from :class:`Numeric`,\n such as :class:`Float`, :class:`Decimal`?\n \"\"\"\n coltype = _coltype_to_typeengine(coltype)\n return isinstance(coltype, sqltypes.Numeric)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns true if the SQLAlchemy column type is a string type.", "response": "def is_sqlatype_string(coltype: Union[TypeEngine, VisitableType]) -> bool:\n \"\"\"\n Is the SQLAlchemy column type a string type?\n \"\"\"\n coltype = _coltype_to_typeengine(coltype)\n return isinstance(coltype, sqltypes.String)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nbeing the SQLAlchemy column type a string type that s at least the specified min_length?", "response": "def is_sqlatype_text_of_length_at_least(\n coltype: Union[TypeEngine, VisitableType],\n min_length: int = 1000) -> bool:\n \"\"\"\n Is the SQLAlchemy column type a string type that's at least the specified\n length?\n \"\"\"\n coltype = _coltype_to_typeengine(coltype)\n if not isinstance(coltype, sqltypes.String):\n return False # not a string/text type at all\n if coltype.length is None:\n return True # string of unlimited length\n return coltype.length >= min_length"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nam the SQLAlchemy column type a string type that s more than one character?", "response": "def is_sqlatype_text_over_one_char(\n coltype: Union[TypeEngine, VisitableType]) -> bool:\n \"\"\"\n Is the SQLAlchemy column type a string type that's more than one character\n long?\n \"\"\"\n coltype = _coltype_to_typeengine(coltype)\n return is_sqlatype_text_of_length_at_least(coltype, 2)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef does_sqlatype_merit_fulltext_index(\n coltype: Union[TypeEngine, VisitableType],\n min_length: int = 1000) -> bool:\n \"\"\"\n Is the SQLAlchemy column type a type that might merit a ``FULLTEXT``\n index (meaning a string type of at least ``min_length``)?\n \"\"\"\n coltype = _coltype_to_typeengine(coltype)\n return is_sqlatype_text_of_length_at_least(coltype, min_length)", "response": "Returns True if the SQLAlchemy column type is a FULLTEXT column type."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns True if the SQLAlchemy column type does not require an index length.", "response": "def does_sqlatype_require_index_len(\n coltype: Union[TypeEngine, VisitableType]) -> bool:\n \"\"\"\n Is the SQLAlchemy column type one that requires its indexes to have a\n length specified?\n\n (MySQL, at least, requires index length to be specified for ``BLOB`` and\n ``TEXT`` columns:\n http://dev.mysql.com/doc/refman/5.7/en/create-index.html.)\n \"\"\"\n coltype = _coltype_to_typeengine(coltype)\n if isinstance(coltype, sqltypes.Text):\n return True\n if isinstance(coltype, sqltypes.LargeBinary):\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchecks that two SQLAlchemy column types are equal.", "response": "def column_types_equal(a_coltype: TypeEngine, b_coltype: TypeEngine) -> bool:\n \"\"\"\n Checks that two SQLAlchemy column types are equal (by comparing ``str()``\n versions of them).\n \n See http://stackoverflow.com/questions/34787794/sqlalchemy-column-type-comparison.\n \n IMPERFECT. \n \"\"\" # noqa\n return str(a_coltype) == str(b_coltype)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef columns_equal(a: Column, b: Column) -> bool:\n return (\n a.name == b.name and\n column_types_equal(a.type, b.type) and\n a.nullable == b.nullable\n )", "response": "Check if two SQLAlchemy columns are equal?"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef column_lists_equal(a: List[Column], b: List[Column]) -> bool:\n n = len(a)\n if len(b) != n:\n return False\n for i in range(n):\n if not columns_equal(a[i], b[i]):\n log.debug(\"Mismatch: {!r} != {!r}\", a[i], b[i])\n return False\n return True", "response": "Returns True if all columns in list a equal to their counterparts in list b."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nis two indexes equal? Checks by comparing str versions of them.", "response": "def indexes_equal(a: Index, b: Index) -> bool:\n \"\"\"\n Are two indexes equal? Checks by comparing ``str()`` versions of them.\n (AM UNSURE IF THIS IS ENOUGH.)\n \"\"\"\n return str(a) == str(b)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ntest if two lists are equal.", "response": "def index_lists_equal(a: List[Index], b: List[Index]) -> bool:\n \"\"\"\n Are all indexes in list ``a`` equal to their counterparts in list ``b``,\n as per :func:`indexes_equal`?\n \"\"\"\n n = len(a)\n if len(b) != n:\n return False\n for i in range(n):\n if not indexes_equal(a[i], b[i]):\n log.debug(\"Mismatch: {!r} != {!r}\", a[i], b[i])\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef to_bytes(data: Any) -> bytearray:\n if isinstance(data, int):\n return bytearray([data])\n return bytearray(data, encoding='latin-1')", "response": "Convert any string to a bytearray."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert a signed integer to its two s complement representation.", "response": "def signed_to_twos_comp(val: int, n_bits: int) -> int:\n \"\"\"\n Convert a signed integer to its \"two's complement\" representation.\n\n Args:\n val: signed integer\n n_bits: number of bits (which must reflect a whole number of bytes)\n\n Returns:\n unsigned integer: two's complement version\n\n \"\"\"\n assert n_bits % 8 == 0, \"Must specify a whole number of bytes\"\n n_bytes = n_bits // 8\n b = val.to_bytes(n_bytes, byteorder=sys.byteorder, signed=True)\n return int.from_bytes(b, byteorder=sys.byteorder, signed=False)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting an 8 - byte sequence to a long integer.", "response": "def bytes_to_long(bytesdata: bytes) -> int:\n \"\"\"\n Converts an 8-byte sequence to a long integer.\n\n Args:\n bytesdata: 8 consecutive bytes, as a ``bytes`` object, in\n little-endian format (least significant byte [LSB] first)\n\n Returns:\n integer\n\n \"\"\"\n assert len(bytesdata) == 8\n return sum((b << (k * 8) for k, b in enumerate(bytesdata)))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nimplementing 128 - bit murmur3 hash for x64.", "response": "def pymmh3_hash128_x64(key: Union[bytes, bytearray], seed: int) -> int:\n \"\"\"\n Implements 128-bit murmur3 hash for x64, as per ``pymmh3``, with some\n bugfixes.\n\n Args:\n key: data to hash\n seed: seed\n\n Returns:\n integer hash\n \"\"\"\n\n def fmix(k):\n k ^= k >> 33\n k = (k * 0xff51afd7ed558ccd) & 0xFFFFFFFFFFFFFFFF\n k ^= k >> 33\n k = (k * 0xc4ceb9fe1a85ec53) & 0xFFFFFFFFFFFFFFFF\n k ^= k >> 33\n return k\n\n length = len(key)\n nblocks = int(length / 16)\n\n h1 = seed\n h2 = seed\n\n c1 = 0x87c37b91114253d5\n c2 = 0x4cf5ad432745937f\n\n # body\n for block_start in range(0, nblocks * 8, 8):\n # ??? big endian?\n k1 = (\n key[2 * block_start + 7] << 56 |\n key[2 * block_start + 6] << 48 |\n key[2 * block_start + 5] << 40 |\n key[2 * block_start + 4] << 32 |\n key[2 * block_start + 3] << 24 |\n key[2 * block_start + 2] << 16 |\n key[2 * block_start + 1] << 8 |\n key[2 * block_start + 0]\n )\n\n k2 = (\n key[2 * block_start + 15] << 56 |\n key[2 * block_start + 14] << 48 |\n key[2 * block_start + 13] << 40 |\n key[2 * block_start + 12] << 32 |\n key[2 * block_start + 11] << 24 |\n key[2 * block_start + 10] << 16 |\n key[2 * block_start + 9] << 8 |\n key[2 * block_start + 8]\n )\n\n k1 = (c1 * k1) & 0xFFFFFFFFFFFFFFFF\n k1 = (k1 << 31 | k1 >> 33) & 0xFFFFFFFFFFFFFFFF # inlined ROTL64\n k1 = (c2 * k1) & 0xFFFFFFFFFFFFFFFF\n h1 ^= k1\n\n h1 = (h1 << 27 | h1 >> 37) & 0xFFFFFFFFFFFFFFFF # inlined ROTL64\n h1 = (h1 + h2) & 0xFFFFFFFFFFFFFFFF\n h1 = (h1 * 5 + 0x52dce729) & 0xFFFFFFFFFFFFFFFF\n\n k2 = (c2 * k2) & 0xFFFFFFFFFFFFFFFF\n k2 = (k2 << 33 | k2 >> 31) & 0xFFFFFFFFFFFFFFFF # inlined ROTL64\n k2 = (c1 * k2) & 0xFFFFFFFFFFFFFFFF\n h2 ^= k2\n\n h2 = (h2 << 31 | h2 >> 33) & 0xFFFFFFFFFFFFFFFF # inlined ROTL64\n h2 = (h1 + h2) & 0xFFFFFFFFFFFFFFFF\n h2 = (h2 * 5 + 0x38495ab5) & 0xFFFFFFFFFFFFFFFF\n\n # tail\n tail_index = nblocks * 16\n k1 = 0\n k2 = 0\n tail_size = length & 15\n\n if tail_size >= 15:\n k2 ^= key[tail_index + 14] << 48\n if tail_size >= 14:\n k2 ^= key[tail_index + 13] << 40\n if tail_size >= 13:\n k2 ^= key[tail_index + 12] << 32\n if tail_size >= 12:\n k2 ^= key[tail_index + 11] << 24\n if tail_size >= 11:\n k2 ^= key[tail_index + 10] << 16\n if tail_size >= 10:\n k2 ^= key[tail_index + 9] << 8\n if tail_size >= 9:\n k2 ^= key[tail_index + 8]\n\n if tail_size > 8:\n k2 = (k2 * c2) & 0xFFFFFFFFFFFFFFFF\n k2 = (k2 << 33 | k2 >> 31) & 0xFFFFFFFFFFFFFFFF # inlined ROTL64\n k2 = (k2 * c1) & 0xFFFFFFFFFFFFFFFF\n h2 ^= k2\n\n if tail_size >= 8:\n k1 ^= key[tail_index + 7] << 56\n if tail_size >= 7:\n k1 ^= key[tail_index + 6] << 48\n if tail_size >= 6:\n k1 ^= key[tail_index + 5] << 40\n if tail_size >= 5:\n k1 ^= key[tail_index + 4] << 32\n if tail_size >= 4:\n k1 ^= key[tail_index + 3] << 24\n if tail_size >= 3:\n k1 ^= key[tail_index + 2] << 16\n if tail_size >= 2:\n k1 ^= key[tail_index + 1] << 8\n if tail_size >= 1:\n k1 ^= key[tail_index + 0]\n\n if tail_size > 0:\n k1 = (k1 * c1) & 0xFFFFFFFFFFFFFFFF\n k1 = (k1 << 31 | k1 >> 33) & 0xFFFFFFFFFFFFFFFF # inlined ROTL64\n k1 = (k1 * c2) & 0xFFFFFFFFFFFFFFFF\n h1 ^= k1\n\n # finalization\n h1 ^= length\n h2 ^= length\n\n h1 = (h1 + h2) & 0xFFFFFFFFFFFFFFFF\n h2 = (h1 + h2) & 0xFFFFFFFFFFFFFFFF\n\n h1 = fmix(h1)\n h2 = fmix(h2)\n\n h1 = (h1 + h2) & 0xFFFFFFFFFFFFFFFF\n h2 = (h1 + h2) & 0xFFFFFFFFFFFFFFFF\n\n return h2 << 64 | h1"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nimplementing 128 - bit murmur3 hash for x86.", "response": "def pymmh3_hash128_x86(key: Union[bytes, bytearray], seed: int) -> int:\n \"\"\"\n Implements 128-bit murmur3 hash for x86, as per ``pymmh3``, with some\n bugfixes.\n\n Args:\n key: data to hash\n seed: seed\n\n Returns:\n integer hash\n \"\"\"\n\n def fmix(h):\n h ^= h >> 16\n h = (h * 0x85ebca6b) & 0xFFFFFFFF\n h ^= h >> 13\n h = (h * 0xc2b2ae35) & 0xFFFFFFFF\n h ^= h >> 16\n return h\n\n length = len(key)\n nblocks = int(length / 16)\n\n h1 = seed\n h2 = seed\n h3 = seed\n h4 = seed\n\n c1 = 0x239b961b\n c2 = 0xab0e9789\n c3 = 0x38b34ae5\n c4 = 0xa1e38b93\n\n # body\n for block_start in range(0, nblocks * 16, 16):\n k1 = (\n key[block_start + 3] << 24 |\n key[block_start + 2] << 16 |\n key[block_start + 1] << 8 |\n key[block_start + 0]\n )\n k2 = (\n key[block_start + 7] << 24 |\n key[block_start + 6] << 16 |\n key[block_start + 5] << 8 |\n key[block_start + 4]\n )\n k3 = (\n key[block_start + 11] << 24 |\n key[block_start + 10] << 16 |\n key[block_start + 9] << 8 |\n key[block_start + 8]\n )\n k4 = (\n key[block_start + 15] << 24 |\n key[block_start + 14] << 16 |\n key[block_start + 13] << 8 |\n key[block_start + 12]\n )\n\n k1 = (c1 * k1) & 0xFFFFFFFF\n k1 = (k1 << 15 | k1 >> 17) & 0xFFFFFFFF # inlined ROTL32\n k1 = (c2 * k1) & 0xFFFFFFFF\n h1 ^= k1\n\n h1 = (h1 << 19 | h1 >> 13) & 0xFFFFFFFF # inlined ROTL32\n h1 = (h1 + h2) & 0xFFFFFFFF\n h1 = (h1 * 5 + 0x561ccd1b) & 0xFFFFFFFF\n\n k2 = (c2 * k2) & 0xFFFFFFFF\n k2 = (k2 << 16 | k2 >> 16) & 0xFFFFFFFF # inlined ROTL32\n k2 = (c3 * k2) & 0xFFFFFFFF\n h2 ^= k2\n\n h2 = (h2 << 17 | h2 >> 15) & 0xFFFFFFFF # inlined ROTL32\n h2 = (h2 + h3) & 0xFFFFFFFF\n h2 = (h2 * 5 + 0x0bcaa747) & 0xFFFFFFFF\n\n k3 = (c3 * k3) & 0xFFFFFFFF\n k3 = (k3 << 17 | k3 >> 15) & 0xFFFFFFFF # inlined ROTL32\n k3 = (c4 * k3) & 0xFFFFFFFF\n h3 ^= k3\n\n h3 = (h3 << 15 | h3 >> 17) & 0xFFFFFFFF # inlined ROTL32\n h3 = (h3 + h4) & 0xFFFFFFFF\n h3 = (h3 * 5 + 0x96cd1c35) & 0xFFFFFFFF\n\n k4 = (c4 * k4) & 0xFFFFFFFF\n k4 = (k4 << 18 | k4 >> 14) & 0xFFFFFFFF # inlined ROTL32\n k4 = (c1 * k4) & 0xFFFFFFFF\n h4 ^= k4\n\n h4 = (h4 << 13 | h4 >> 19) & 0xFFFFFFFF # inlined ROTL32\n h4 = (h1 + h4) & 0xFFFFFFFF\n h4 = (h4 * 5 + 0x32ac3b17) & 0xFFFFFFFF\n\n # tail\n tail_index = nblocks * 16\n k1 = 0\n k2 = 0\n k3 = 0\n k4 = 0\n tail_size = length & 15\n\n if tail_size >= 15:\n k4 ^= key[tail_index + 14] << 16\n if tail_size >= 14:\n k4 ^= key[tail_index + 13] << 8\n if tail_size >= 13:\n k4 ^= key[tail_index + 12]\n\n if tail_size > 12:\n k4 = (k4 * c4) & 0xFFFFFFFF\n k4 = (k4 << 18 | k4 >> 14) & 0xFFFFFFFF # inlined ROTL32\n k4 = (k4 * c1) & 0xFFFFFFFF\n h4 ^= k4\n\n if tail_size >= 12:\n k3 ^= key[tail_index + 11] << 24\n if tail_size >= 11:\n k3 ^= key[tail_index + 10] << 16\n if tail_size >= 10:\n k3 ^= key[tail_index + 9] << 8\n if tail_size >= 9:\n k3 ^= key[tail_index + 8]\n\n if tail_size > 8:\n k3 = (k3 * c3) & 0xFFFFFFFF\n k3 = (k3 << 17 | k3 >> 15) & 0xFFFFFFFF # inlined ROTL32\n k3 = (k3 * c4) & 0xFFFFFFFF\n h3 ^= k3\n\n if tail_size >= 8:\n k2 ^= key[tail_index + 7] << 24\n if tail_size >= 7:\n k2 ^= key[tail_index + 6] << 16\n if tail_size >= 6:\n k2 ^= key[tail_index + 5] << 8\n if tail_size >= 5:\n k2 ^= key[tail_index + 4]\n\n if tail_size > 4:\n k2 = (k2 * c2) & 0xFFFFFFFF\n k2 = (k2 << 16 | k2 >> 16) & 0xFFFFFFFF # inlined ROTL32\n k2 = (k2 * c3) & 0xFFFFFFFF\n h2 ^= k2\n\n if tail_size >= 4:\n k1 ^= key[tail_index + 3] << 24\n if tail_size >= 3:\n k1 ^= key[tail_index + 2] << 16\n if tail_size >= 2:\n k1 ^= key[tail_index + 1] << 8\n if tail_size >= 1:\n k1 ^= key[tail_index + 0]\n\n if tail_size > 0:\n k1 = (k1 * c1) & 0xFFFFFFFF\n k1 = (k1 << 15 | k1 >> 17) & 0xFFFFFFFF # inlined ROTL32\n k1 = (k1 * c2) & 0xFFFFFFFF\n h1 ^= k1\n\n # finalization\n h1 ^= length\n h2 ^= length\n h3 ^= length\n h4 ^= length\n\n h1 = (h1 + h2) & 0xFFFFFFFF\n h1 = (h1 + h3) & 0xFFFFFFFF\n h1 = (h1 + h4) & 0xFFFFFFFF\n h2 = (h1 + h2) & 0xFFFFFFFF\n h3 = (h1 + h3) & 0xFFFFFFFF\n h4 = (h1 + h4) & 0xFFFFFFFF\n\n h1 = fmix(h1)\n h2 = fmix(h2)\n h3 = fmix(h3)\n h4 = fmix(h4)\n\n h1 = (h1 + h2) & 0xFFFFFFFF\n h1 = (h1 + h3) & 0xFFFFFFFF\n h1 = (h1 + h4) & 0xFFFFFFFF\n h2 = (h1 + h2) & 0xFFFFFFFF\n h3 = (h1 + h3) & 0xFFFFFFFF\n h4 = (h1 + h4) & 0xFFFFFFFF\n\n return h4 << 96 | h3 << 64 | h2 << 32 | h1"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef pymmh3_hash128(key: Union[bytes, bytearray],\n seed: int = 0,\n x64arch: bool = True) -> int:\n \"\"\"\n Implements 128bit murmur3 hash, as per ``pymmh3``.\n\n Args:\n key: data to hash\n seed: seed\n x64arch: is a 64-bit architecture available?\n\n Returns:\n integer hash\n\n \"\"\"\n if x64arch:\n return pymmh3_hash128_x64(key, seed)\n else:\n return pymmh3_hash128_x86(key, seed)", "response": "Implements 128bit murmur3 hash as per pymmh3."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nimplements 64bit murmur3 hash as per pymmh3. Returns a tuple of integers signed_val1 signed_val2.", "response": "def pymmh3_hash64(key: Union[bytes, bytearray],\n seed: int = 0,\n x64arch: bool = True) -> Tuple[int, int]:\n \"\"\"\n Implements 64bit murmur3 hash, as per ``pymmh3``. Returns a tuple.\n\n Args:\n key: data to hash\n seed: seed\n x64arch: is a 64-bit architecture available?\n\n Returns:\n tuple: tuple of integers, ``(signed_val1, signed_val2)``\n\n \"\"\"\n\n hash_128 = pymmh3_hash128(key, seed, x64arch)\n\n unsigned_val1 = hash_128 & 0xFFFFFFFFFFFFFFFF # low half\n if unsigned_val1 & 0x8000000000000000 == 0:\n signed_val1 = unsigned_val1\n else:\n signed_val1 = -((unsigned_val1 ^ 0xFFFFFFFFFFFFFFFF) + 1)\n\n unsigned_val2 = (hash_128 >> 64) & 0xFFFFFFFFFFFFFFFF # high half\n if unsigned_val2 & 0x8000000000000000 == 0:\n signed_val2 = unsigned_val2\n else:\n signed_val2 = -((unsigned_val2 ^ 0xFFFFFFFFFFFFFFFF) + 1)\n\n return signed_val1, signed_val2"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef compare_python_to_reference_murmur3_32(data: Any, seed: int = 0) -> None:\n assert mmh3, \"Need mmh3 module\"\n c_data = to_str(data)\n c_signed = mmh3.hash(c_data, seed=seed) # 32 bit\n py_data = to_bytes(c_data)\n py_unsigned = murmur3_x86_32(py_data, seed=seed)\n py_signed = twos_comp_to_signed(py_unsigned, n_bits=32)\n preamble = \"Hashing {data} with MurmurHash3/32-bit/seed={seed}\".format(\n data=repr(data), seed=seed)\n if c_signed == py_signed:\n print(preamble + \" -> {result}: OK\".format(result=c_signed))\n else:\n raise AssertionError(\n preamble + \"; mmh3 says \"\n \"{c_data} -> {c_signed}, Python version says {py_data} -> \"\n \"{py_unsigned} = {py_signed}\".format(\n c_data=repr(c_data),\n c_signed=c_signed,\n py_data=repr(py_data),\n py_unsigned=py_unsigned,\n py_signed=py_signed))", "response": "Compare the pure Python implementation of 32 - bit murmur3 against the MurmurHash3 C - based module."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef compare_python_to_reference_murmur3_64(data: Any, seed: int = 0) -> None:\n assert mmh3, \"Need mmh3 module\"\n c_data = to_str(data)\n c_signed_low, c_signed_high = mmh3.hash64(c_data, seed=seed,\n x64arch=IS_64_BIT)\n py_data = to_bytes(c_data)\n py_signed_low, py_signed_high = pymmh3_hash64(py_data, seed=seed)\n preamble = \"Hashing {data} with MurmurHash3/64-bit values from 128-bit \" \\\n \"hash/seed={seed}\".format(data=repr(data), seed=seed)\n if c_signed_low == py_signed_low and c_signed_high == py_signed_high:\n print(preamble + \" -> (low={low}, high={high}): OK\".format(\n low=c_signed_low, high=c_signed_high))\n else:\n raise AssertionError(\n preamble +\n \"; mmh3 says {c_data} -> (low={c_low}, high={c_high}), Python \"\n \"version says {py_data} -> (low={py_low}, high={py_high})\".format(\n c_data=repr(c_data),\n c_low=c_signed_low,\n c_high=c_signed_high,\n py_data=repr(py_data),\n py_low=py_signed_low,\n py_high=py_signed_high))", "response": "Compare the pure Python implementation of 64 - bit murmur3 against the MurmurHash3 C - based module."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nhashing the data with the specified seed.", "response": "def hash32(data: Any, seed=0) -> int:\n \"\"\"\n Non-cryptographic, deterministic, fast hash.\n\n Args:\n data: data to hash\n seed: seed\n\n Returns:\n signed 32-bit integer\n \"\"\"\n with MultiTimerContext(timer, TIMING_HASH):\n c_data = to_str(data)\n if mmh3:\n return mmh3.hash(c_data, seed=seed)\n py_data = to_bytes(c_data)\n py_unsigned = murmur3_x86_32(py_data, seed=seed)\n return twos_comp_to_signed(py_unsigned, n_bits=32)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef hash64(data: Any, seed: int = 0) -> int:\n # -------------------------------------------------------------------------\n # MurmurHash3\n # -------------------------------------------------------------------------\n c_data = to_str(data)\n if mmh3:\n c_signed_low, _ = mmh3.hash64(data, seed=seed, x64arch=IS_64_BIT)\n return c_signed_low\n py_data = to_bytes(c_data)\n py_signed_low, _ = pymmh3_hash64(py_data, seed=seed)\n return py_signed_low", "response": "Hash the data with the seed and return the signed 64 - bit integer."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncommands - line validation checks.", "response": "def main() -> None:\n \"\"\"\n Command-line validation checks.\n \"\"\"\n _ = \"\"\"\n print(twos_comp_to_signed(0, n_bits=32)) # 0\n print(twos_comp_to_signed(2 ** 31 - 1, n_bits=32)) # 2147483647\n print(twos_comp_to_signed(2 ** 31, n_bits=32)) # -2147483648 == -(2 ** 31)\n print(twos_comp_to_signed(2 ** 32 - 1, n_bits=32)) # -1\n print(signed_to_twos_comp(-1, n_bits=32)) # 4294967295 = 2 ** 32 - 1\n print(signed_to_twos_comp(-(2 ** 31), n_bits=32)) # 2147483648 = 2 ** 31 - 1\n \"\"\" # noqa\n testdata = [\n \"hello\",\n 1,\n [\"bongos\", \"today\"],\n ]\n for data in testdata:\n compare_python_to_reference_murmur3_32(data, seed=0)\n compare_python_to_reference_murmur3_64(data, seed=0)\n print(\"All OK\")"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef hash(self, raw: Any) -> str:\n with MultiTimerContext(timer, TIMING_HASH):\n raw_bytes = str(raw).encode('utf-8')\n hmac_obj = hmac.new(key=self.key_bytes, msg=raw_bytes,\n digestmod=self.digestmod)\n return hmac_obj.hexdigest()", "response": "Returns the hex digest of a HMAC - encoded version of the input."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef mean(values: Sequence[Union[int, float, None]]) -> Optional[float]:\n total = 0.0 # starting with \"0.0\" causes automatic conversion to float\n n = 0\n for x in values:\n if x is not None:\n total += x\n n += 1\n return total / n if n > 0 else None", "response": "Returns the mean of a list of numbers."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef safe_logit(p: Union[float, int]) -> Optional[float]:\n if p > 1 or p < 0:\n return None # can't take log of negative number\n if p == 1:\n return float(\"inf\")\n if p == 0:\n return float(\"-inf\")\n return math.log(p / (1 - p))", "response": "r Returns the logit of the input probability p"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef normal_round_float(x: float, dp: int = 0) -> float:\n if not math.isfinite(x):\n return x\n factor = pow(10, dp)\n x = x * factor\n if x >= 0:\n x = math.floor(x + 0.5)\n else:\n x = math.ceil(x - 0.5)\n x = x / factor\n return x", "response": "Normal round to a number of decimal places."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef normal_round_int(x: float) -> int:\n if not math.isfinite(x):\n raise ValueError(\"Input to normal_round_int() is not finite\")\n if x >= 0:\n # noinspection PyTypeChecker\n return math.floor(x + 0.5)\n else:\n # noinspection PyTypeChecker\n return math.ceil(x - 0.5)", "response": "Returns an integer from 0 to x."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a list of command - line arguments for a mysqldump call.", "response": "def cmdargs(mysqldump: str,\n username: str,\n password: str,\n database: str,\n verbose: bool,\n with_drop_create_database: bool,\n max_allowed_packet: str,\n hide_password: bool = False) -> List[str]:\n \"\"\"\n Returns command arguments for a ``mysqldump`` call.\n\n Args:\n mysqldump: ``mysqldump`` executable filename\n username: user name\n password: password\n database: database name\n verbose: verbose output?\n with_drop_create_database: produce commands to ``DROP`` the database\n and recreate it?\n max_allowed_packet: passed to ``mysqldump``\n hide_password: obscure the password (will break the arguments but\n provide a safe version to show the user)?\n\n Returns:\n list of command-line arguments\n \"\"\"\n ca = [\n mysqldump,\n \"-u\", username,\n \"-p{}\".format(\"*****\" if hide_password else password),\n \"--max_allowed_packet={}\".format(max_allowed_packet),\n \"--hex-blob\", # preferable to raw binary in our .sql file\n ]\n if verbose:\n ca.append(\"--verbose\")\n if with_drop_create_database:\n ca.extend([\n \"--add-drop-database\",\n \"--databases\",\n database\n ])\n else:\n ca.append(database)\n pass\n return ca"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the SQLAlchemy dialect that is used by the given object.", "response": "def get_dialect(mixed: Union[SQLCompiler, Engine, Dialect]) -> Dialect:\n \"\"\"\n Finds the SQLAlchemy dialect in use.\n\n Args:\n mixed: an SQLAlchemy :class:`SQLCompiler`, :class:`Engine`, or\n :class:`Dialect` object\n\n Returns: the SQLAlchemy :class:`Dialect` being used\n\n \"\"\"\n if isinstance(mixed, Dialect):\n return mixed\n elif isinstance(mixed, Engine):\n return mixed.dialect\n elif isinstance(mixed, SQLCompiler):\n return mixed.dialect\n else:\n raise ValueError(\"get_dialect: 'mixed' parameter of wrong type\")"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_dialect_name(mixed: Union[SQLCompiler, Engine, Dialect]) -> str:\n dialect = get_dialect(mixed)\n # noinspection PyUnresolvedReferences\n return dialect.name", "response": "Returns the name of the SQLAlchemy dialect in use."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_preparer(mixed: Union[SQLCompiler, Engine,\n Dialect]) -> IdentifierPreparer:\n \"\"\"\n Returns the SQLAlchemy :class:`IdentifierPreparer` in use for the dialect\n being used.\n\n Args:\n mixed: an SQLAlchemy :class:`SQLCompiler`, :class:`Engine`, or\n :class:`Dialect` object\n\n Returns: an :class:`IdentifierPreparer`\n\n \"\"\"\n dialect = get_dialect(mixed)\n # noinspection PyUnresolvedReferences\n return dialect.preparer(dialect)", "response": "Returns the SQLAlchemy identifier preparer for the given SQLAlchemy object."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef quote_identifier(identifier: str,\n mixed: Union[SQLCompiler, Engine, Dialect]) -> str:\n \"\"\"\n Converts an SQL identifier to a quoted version, via the SQL dialect in\n use.\n\n Args:\n identifier: the identifier to be quoted\n mixed: an SQLAlchemy :class:`SQLCompiler`, :class:`Engine`, or\n :class:`Dialect` object\n\n Returns:\n the quoted identifier\n\n \"\"\"\n # See also http://sqlalchemy-utils.readthedocs.io/en/latest/_modules/sqlalchemy_utils/functions/orm.html # noqa\n return get_preparer(mixed).quote(identifier)", "response": "Converts an SQL identifier to a quoted version via the SQL dialect in\nattery."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef install(force=False, lazy=False):\n\n if exists(antlr_jar_path):\n if lazy:\n return\n if not force:\n raise OSError(errno.EEXIST, 'file already exists', antlr_jar_path)\n\n tool_url = config['tool_url']\n ssl_context = ssl.create_default_context(purpose=ssl.Purpose.CLIENT_AUTH)\n with contextlib.closing(urlopen(tool_url, context=ssl_context)) as response:\n tool_jar = response.read()\n\n if not isdir(dirname(antlr_jar_path)):\n makedirs(dirname(antlr_jar_path))\n\n with open(antlr_jar_path, mode='wb') as tool_file:\n tool_file.write(tool_jar)", "response": "Download the ANTLR v4 tool jar."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef tsv_escape(x: Any) -> str:\n if x is None:\n return \"\"\n x = str(x)\n return x.replace(\"\\t\", \"\\\\t\").replace(\"\\n\", \"\\\\n\")", "response": "Escape a single tab - separated value for TSV format."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef make_tsv_row(values: List[Any]) -> str:\n return \"\\t\".join([tsv_escape(x) for x in values]) + \"\\n\"", "response": "Make a TSV row from a list of values."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef dictlist_to_tsv(dictlist: List[Dict[str, Any]]) -> str:\n if not dictlist:\n return \"\"\n fieldnames = dictlist[0].keys()\n tsv = \"\\t\".join([tsv_escape(f) for f in fieldnames]) + \"\\n\"\n for d in dictlist:\n tsv += \"\\t\".join([tsv_escape(v) for v in d.values()]) + \"\\n\"\n return tsv", "response": "Converts a consistent list of dictionaries mapping fieldnames to values make a TSV file."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef tsv_pairs_to_dict(line: str, key_lower: bool = True) -> Dict[str, str]:\n items = line.split(\"\\t\")\n d = {} # type: Dict[str, str]\n for chunk in chunks(items, 2):\n if len(chunk) < 2:\n log.warning(\"Bad chunk, not of length 2: {!r}\", chunk)\n continue\n key = chunk[0]\n value = unescape_tabs_newlines(chunk[1])\n if key_lower:\n key = key.lower()\n d[key] = value\n return d", "response": "r Converts a TSV line into a dictionary."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef import_submodules(package: Union[str, ModuleType],\n base_package_for_relative_import: str = None,\n recursive: bool = True) -> Dict[str, ModuleType]:\n \"\"\"\n Import all submodules of a module, recursively, including subpackages.\n\n Args:\n package: package (name or actual module)\n base_package_for_relative_import: path to prepend?\n recursive: import submodules too?\n\n Returns:\n dict: mapping from full module name to module\n\n \"\"\"\n # http://stackoverflow.com/questions/3365740/how-to-import-all-submodules\n if isinstance(package, str):\n package = importlib.import_module(package,\n base_package_for_relative_import)\n results = {}\n for loader, name, is_pkg in pkgutil.walk_packages(package.__path__):\n full_name = package.__name__ + '.' + name\n log.debug(\"importing: {}\", full_name)\n results[full_name] = importlib.import_module(full_name)\n if recursive and is_pkg:\n results.update(import_submodules(full_name))\n return results", "response": "Imports all submodules of a module recursively including subpackages."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef is_c_extension(module: ModuleType) -> bool:\n assert inspect.ismodule(module), '\"{}\" not a module.'.format(module)\n\n # If this module was loaded by a PEP 302-compliant CPython-specific loader\n # loading only C extensions, this module is a C extension.\n if isinstance(getattr(module, '__loader__', None), ExtensionFileLoader):\n return True\n\n # If it's built-in, it's not a C extension.\n if is_builtin_module(module):\n return False\n\n # Else, fallback to filetype matching heuristics.\n #\n # Absolute path of the file defining this module.\n module_filename = inspect.getfile(module)\n\n # \".\"-prefixed filetype of this path if any or the empty string otherwise.\n module_filetype = os.path.splitext(module_filename)[1]\n\n # This module is only a C extension if this path's filetype is that of a\n # C extension specific to the current platform.\n return module_filetype in EXTENSION_SUFFIXES", "response": "Tests if a module is a C extension."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef contains_c_extension(module: ModuleType,\n import_all_submodules: bool = True,\n include_external_imports: bool = False,\n seen: List[ModuleType] = None) -> bool:\n \"\"\"\n Extends :func:`is_c_extension` by asking: is this module, or any of its\n submodules, a C extension?\n\n Args:\n module: Previously imported module object to be tested.\n import_all_submodules: explicitly import all submodules of this module?\n include_external_imports: check modules in other packages that this\n module imports?\n seen: used internally for recursion (to deal with recursive modules);\n should be ``None`` when called by users\n\n Returns:\n bool: ``True`` only if this module or one of its submodules is a C\n extension.\n\n Examples:\n\n .. code-block:: python\n\n import logging\n from cardinal_pythonlib.modules import contains_c_extension\n from cardinal_pythonlib.logs import main_only_quicksetup_rootlogger\n \n import _elementtree as et\n import os\n \n import arrow\n import alembic\n import django\n import numpy\n import numpy.core.multiarray as numpy_multiarray\n \n log = logging.getLogger(__name__)\n # logging.basicConfig(level=logging.DEBUG) # be verbose\n main_only_quicksetup_rootlogger(level=logging.DEBUG)\n \n contains_c_extension(os) # False\n contains_c_extension(et) # False\n \n contains_c_extension(numpy) # True -- different from is_c_extension()\n contains_c_extension(numpy_multiarray) # True\n \n contains_c_extension(arrow) # False\n \n contains_c_extension(alembic) # False\n contains_c_extension(alembic, include_external_imports=True) # True\n # ... this example shows that Alembic imports hashlib, which can import\n # _hashlib, which is a C extension; however, that doesn't stop us (for\n # example) installing Alembic on a machine with no C compiler\n \n contains_c_extension(django)\n\n \"\"\" # noqa\n assert inspect.ismodule(module), '\"{}\" not a module.'.format(module)\n\n if seen is None: # only true for the top-level call\n seen = [] # type: List[ModuleType]\n if module in seen: # modules can \"contain\" themselves\n # already inspected; avoid infinite loops\n return False\n seen.append(module)\n\n # Check the thing we were asked about\n is_c_ext = is_c_extension(module)\n log.info(\"Is module {!r} a C extension? {}\", module, is_c_ext)\n if is_c_ext:\n return True\n if is_builtin_module(module):\n # built-in, therefore we stop searching it\n return False\n\n # Now check any children, in a couple of ways\n\n top_level_module = seen[0]\n top_path = os.path.dirname(top_level_module.__file__)\n\n # Recurse using dir(). This picks up modules that are automatically\n # imported by our top-level model. But it won't pick up all submodules;\n # try e.g. for django.\n for candidate_name in dir(module):\n candidate = getattr(module, candidate_name)\n # noinspection PyBroadException\n try:\n if not inspect.ismodule(candidate):\n # not a module\n continue\n except Exception:\n # e.g. a Django module that won't import until we configure its\n # settings\n log.error(\"Failed to test ismodule() status of {!r}\", candidate)\n continue\n if is_builtin_module(candidate):\n # built-in, therefore we stop searching it\n continue\n\n candidate_fname = getattr(candidate, \"__file__\")\n if not include_external_imports:\n if os.path.commonpath([top_path, candidate_fname]) != top_path:\n log.debug(\"Skipping, not within the top-level module's \"\n \"directory: {!r}\", candidate)\n continue\n # Recurse:\n if contains_c_extension(\n module=candidate,\n import_all_submodules=False, # only done at the top level, below # noqa\n include_external_imports=include_external_imports,\n seen=seen):\n return True\n\n if import_all_submodules:\n if not is_module_a_package(module):\n log.debug(\"Top-level module is not a package: {!r}\", module)\n return False\n\n # Otherwise, for things like Django, we need to recurse in a different\n # way to scan everything.\n # See https://stackoverflow.com/questions/3365740/how-to-import-all-submodules. # noqa\n log.debug(\"Walking path: {!r}\", top_path)\n # noinspection PyBroadException\n try:\n for loader, module_name, is_pkg in pkgutil.walk_packages([top_path]): # noqa\n if not is_pkg:\n log.debug(\"Skipping, not a package: {!r}\", module_name)\n continue\n log.debug(\"Manually importing: {!r}\", module_name)\n # noinspection PyBroadException\n try:\n candidate = loader.find_module(module_name)\\\n .load_module(module_name) # noqa\n except Exception:\n # e.g. Alembic \"autogenerate\" gives: \"ValueError: attempted\n # relative import beyond top-level package\"; or Django\n # \"django.core.exceptions.ImproperlyConfigured\"\n log.error(\"Package failed to import: {!r}\", module_name)\n continue\n if contains_c_extension(\n module=candidate,\n import_all_submodules=False, # only done at the top level # noqa\n include_external_imports=include_external_imports,\n seen=seen):\n return True\n except Exception:\n log.error(\"Unable to walk packages further; no C extensions \"\n \"detected so far!\")\n raise\n\n return False", "response": "Returns True if the given module object contains a C extension."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nperform a :func:`shutil.which` command using the PATH from the specified environment. Reason: when you use ``run([executable, ...], env)`` and therefore ``subprocess.run([executable, ...], env=env)``, the PATH that's searched for ``executable`` is the parent's, not the new child's -- so you have to find the executable manually. Args: executable: executable to find env: environment to fetch the PATH variable from", "response": "def which_with_envpath(executable: str, env: Dict[str, str]) -> str:\n \"\"\"\n Performs a :func:`shutil.which` command using the PATH from the specified\n environment.\n\n Reason: when you use ``run([executable, ...], env)`` and therefore\n ``subprocess.run([executable, ...], env=env)``, the PATH that's searched\n for ``executable`` is the parent's, not the new child's -- so you have to\n find the executable manually.\n\n Args:\n executable: executable to find\n env: environment to fetch the PATH variable from\n \"\"\"\n oldpath = os.environ.get(\"PATH\", \"\")\n os.environ[\"PATH\"] = env.get(\"PATH\")\n which = shutil.which(executable)\n os.environ[\"PATH\"] = oldpath\n return which"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nrequiring that the given executable is available on the PATH.", "response": "def require_executable(executable: str) -> None:\n \"\"\"\n If ``executable`` is not found by :func:`shutil.which`, raise\n :exc:`FileNotFoundError`.\n \"\"\"\n if shutil.which(executable):\n return\n errmsg = \"Missing command (must be on the PATH): \" + executable\n log.critical(errmsg)\n raise FileNotFoundError(errmsg)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nmake a directory and any intermediate directories if required.", "response": "def mkdir_p(path: str) -> None:\n \"\"\"\n Makes a directory, and any intermediate (parent) directories if required.\n\n This is the UNIX ``mkdir -p DIRECTORY`` command; of course, we use\n :func:`os.makedirs` instead, for portability.\n \"\"\"\n log.debug(\"mkdir -p \" + path)\n os.makedirs(path, exist_ok=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef preserve_cwd(func: Callable) -> Callable:\n # http://stackoverflow.com/questions/169070/python-how-do-i-write-a-decorator-that-restores-the-cwd # noqa\n def decorator(*args_, **kwargs) -> Any:\n cwd = os.getcwd()\n result = func(*args_, **kwargs)\n os.chdir(cwd)\n return result\n return decorator", "response": "Decorator to preserve the current working directory in calls to the\n decorated function."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef copyglob(src: str, dest: str, allow_nothing: bool = False,\n allow_nonfiles: bool = False) -> None:\n \"\"\"\n Copies files whose filenames match the glob src\" into the directory\n \"dest\". Raises an error if no files are copied, unless allow_nothing is\n True.\n\n Args:\n src: source glob (e.g. ``/somewhere/*.txt``)\n dest: destination directory\n allow_nothing: don't raise an exception if no files are found\n allow_nonfiles: copy things that are not files too (as judged by\n :func:`os.path.isfile`).\n\n Raises:\n ValueError: if no files are found and ``allow_nothing`` is not set\n \"\"\"\n something = False\n for filename in glob.glob(src):\n if allow_nonfiles or os.path.isfile(filename):\n shutil.copy(filename, dest)\n something = True\n if something or allow_nothing:\n return\n raise ValueError(\"No files found matching: {}\".format(src))", "response": "Copies files whose filenames match the glob src into the directory\n dest."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef copy_tree_root(src_dir: str, dest_parent: str) -> None:\n dirname = os.path.basename(os.path.normpath(src_dir))\n dest_dir = os.path.join(dest_parent, dirname)\n shutil.copytree(src_dir, dest_dir)", "response": "Copies a directory src_dir into dest_parent."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef copy_tree_contents(srcdir: str, destdir: str,\n destroy: bool = False) -> None:\n \"\"\"\n Recursive copy. Unlike :func:`copy_tree_root`, :func:`copy_tree_contents`\n works as follows. With the file structure:\n\n .. code-block:: none\n\n /source/thing/a.txt\n /source/thing/b.txt\n /source/thing/somedir/c.txt\n\n the command\n\n .. code-block:: python\n\n copy_tree_contents(\"/source/thing\", \"/dest\")\n\n ends up creating:\n\n .. code-block:: none\n\n /dest/a.txt\n /dest/b.txt\n /dest/somedir/c.txt\n\n \"\"\"\n log.info(\"Copying directory {} -> {}\", srcdir, destdir)\n if os.path.exists(destdir):\n if not destroy:\n raise ValueError(\"Destination exists!\")\n if not os.path.isdir(destdir):\n raise ValueError(\"Destination exists but isn't a directory!\")\n log.debug(\"... removing old contents\")\n rmtree(destdir)\n log.debug(\"... now copying\")\n shutil.copytree(srcdir, destdir)", "response": "Recursively copy the contents of srcdir to destdir."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rmglob(pattern: str) -> None:\n for f in glob.glob(pattern):\n os.remove(f)", "response": "Deletes all files whose filename matches the glob pattern."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef purge(path: str, pattern: str) -> None:\n for f in find(pattern, path):\n log.info(\"Deleting {}\", f)\n os.remove(f)", "response": "Delete all files in path that match pattern."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndeleting files within a directory.", "response": "def delete_files_within_dir(directory: str, filenames: List[str]) -> None:\n \"\"\"\n Delete files within ``directory`` whose filename *exactly* matches one of\n ``filenames``.\n \"\"\"\n for dirpath, dirnames, fnames in os.walk(directory):\n for f in fnames:\n if f in filenames:\n fullpath = os.path.join(dirpath, f)\n log.debug(\"Deleting {!r}\", fullpath)\n os.remove(fullpath)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef shutil_rmtree_onerror(func: Callable[[str], None],\n path: str,\n exc_info: EXC_INFO_TYPE) -> None:\n \"\"\"\n Error handler for ``shutil.rmtree``.\n\n If the error is due to an access error (read only file)\n it attempts to add write permission and then retries.\n\n If the error is for another reason it re-raises the error.\n\n Usage: ``shutil.rmtree(path, onerror=shutil_rmtree_onerror)``\n \n See\n https://stackoverflow.com/questions/2656322/shutil-rmtree-fails-on-windows-with-access-is-denied\n \"\"\" # noqa\n if not os.access(path, os.W_OK):\n # Is the error an access error ?\n os.chmod(path, stat.S_IWUSR)\n func(path)\n else:\n exc = exc_info[1]\n raise exc", "response": "Error handler for shutil. rmtree."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef rmtree(directory: str) -> None:\n log.debug(\"Deleting directory {!r}\", directory)\n shutil.rmtree(directory, onerror=shutil_rmtree_onerror)", "response": "Deletes a directory tree."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef chown_r(path: str, user: str, group: str) -> None:\n for root, dirs, files in os.walk(path):\n for x in dirs:\n shutil.chown(os.path.join(root, x), user, group)\n for x in files:\n shutil.chown(os.path.join(root, x), user, group)", "response": "Performs a recursive chown on the given path."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef find(pattern: str, path: str) -> List[str]:\n result = []\n for root, dirs, files in os.walk(path):\n for name in files:\n if fnmatch.fnmatch(name, pattern):\n result.append(os.path.join(root, name))\n return result", "response": "Find files in path whose filenames match pattern."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef find_first(pattern: str, path: str) -> str:\n try:\n return find(pattern, path)[0]\n except IndexError:\n log.critical('''Couldn't find \"{}\" in \"{}\"''', pattern, path)\n raise", "response": "Find first file in path whose filename matches pattern."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef gen_filenames(starting_filenames: List[str],\n recursive: bool) -> Generator[str, None, None]:\n \"\"\"\n From a starting list of files and/or directories, generates filenames of\n all files in the list, and (if ``recursive`` is set) all files within\n directories in the list.\n\n Args:\n starting_filenames: files and/or directories\n recursive: walk down any directories in the starting list, recursively?\n\n Yields:\n each filename\n\n \"\"\"\n for base_filename in starting_filenames:\n if os.path.isfile(base_filename):\n yield os.path.abspath(base_filename)\n elif os.path.isdir(base_filename) and recursive:\n for dirpath, dirnames, filenames in os.walk(base_filename):\n for fname in filenames:\n yield os.path.abspath(os.path.join(dirpath, fname))", "response": "Generates filenames of\n archive from a starting list of files and directories."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if a file is locked by opening it in append mode.", "response": "def exists_locked(filepath: str) -> Tuple[bool, bool]:\n \"\"\"\n Checks if a file is locked by opening it in append mode.\n (If no exception is thrown in that situation, then the file is not locked.)\n\n Args:\n filepath: file to check\n\n Returns:\n tuple: ``(exists, locked)``\n\n See https://www.calazan.com/how-to-check-if-a-file-is-locked-in-python/.\n \"\"\"\n exists = False\n locked = None\n file_object = None\n if os.path.exists(filepath):\n exists = True\n locked = True\n try:\n buffer_size = 8\n # Opening file in append mode and read the first 8 characters.\n file_object = open(filepath, 'a', buffer_size)\n if file_object:\n locked = False # exists and not locked\n except IOError:\n pass\n finally:\n if file_object:\n file_object.close()\n return exists, locked"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the part of the filename that is relative to the directory directory.", "response": "def relative_filename_within_dir(filename: str, directory: str) -> str:\n \"\"\"\n Starting with a (typically absolute) ``filename``, returns the part of the\n filename that is relative to the directory ``directory``.\n If the file is *not* within the directory, returns an empty string.\n \"\"\"\n filename = os.path.abspath(filename)\n directory = os.path.abspath(directory)\n if os.path.commonpath([directory, filename]) != directory:\n # Filename is not within directory\n return \"\"\n return os.path.relpath(filename, start=directory)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _get_errors(self):\n errors = self.json.get('data').get('failures')\n if errors:\n logger.error(errors)\n return errors", "response": "Gets errors from HTTP response"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef page_sequence(n_sheets: int, one_based: bool = True) -> List[int]:\n n_pages = calc_n_virtual_pages(n_sheets)\n assert n_pages % 4 == 0\n half_n_pages = n_pages // 2\n firsthalf = list(range(half_n_pages))\n secondhalf = list(reversed(range(half_n_pages, n_pages)))\n # Seen from the top of an UNFOLDED booklet (e.g. a stack of paper that's\n # come out of your printer), \"firsthalf\" are on the right (from top to\n # bottom: recto facing up, then verso facing down, then recto, then verso)\n # and \"secondhalf\" are on the left (from top to bottom: verso facing up,\n # then recto facing down, etc.).\n sequence = [] # type: List[int]\n top = True\n for left, right in zip(secondhalf, firsthalf):\n if not top:\n left, right = right, left\n sequence += [left, right]\n top = not top\n if one_based:\n sequence = [x + 1 for x in sequence]\n log.debug(\"{} sheets => page sequence {!r}\", n_sheets, sequence)\n return sequence", "response": "Generates the final page sequence from the starting number of sheets."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef require(executable: str, explanation: str = \"\") -> None:\n assert shutil.which(executable), \"Need {!r} on the PATH.{}\".format(\n executable, \"\\n\" + explanation if explanation else \"\")", "response": "Ensures that the external tool is available on the PATH."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef run(args: List[str],\n get_output: bool = False,\n encoding: str = sys.getdefaultencoding()) -> Tuple[str, str]:\n \"\"\"\n Run an external command +/- return the results.\n Returns a ``(stdout, stderr)`` tuple (both are blank strings if the output\n wasn't wanted).\n \"\"\"\n printable = \" \".join(shlex.quote(x) for x in args).replace(\"\\n\", r\"\\n\")\n log.debug(\"Running external command: {}\", printable)\n if get_output:\n p = subprocess.run(args, stdout=subprocess.PIPE,\n stderr=subprocess.PIPE, check=True)\n stdout, stderr = p.stdout.decode(encoding), p.stderr.decode(encoding)\n else:\n subprocess.check_call(args)\n stdout, stderr = \"\", \"\"\n return stdout, stderr", "response": "Run an external command + / - return the results."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_page_count(filename: str) -> int:\n log.debug(\"Getting page count for {!r}\", filename)\n require(PDFTK, HELP_MISSING_PDFTK)\n stdout, _ = run([PDFTK, filename, \"dump_data\"], get_output=True)\n regex = re.compile(r\"^NumberOfPages: (\\d+)$\", re.MULTILINE)\n m = regex.search(stdout)\n if m:\n return int(m.group(1))\n raise ValueError(\"Can't get PDF page count for: {!r}\".format(filename))", "response": "Get the number of pages in a PDF file."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nmake a blank PDF from a single page PDF.", "response": "def make_blank_pdf(filename: str, paper: str = \"A4\") -> None:\n \"\"\"\n NOT USED.\n Makes a blank single-page PDF, using ImageMagick's ``convert``.\n \"\"\"\n # https://unix.stackexchange.com/questions/277892/how-do-i-create-a-blank-pdf-from-the-command-line # noqa\n require(CONVERT, HELP_MISSING_IMAGEMAGICK)\n run([CONVERT, \"xc:none\", \"-page\", paper, filename])"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef slice_pdf(input_filename: str, output_filename: str,\n slice_horiz: int, slice_vert: int) -> str:\n \"\"\"\n Slice each page of the original, to convert to \"one real page per PDF\n page\". Return the output filename.\n \"\"\"\n if slice_horiz == 1 and slice_vert == 1:\n log.debug(\"No slicing required\")\n return input_filename # nothing to do\n log.info(\"Slicing each source page mv into {} horizontally x {} vertically\",\n slice_horiz, slice_vert)\n log.debug(\"... from {!r} to {!r}\", input_filename, output_filename)\n require(MUTOOL, HELP_MISSING_MUTOOL)\n run([\n MUTOOL,\n \"poster\",\n \"-x\", str(slice_horiz),\n \"-y\", str(slice_vert),\n input_filename,\n output_filename\n ])\n return output_filename", "response": "Slice each page of the original to one real page per PDF\n page. Return the output filename."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef booklet_nup_pdf(input_filename: str, output_filename: str,\n latex_paper_size: str = LATEX_PAPER_SIZE_A4) -> str:\n \"\"\"\n Takes a PDF (e.g. A4) and makes a 2x1 booklet (e.g. 2xA5 per A4).\n The booklet can be folded like a book and the final pages will be in order.\n Returns the output filename.\n \"\"\"\n log.info(\"Creating booklet\")\n log.debug(\"... {!r} -> {!r}\", input_filename, output_filename)\n require(PDFJAM, HELP_MISSING_PDFJAM)\n n_pages = get_page_count(input_filename)\n n_sheets = calc_n_sheets(n_pages)\n log.debug(\"{} pages => {} sheets\", n_pages, n_sheets)\n pagenums = page_sequence(n_sheets, one_based=True)\n pagespeclist = [str(p) if p <= n_pages else \"{}\"\n for p in pagenums]\n # ... switches empty pages to \"{}\", which is pdfjam notation for\n # an empty page.\n pagespec = \",\".join(pagespeclist)\n pdfjam_tidy = True # clean up after yourself?\n args = [\n PDFJAM,\n \"--paper\", latex_paper_size,\n \"--landscape\",\n \"--nup\", \"2x1\",\n \"--keepinfo\", # e.g. author information\n \"--outfile\", output_filename,\n \"--tidy\" if pdfjam_tidy else \"--no-tidy\",\n \"--\", # \"no more options\"\n input_filename, pagespec\n ]\n run(args)\n return output_filename", "response": "Creates a 2x1 booklet from a PDF file."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef rotate_even_pages_180(input_filename: str, output_filename: str) -> str:\n log.info(\"Rotating even-numbered pages 180 degrees for long-edge \"\n \"duplex printing\")\n log.debug(\"... {!r} -> {!r}\", input_filename, output_filename)\n require(PDFTK, HELP_MISSING_PDFTK)\n args = [\n PDFTK,\n \"A=\" + input_filename, # give it handle 'A'\n # handles are one or more UPPER CASE letters\n \"shuffle\",\n \"Aoddnorth\", # for 'A', keep odd pages as they are\n \"Aevensouth\", # for 'A', rotate even pages 180 degrees\n \"output\", output_filename,\n ]\n run(args)\n return output_filename", "response": "Rotates even - numbered pages 180 degrees."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef convert_to_foldable(input_filename: str,\n output_filename: str,\n slice_horiz: int,\n slice_vert: int,\n overwrite: bool = False,\n longedge: bool = False,\n latex_paper_size: str = LATEX_PAPER_SIZE_A4) -> bool:\n \"\"\"\n Runs a chain of tasks to convert a PDF to a useful booklet PDF.\n \"\"\"\n if not os.path.isfile(input_filename):\n log.warning(\"Input file does not exist or is not a file\")\n return False\n if not overwrite and os.path.isfile(output_filename):\n log.error(\"Output file exists; not authorized to overwrite (use \"\n \"--overwrite if you are sure)\")\n return False\n log.info(\"Processing {!r}\", input_filename)\n with tempfile.TemporaryDirectory() as tmpdir:\n log.debug(\"Using temporary directory {!r}\", tmpdir)\n intermediate_num = 0\n\n def make_intermediate() -> str:\n nonlocal intermediate_num\n intermediate_num += 1\n return os.path.join(tmpdir,\n \"intermediate_{}.pdf\".format(intermediate_num))\n\n # Run this as a chain, rewriting input_filename at each step:\n # Slice, if necessary.\n input_filename = slice_pdf(\n input_filename=input_filename,\n output_filename=make_intermediate(),\n slice_horiz=slice_horiz,\n slice_vert=slice_vert\n )\n # Make the final n-up\n input_filename = booklet_nup_pdf(\n input_filename=input_filename,\n output_filename=make_intermediate(),\n latex_paper_size=latex_paper_size\n )\n # Rotate?\n if longedge:\n input_filename = rotate_even_pages_180(\n input_filename=input_filename,\n output_filename=make_intermediate(),\n )\n # Done.\n log.info(\"Writing to {!r}\", output_filename)\n shutil.move(input_filename, output_filename)\n return True", "response": "Convert a PDF file to a useful booklet PDF file."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_mysql_vars(mysql: str,\n host: str,\n port: int,\n user: str) -> Dict[str, str]:\n \"\"\"\n Asks MySQL for its variables and status.\n\n Args:\n mysql: ``mysql`` executable filename\n host: host name\n port: TCP/IP port number\n user: username\n\n Returns:\n dictionary of MySQL variables/values\n\n \"\"\"\n cmdargs = [\n mysql,\n \"-h\", host,\n \"-P\", str(port),\n \"-e\", \"SHOW VARIABLES; SHOW STATUS\",\n \"-u\", user,\n \"-p\" # prompt for password\n ]\n log.info(\"Connecting to MySQL with user: {}\", user)\n log.debug(cmdargs)\n process = subprocess.Popen(cmdargs, stdout=subprocess.PIPE)\n out, err = process.communicate()\n lines = out.decode(\"utf8\").splitlines()\n mysqlvars = {}\n for line in lines:\n var, val = line.split(\"\\t\")\n mysqlvars[var] = val\n return mysqlvars", "response": "Asks MySQL for its variables and status."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert a value in bytes to megabytes.", "response": "def val_mb(valstr: Union[int, str]) -> str:\n \"\"\"\n Converts a value in bytes (in string format) to megabytes.\n \"\"\"\n try:\n return \"{:.3f}\".format(int(valstr) / (1024 * 1024))\n except (TypeError, ValueError):\n return '?'"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef add_var_mb(table: PrettyTable,\n vardict: Dict[str, str],\n varname: str) -> None:\n \"\"\"\n Adds a row to ``table`` for ``varname``, in megabytes.\n \"\"\"\n valstr = vardict.get(varname, None)\n table.add_row([varname, val_mb(valstr), UNITS_MB])", "response": "Adds a row to table for varname in megabytes."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncommands - line processor for the formula.", "response": "def main():\n \"\"\"\n Command-line processor. See ``--help`` for details.\n \"\"\"\n main_only_quicksetup_rootlogger(level=logging.DEBUG)\n parser = argparse.ArgumentParser()\n parser.add_argument(\n \"--mysql\", default=\"mysql\",\n help=\"MySQL program (default=mysql)\")\n parser.add_argument(\n \"--host\", default=\"127.0.0.1\",\n help=\"MySQL server/host (prefer '127.0.0.1' to 'localhost')\")\n parser.add_argument(\n \"--port\", type=int, default=MYSQL_DEFAULT_PORT,\n help=\"MySQL port (default={})\".format(MYSQL_DEFAULT_PORT))\n parser.add_argument(\n \"--user\", default=MYSQL_DEFAULT_USER,\n help=\"MySQL user (default={})\".format(MYSQL_DEFAULT_USER))\n args = parser.parse_args()\n\n vardict = get_mysql_vars(\n mysql=args.mysql,\n host=args.host,\n port=args.port,\n user=args.user,\n )\n max_conn = int(vardict[\"max_connections\"])\n max_used_conn = int(vardict[\"Max_used_connections\"])\n base_mem = (\n int(vardict[\"key_buffer_size\"]) +\n int(vardict[\"query_cache_size\"]) +\n int(vardict[\"innodb_buffer_pool_size\"]) +\n # int(vardict[\"innodb_additional_mem_pool_size\"]) +\n int(vardict[\"innodb_log_buffer_size\"])\n )\n mem_per_conn = (\n int(vardict[\"read_buffer_size\"]) +\n int(vardict[\"read_rnd_buffer_size\"]) +\n int(vardict[\"sort_buffer_size\"]) +\n int(vardict[\"join_buffer_size\"]) +\n int(vardict[\"binlog_cache_size\"]) +\n int(vardict[\"thread_stack\"]) +\n int(vardict[\"tmp_table_size\"])\n )\n mem_total_min = base_mem + mem_per_conn * max_used_conn\n mem_total_max = base_mem + mem_per_conn * max_conn\n\n table = PrettyTable([\"Variable\", \"Value\", \"Units\"])\n table.align[\"Variable\"] = \"l\"\n table.align[\"Value\"] = \"r\"\n table.align[\"Units\"] = \"l\"\n\n add_var_mb(table, vardict, \"key_buffer_size\")\n add_var_mb(table, vardict, \"query_cache_size\")\n add_var_mb(table, vardict, \"innodb_buffer_pool_size\")\n # print_var_mb(table, vardict, \"innodb_additional_mem_pool_size\")\n add_var_mb(table, vardict, \"innodb_log_buffer_size\")\n add_blank_row(table)\n table.add_row([\"BASE MEMORY\", val_mb(base_mem), UNITS_MB])\n add_blank_row(table)\n add_var_mb(table, vardict, \"sort_buffer_size\")\n add_var_mb(table, vardict, \"read_buffer_size\")\n add_var_mb(table, vardict, \"read_rnd_buffer_size\")\n add_var_mb(table, vardict, \"join_buffer_size\")\n add_var_mb(table, vardict, \"thread_stack\")\n add_var_mb(table, vardict, \"binlog_cache_size\")\n add_var_mb(table, vardict, \"tmp_table_size\")\n add_blank_row(table)\n table.add_row([\"MEMORY PER CONNECTION\", val_mb(mem_per_conn), UNITS_MB])\n add_blank_row(table)\n table.add_row([\"Max_used_connections\", val_int(max_used_conn), ''])\n table.add_row([\"max_connections\", val_int(max_conn), ''])\n add_blank_row(table)\n table.add_row([\"TOTAL (MIN)\", val_mb(mem_total_min), UNITS_MB])\n table.add_row([\"TOTAL (MAX)\", val_mb(mem_total_max), UNITS_MB])\n\n print(table.get_string())"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nprint either the. zip filename or the. zip filename of the inner file.", "response": "def report_hit_filename(zipfilename: str, contentsfilename: str,\n show_inner_file: bool) -> None:\n \"\"\"\n For \"hits\": prints either the ``.zip`` filename, or the ``.zip`` filename\n and the inner filename.\n\n Args:\n zipfilename: filename of the ``.zip`` file\n contentsfilename: filename of the inner file\n show_inner_file: if ``True``, show both; if ``False``, show just the\n ``.zip`` filename\n\n Returns:\n\n \"\"\"\n if show_inner_file:\n print(\"{} [{}]\".format(zipfilename, contentsfilename))\n else:\n print(zipfilename)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef report_line(zipfilename: str, contentsfilename: str, line: str,\n show_inner_file: bool) -> None:\n \"\"\"\n Prints a line from a file, with the ``.zip`` filename and optionally also\n the inner filename.\n\n Args:\n zipfilename: filename of the ``.zip`` file\n contentsfilename: filename of the inner file\n line: the line from the inner file\n show_inner_file: if ``True``, show both filenames; if ``False``, show\n just the ``.zip`` filename\n \"\"\"\n if show_inner_file:\n print(\"{} [{}]: {}\".format(zipfilename, contentsfilename, line))\n else:\n print(\"{}: {}\".format(zipfilename, line))", "response": "Prints a line from a file with the. zip filename and optionally also the inner filename."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse_zip(zipfilename: str,\n regex: Pattern,\n invert_match: bool,\n files_with_matches: bool,\n files_without_match: bool,\n grep_inner_file_name: bool,\n show_inner_file: bool) -> None:\n \"\"\"\n Implement a \"grep within an OpenXML file\" for a single OpenXML file, which\n is by definition a ``.zip`` file.\n\n Args:\n zipfilename: name of the OpenXML (zip) file\n regex: regular expression to match\n invert_match: find files that do NOT match, instead of ones that do?\n files_with_matches: show filenames of files with a match?\n files_without_match: show filenames of files with no match?\n grep_inner_file_name: search the names of \"inner\" files, rather than\n their contents?\n show_inner_file: show the names of the \"inner\" files, not just the\n \"outer\" (OpenXML) file?\n \"\"\"\n assert not (files_without_match and files_with_matches)\n report_lines = (not files_without_match) and (not files_with_matches)\n report_hit_lines = report_lines and not invert_match\n report_miss_lines = report_lines and invert_match\n log.debug(\"Checking ZIP: \" + zipfilename)\n found_in_zip = False\n try:\n with ZipFile(zipfilename, 'r') as zf:\n for contentsfilename in zf.namelist():\n log.debug(\"... checking file: \" + contentsfilename)\n if grep_inner_file_name:\n found_in_filename = bool(regex.search(contentsfilename))\n found_in_zip = found_in_zip or found_in_filename\n if files_with_matches and found_in_zip:\n report_hit_filename(zipfilename, contentsfilename,\n show_inner_file)\n return\n if ((report_hit_lines and found_in_filename) or\n (report_miss_lines and not found_in_filename)):\n report_line(zipfilename, contentsfilename,\n contentsfilename, show_inner_file)\n else:\n try:\n with zf.open(contentsfilename, 'r') as file:\n try:\n for line in file.readlines():\n # log.debug(\"line: {!r}\", line)\n found_in_line = bool(regex.search(line))\n found_in_zip = found_in_zip or found_in_line\n if files_with_matches and found_in_zip:\n report_hit_filename(zipfilename,\n contentsfilename,\n show_inner_file)\n return\n if ((report_hit_lines and found_in_line) or\n (report_miss_lines and\n not found_in_line)):\n report_line(zipfilename,\n contentsfilename,\n line, show_inner_file)\n except EOFError:\n pass\n except RuntimeError as e:\n log.warning(\n \"RuntimeError whilst processing {} [{}]: probably \"\n \"encrypted contents; error was {!r}\",\n zipfilename, contentsfilename, e)\n except (zlib.error, BadZipFile) as e:\n log.debug(\"Invalid zip: {}; error was {!r}\", zipfilename, e)\n if files_without_match and not found_in_zip:\n report_miss_filename(zipfilename)", "response": "Parse a zip file into a set of items."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a dictionary of all the timelines that are amalgamated for each patient.", "response": "def drug_timelines(\r\n drug_events_df: DataFrame,\r\n event_lasts_for: datetime.timedelta,\r\n patient_colname: str = DEFAULT_PATIENT_COLNAME,\r\n event_datetime_colname: str = DEFAULT_DRUG_EVENT_DATETIME_COLNAME) \\\r\n -> Dict[Any, IntervalList]:\r\n \"\"\"\r\n Takes a set of drug event start times (one or more per patient), plus a\r\n fixed time that each event is presumed to last for, and returns an\r\n :class:`IntervalList` for each patient representing the set of events\r\n (which may overlap, in which case they will be amalgamated).\r\n\r\n Args:\r\n drug_events_df:\r\n pandas :class:`DataFrame` containing the event data\r\n event_lasts_for:\r\n when an event occurs, how long is it assumed to last for? For\r\n example, if a prescription of lithium occurs on 2001-01-01, how\r\n long is the patient presumed to be taking lithium as a consequence\r\n (e.g. 1 day? 28 days? 6 months?)\r\n patient_colname:\r\n name of the column in ``drug_events_df`` containing the patient ID\r\n event_datetime_colname:\r\n name of the column in ``drug_events_df`` containing the date/time\r\n of each event\r\n\r\n Returns:\r\n dict: mapping patient ID to a :class:`IntervalList` object indicating\r\n the amalgamated intervals from the events\r\n\r\n \"\"\"\r\n sourcecolnum_pt = drug_events_df.columns.get_loc(patient_colname)\r\n sourcecolnum_when = drug_events_df.columns.get_loc(event_datetime_colname)\r\n timelines = defaultdict(IntervalList)\r\n nrows = len(drug_events_df)\r\n for rowidx in range(nrows):\r\n patient_id = drug_events_df.iat[rowidx, sourcecolnum_pt]\r\n event_when = drug_events_df.iat[rowidx, sourcecolnum_when]\r\n interval = Interval(event_when, event_when + event_lasts_for)\r\n ivlist = timelines[patient_id] # will create if unknown\r\n ivlist.add(interval)\r\n return timelines"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a DataFrame containing the cumulative time of the drug events and the time of the last event in the query.", "response": "def cumulative_time_on_drug(\r\n drug_events_df: DataFrame,\r\n query_times_df: DataFrame,\r\n event_lasts_for_timedelta: datetime.timedelta = None,\r\n event_lasts_for_quantity: float = None,\r\n event_lasts_for_units: str = None,\r\n patient_colname: str = DEFAULT_PATIENT_COLNAME,\r\n event_datetime_colname: str = DEFAULT_DRUG_EVENT_DATETIME_COLNAME,\r\n start_colname: str = DEFAULT_START_DATETIME_COLNAME,\r\n when_colname: str = DEFAULT_QUERY_DATETIME_COLNAME,\r\n include_timedelta_in_output: bool = False,\r\n debug: bool = False) \\\r\n -> DataFrame:\r\n \"\"\"\r\n\r\n Args:\r\n drug_events_df:\r\n pandas :class:`DataFrame` containing the event data, with columns\r\n named according to ``patient_colname``, ``event_datetime_colname``\r\n event_lasts_for_timedelta:\r\n when an event occurs, how long is it assumed to last for? For\r\n example, if a prescription of lithium occurs on 2001-01-01, how\r\n long is the patient presumed to be taking lithium as a consequence\r\n (e.g. 1 day? 28 days? 6 months?)\r\n event_lasts_for_quantity:\r\n as an alternative to ``event_lasts_for_timedelta``, particularly if\r\n you are calling from R to Python via ``reticulate`` (which doesn't\r\n convert R ``as.difftime()`` to Python ``datetime.timedelta``), you\r\n can specify ``event_lasts_for_quantity``, a number and\r\n ``event_lasts_for_units`` (q.v.).\r\n event_lasts_for_units:\r\n specify the units for ``event_lasts_for_quantity`` (q.v.), if used;\r\n e.g. ``\"days\"``. The string value must be the name of an argument\r\n to the Python ``datetime.timedelta`` constructor.\r\n query_times_df:\r\n times to query for, with columns named according to\r\n ``patient_colname``, ``start_colname``, and ``when_colname``\r\n patient_colname:\r\n name of the column in ``drug_events_df`` and ``query_time_df``\r\n containing the patient ID\r\n event_datetime_colname:\r\n name of the column in ``drug_events_df`` containing the date/time\r\n of each event\r\n start_colname:\r\n name of the column in ``query_time_df`` containing the date/time\r\n representing the overall start time for the relevant patient (from\r\n which cumulative times are calculated)\r\n when_colname:\r\n name of the column in ``query_time_df`` containing date/time\r\n values at which to query\r\n include_timedelta_in_output:\r\n include ``datetime.timedelta`` values in the output? The default is\r\n ``False`` as this isn't supported by R/``reticulate``.\r\n debug:\r\n print debugging information to the log?\r\n\r\n Returns:\r\n :class:`DataFrame` with the requested data\r\n\r\n \"\"\"\r\n if event_lasts_for_timedelta is None:\r\n assert event_lasts_for_quantity and event_lasts_for_units\r\n timedelta_dict = {event_lasts_for_units: event_lasts_for_quantity}\r\n event_lasts_for_timedelta = datetime.timedelta(**timedelta_dict)\r\n if debug:\r\n log.critical(\"drug_events_df:\\n{!r}\", drug_events_df)\r\n log.critical(\"event_lasts_for:\\n{!r}\", event_lasts_for_timedelta)\r\n log.critical(\"query_times_df:\\n{!r}\", query_times_df)\r\n timelines = drug_timelines(\r\n drug_events_df=drug_events_df,\r\n event_lasts_for=event_lasts_for_timedelta,\r\n patient_colname=patient_colname,\r\n event_datetime_colname=event_datetime_colname,\r\n )\r\n query_nrow = len(query_times_df)\r\n ct_coldefs = [ # column definitions:\r\n (RCN_PATIENT_ID, DTYPE_STRING),\r\n (RCN_START, DTYPE_DATETIME),\r\n (RCN_TIME, DTYPE_DATETIME),\r\n (RCN_BEFORE_DAYS, DTYPE_FLOAT),\r\n (RCN_DURING_DAYS, DTYPE_FLOAT),\r\n (RCN_AFTER_DAYS, DTYPE_FLOAT),\r\n ]\r\n if include_timedelta_in_output:\r\n ct_coldefs.extend([\r\n (RCN_BEFORE_TIMEDELTA, DTYPE_TIMEDELTA),\r\n (RCN_DURING_TIMEDELTA, DTYPE_TIMEDELTA),\r\n (RCN_AFTER_TIMEDELTA, DTYPE_TIMEDELTA),\r\n ])\r\n ct_arr = array([None] * query_nrow, dtype=ct_coldefs)\r\n # log.debug(\"ct_arr:\\n{!r}\", ct_arr)\r\n cumulative_times = DataFrame(ct_arr, index=list(range(query_nrow)))\r\n # log.debug(\"cumulative_times:\\n{!r}\", cumulative_times)\r\n # So we can use the fast \"iat\" function.\r\n sourcecolnum_pt = query_times_df.columns.get_loc(patient_colname)\r\n sourcecolnum_start = query_times_df.columns.get_loc(start_colname)\r\n sourcecolnum_when = query_times_df.columns.get_loc(when_colname)\r\n dest_colnum_pt = cumulative_times.columns.get_loc(RCN_PATIENT_ID)\r\n dest_colnum_start = cumulative_times.columns.get_loc(RCN_START)\r\n dest_colnum_t = cumulative_times.columns.get_loc(RCN_TIME)\r\n dest_colnum_before_days = cumulative_times.columns.get_loc(RCN_BEFORE_DAYS)\r\n dest_colnum_during_days = cumulative_times.columns.get_loc(RCN_DURING_DAYS)\r\n dest_colnum_after_days = cumulative_times.columns.get_loc(RCN_AFTER_DAYS)\r\n if include_timedelta_in_output:\r\n dest_colnum_before_dt = cumulative_times.columns.get_loc(RCN_BEFORE_TIMEDELTA) # noqa\r\n dest_colnum_during_dt = cumulative_times.columns.get_loc(RCN_DURING_TIMEDELTA) # noqa\r\n dest_colnum_after_dt = cumulative_times.columns.get_loc(RCN_AFTER_TIMEDELTA) # noqa\r\n else:\r\n # for type checker\r\n dest_colnum_before_dt = 0\r\n dest_colnum_during_dt = 0\r\n dest_colnum_after_dt = 0\r\n for rowidx in range(query_nrow):\r\n patient_id = query_times_df.iat[rowidx, sourcecolnum_pt]\r\n start = query_times_df.iat[rowidx, sourcecolnum_start]\r\n when = query_times_df.iat[rowidx, sourcecolnum_when]\r\n ivlist = timelines[patient_id]\r\n # log.critical(\"ivlist: {!r}\", ivlist)\r\n before, during, after = ivlist.cumulative_before_during_after(start,\r\n when)\r\n # log.critical(\r\n # \"{!r}.cumulative_before_during_after(start={!r}, when={!r}) \"\r\n # \"-> {!r}, {!r}, {!r}\",\r\n # ivlist, start, when,\r\n # before, during, after\r\n # )\r\n cumulative_times.iat[rowidx, dest_colnum_pt] = patient_id\r\n cumulative_times.iat[rowidx, dest_colnum_start] = start\r\n cumulative_times.iat[rowidx, dest_colnum_t] = when\r\n cumulative_times.iat[rowidx, dest_colnum_before_days] = before.days\r\n cumulative_times.iat[rowidx, dest_colnum_during_days] = during.days\r\n cumulative_times.iat[rowidx, dest_colnum_after_days] = after.days\r\n if include_timedelta_in_output:\r\n cumulative_times.iat[rowidx, dest_colnum_before_dt] = before\r\n cumulative_times.iat[rowidx, dest_colnum_during_dt] = during\r\n cumulative_times.iat[rowidx, dest_colnum_after_dt] = after\r\n return cumulative_times"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfind an element by xpath.", "response": "def find_element_by_xpath(self, xpath):\n \"\"\"\n Finds an element by xpath.\n\n :param xpath: The xpath locator of the element to find.\n :return: ElemLxml\n \n See lxml xpath expressions `here <http://lxml.de/xpathxslt.html#xpath>`_\n \"\"\"\n elems = self.find_elements_by_xpath(xpath)\n if isinstance(elems, list) and len(elems) > 0:\n return elems[0]"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfinds multiple elements by xpath.", "response": "def find_elements_by_xpath(self, xpath):\n \"\"\"\n Finds multiple elements by xpath.\n\n :param xpath: The xpath locator of the elements to be found.\n :return: list of ElemLxml\n \n See lxml xpath expressions `here <http://lxml.de/xpathxslt.html#xpath>`_\n \"\"\"\n from sdklib.html.elem import ElemLxml\n\n elements = self.html_obj.xpath(xpath)\n return [ElemLxml(e) for e in elements]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef find_element_by_xpath(self, xpath):\n from sdklib.html.elem import Elem5lib\n\n return Elem5lib(self.html_obj.find(self._convert_xpath(xpath)))", "response": "Finds an element by xpath."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef find_elements_by_xpath(self, xpath):\n from sdklib.html.elem import Elem5lib\n\n return [Elem5lib(e) for e in self.html_obj.findall(self._convert_xpath(xpath))]", "response": "Finds multiple elements by xpath."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking that the HTTP response body is the same as the JSON.", "response": "def http_response_body_should_be_this_json(context):\n \"\"\"\n Parameters:\n\n .. code-block:: json\n\n {\n \"param1\": \"value1\",\n \"param2\": \"value2\",\n \"param3\": {\n \"param31\": \"value31\"\n }\n }\n \"\"\"\n body_params = json.loads(context.text)\n assert body_params == context.api_response.data, \\\n \"Expected: {}; Message: {}\".format(body_params, context.api_response.data)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef queue(self, queue, message, params={}, uids=[]):\n\n d = {\n 'id': uuid.uuid1().get_hex(),\n 'version': 1,\n 'message': message,\n 'params': params\n }\n\n ser = json.dumps(d)\n \n if uids:\n for uid in uids:\n path = '/dq/%s/%s' % (uid, queue)\n yield self._put_queue(path, ser)\n\n else:\n path = '/q/%s' % queue\n yield self._put_queue(path, ser)\n\n defer.returnValue(d['id'])", "response": "Queue a job in Rhumba\n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef clusterStatus(self):\n servers = yield self.getClusterServers()\n\n d = {\n 'workers': {},\n 'crons': {},\n 'queues': {}\n }\n\n now = time.time()\n\n reverse_map = {}\n\n for sname in servers:\n last = yield self._get_key('/server/%s/heartbeat' % sname)\n status = yield self._get_key('/server/%s/status' % sname)\n uuid = yield self._get_key('/server/%s/uuid' % sname)\n\n reverse_map[uuid] = sname\n\n if not last:\n last = 0\n\n last = float(last)\n\n if (status == 'ready') and (now - last > 5):\n status = 'offline'\n\n if not sname in d['workers']:\n d['workers'][sname] = []\n\n d['workers'][sname].append({\n 'lastseen': last,\n 'status': status,\n 'id': uuid\n })\n\n # Crons\n crons = yield self.keys('/crons')\n\n for queue in crons:\n if queue not in d['crons']:\n d['crons'][queue] = {'methods': {}}\n\n methods = yield self.keys('/crons/%s' % queue)\n\n for method in methods:\n last = yield self._get_key('/crons/%s/%s' % (queue, method))\n if last:\n d['crons'][queue]['methods'][method] = float(last)\n \n uid = yield self._get_key('/croner/%s' % queue)\n if uid:\n d['crons'][queue]['master'] = '%s:%s' % (uid, reverse_map[uid])\n\n # Queues\n queue_keys = yield self.keys('/qstats')\n\n for qname in queue_keys:\n if qname not in d['queues']:\n qlen = yield self.queueSize(qname)\n\n stats = yield self.getQueueMessageStats(qname)\n\n d['queues'][qname] = {\n 'waiting': qlen,\n 'messages': stats\n }\n\n defer.returnValue(d)", "response": "Returns a dict of cluster nodes and their status information"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_head_form_html(req: \"Request\", forms: List[Form]) -> str:\n # https://docs.pylonsproject.org/projects/deform/en/latest/widget.html#widget-requirements\n js_resources = [] # type: List[str]\n css_resources = [] # type: List[str]\n for form in forms:\n resources = form.get_widget_resources() # type: Dict[str, List[str]]\n # Add, ignoring duplicates:\n js_resources.extend(x for x in resources['js']\n if x not in js_resources)\n css_resources.extend(x for x in resources['css']\n if x not in css_resources)\n js_links = [req.static_url(r) for r in js_resources]\n css_links = [req.static_url(r) for r in css_resources]\n js_tags = ['<script type=\"text/javascript\" src=\"%s\"></script>' % link\n for link in js_links]\n css_tags = ['<link rel=\"stylesheet\" href=\"%s\"/>' % link\n for link in css_links]\n tags = js_tags + css_tags\n head_html = \"\\n\".join(tags)\n return head_html", "response": "Returns the extra HTML needed to render a Deform form."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nusing as a wrapper around a validator, e.g. .. code-block:: python self.validator = debug_validator(OneOf([\"some\", \"values\"])) If you do this, the log will show the thinking of the validator (what it's trying to validate, and whether it accepted or rejected the value).", "response": "def debug_validator(validator: ValidatorType) -> ValidatorType:\n \"\"\"\n Use as a wrapper around a validator, e.g.\n\n .. code-block:: python\n\n self.validator = debug_validator(OneOf([\"some\", \"values\"]))\n\n If you do this, the log will show the thinking of the validator (what it's\n trying to validate, and whether it accepted or rejected the value).\n \"\"\"\n def _validate(node: SchemaNode, value: Any) -> None:\n log.debug(\"Validating: {!r}\", value)\n try:\n validator(node, value)\n log.debug(\"... accepted\")\n except Invalid:\n log.debug(\"... rejected\")\n raise\n\n return _validate"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef gen_fields(field: Field) -> Generator[Field, None, None]:\n yield field\n for c in field.children:\n for f in gen_fields(c):\n yield f", "response": "Yields all the fields of a Deform."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef validate(self,\n controls: Iterable[Tuple[str, str]],\n subcontrol: str = None) -> Any:\n \"\"\"\n Validates the form.\n\n Args:\n controls: an iterable of ``(key, value)`` tuples\n subcontrol:\n\n Returns:\n a Colander ``appstruct``\n\n Raises:\n ValidationFailure: on failure\n \"\"\"\n try:\n return super().validate(controls, subcontrol)\n except ValidationFailure as e:\n if DEBUG_FORM_VALIDATION:\n log.warning(\"Validation failure: {!r}; {}\",\n e, self._get_form_errors())\n self._show_hidden_widgets_for_fields_with_errors(self)\n raise", "response": "Validates the form.\n\n Args:\n controls: an iterable of ``(key, value)`` tuples\n subcontrol:\n\n Returns:\n a Colander ``appstruct``\n\n Raises:\n ValidationFailure: on failure"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_pstats_print2list(fnames, filter_fnames=None, exclude_fnames=None,\n sort=None, sort_reverse=None, limit=None):\n \"\"\"Print stats with a filter or exclude filenames, sort index and limit.\n :param list fnames: cProfile standard files to process.\n :param list filter_fnames: Relative paths to filter and show them.\n :param list exclude_fnames: Relative paths to avoid show them.\n :param str sort: Standard `pstats` key of value to sort the result.\n \\n\\t\\t\\t'calls' (call count)\n \\n\\t\\t\\t'cumulative' (cumulative time)\n \\n\\t\\t\\t'cumtime' (cumulative time)\n \\n\\t\\t\\t'file' (file name)\n \\n\\t\\t\\t'filename' (file name)\n \\n\\t\\t\\t'module' (file name)\n \\n\\t\\t\\t'ncalls' (call count)\n \\n\\t\\t\\t'pcalls' (primitive call count)\n \\n\\t\\t\\t'line' (line number)\n \\n\\t\\t\\t'name' (function name)\n \\n\\t\\t\\t'nfl' (name/file/line)\n \\n\\t\\t\\t'stdname' (standard name)\n \\n\\t\\t\\t'time' (internal time)\n \\n\\t\\t\\t'tottime' (internal time)\n :param bool sort_reverse: Reverse sort order.\n :param int limit: Limit max result.\n :returns: List of dicts with `pstats` print result after filters, sorted\n and limited.\n \"\"\"\n\n if isinstance(fnames, basestring):\n fnames = [fnames]\n fnames_expanded = [\n os.path.expandvars(os.path.expanduser(fname)) for fname in fnames]\n stream = StringIO()\n try:\n stats = pstats.Stats(fnames[0], stream=stream)\n for fname in fnames_expanded[1:]:\n stats.add(fname)\n except TypeError:\n print(\"No cProfile stats valid.\")\n return False\n except EOFError:\n print(\"Empty file cProfile stats valid.\")\n return False\n except IOError:\n print(\"Error to open file.\")\n return False\n\n stats.print_stats()\n stream.seek(0)\n field_list = get_field_list()\n line_stats_re = re.compile(\n r'(?P<%s>\\d+/?\\d+|\\d+)\\s+(?P<%s>\\d+\\.?\\d+)\\s+(?P<%s>\\d+\\.?\\d+)\\s+'\n r'(?P<%s>\\d+\\.?\\d+)\\s+(?P<%s>\\d+\\.?\\d+)\\s+(?P<%s>.*):(?P<%s>\\d+)'\n r'\\((?P<%s>.*)\\)' % tuple(field_list))\n stats_list = []\n count = 0\n for line in stream:\n line = line.strip('\\r\\n ')\n line_stats_match = line_stats_re.match(line) if line else None\n fname = line_stats_match.group('file') if line_stats_match else None\n if fname and is_fname_match(fname, filter_fnames) and \\\n not is_exclude(fname, exclude_fnames):\n data = dict([(field, line_stats_match.group(field))\n for field in field_list])\n data['rcalls'], data['calls'] = (\n data.get('ncalls', '') + '/' + data.get('ncalls', '')\n ).split('/')[:2]\n data['factor'] = \"%.2f\" % (\n (float(data['rcalls']) - float(data['calls']) + 1) *\n float(data['cumtime']))\n data['cumulative'] = data['cumtime']\n stats_list.append(data)\n count += 1\n return sorted(stats_list, key=lambda key: float(key[sort or 'factor']),\n reverse=not sort_reverse)[:limit]", "response": "Return a list of dicts with pstats print result after filtering or exclude filenames sort index and limit."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef print_pstats_list(pstats, pformat=None):\n if not pstats:\n return False\n if pformat is None:\n pformat = (\"{method:<40s} {factor:>16s} {cumtime:>10s} \"\n \"{calls:>10s} {rcalls:>10s} {tottime:>10s} \"\n \"{tt_percall:>10s} {ct_percall:>10s} \"\n \"<{file}:{lineno}\")\n for pstat_line in [dict(zip(pstats[0].keys(), pstats[0].keys()))] + pstats:\n print(pformat.format(**pstat_line))\n return True", "response": "Print a list of pstats dicts formatted by the given format"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef merge_metrics(self):\n\n self.metrics.extend(self._metrics.values())\n del self._metrics", "response": "Merge metrics in the internal _metrics dict to the internal _metrics list and delete the internal _metrics dict"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd an attribute to the internal list of the attribute list.", "response": "def add_attribute(self, name, value):\n \"\"\"\n :param name: Name of the attribute\n :type name: string\n :param value: Value of the attribute\n :type value: string\n \"\"\"\n\n self.attributes.append(Attribute(name, value))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef add_tag(self, name, value):\n\n self.tags.append(Tag(name, value))", "response": "Adds a tag to the list of tags."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef add_sample(self,\n metricId,\n timestamp,\n value,\n metricType=None,\n host=None,\n sparseDataStrategy='None',\n unit='',\n tags=None,\n min=None,\n max=None,\n avg=None,\n sum=None,\n cnt=None,\n ts_is_ms=False):\n \"\"\"\n :param metricId: Metric FQN\n :type metricId: string\n :param timestamp: Timestamp for the sample\n :type timestamp: int\n :param value: Value of the sample\n :type value: float\n :param metricType: Metric Type\n :type metricType: string\n :param host: Element FQN\n :type host: string\n :param sparseDataStrategy: Sparse data strategy\n :type sparseDataStrategy: string\n :param unit: Metric Unit type\n :type unit: string\n :param tags: List of dicts\n :type tags: list\n :param min: Minimum of the sample\n :type min: float\n :param max: Maximum of the sample\n :type max: float\n :param avg: Average of the sample\n :type avg: float\n :param sum: Sum of the sample\n :type sum: float\n :param cnt: Count of the sample\n :type cnt: float\n :param ts_is_ms: Is the timestamp in milliseconds\n :type ts_is_ms: bool\n\n\n\n \"\"\"\n\n if self.id is None and host is not None:\n self.id = host\n\n if self.name is None and host is not None:\n self.name = host\n\n if tags is not None:\n Tags = []\n for i in tags:\n for k in i:\n Tags.append(Tag(k, i[k]))\n\n else:\n Tags = None\n\n metricIdSan = self._sanitize(metricId)\n\n if not hasattr(self, \"_metrics\"):\n setattr(self, \"_metrics\", {})\n\n if self._metrics.get(metricIdSan) is None:\n self._metrics[metricIdSan] = Metric(metricIdSan,\n metricType,\n sparseDataStrategy,\n unit,\n Tags)\n\n if timestamp is None:\n ts = to_ms_timestamp_int(datetime.datetime.utcnow())\n\n else:\n if ts_is_ms:\n ts = int(timestamp)\n else:\n ts = int(timestamp * 1000)\n\n self.samples.append(Sample(metricIdSan,\n ts,\n value,\n min,\n max,\n avg,\n sum,\n cnt))", "response": "Adds a sample to the internal data structure."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a generator that yields a pair of each element with its neighbors.", "response": "def pairwise(iterable):\n \"\"\"Pair each element with its neighbors.\n\n Arguments\n ---------\n iterable : iterable\n\n Returns\n -------\n The generator produces a tuple containing a pairing of each element with\n its neighbor.\n \"\"\"\n iterable = iter(iterable)\n left = next(iterable)\n for right in iterable:\n yield left, right\n left = right"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\npartition an iterable into two lists where the elements on which the predicate indicated True are partitioned.", "response": "def partition(pred, iterable):\n \"\"\"Partition an iterable.\n\n Arguments\n ---------\n pred : function\n A function that takes an element of the iterable and returns\n a boolen indicating to which partition it belongs\n iterable : iterable\n\n Returns\n -------\n A two-tuple of lists with the first list containing the elements on which\n the predicate indicated False and the second list containing the elements\n on which the predicate indicated True.\n\n Note that, unlike the recipe which returns generators, this version\n returns lists.\n \"\"\"\n pos, neg = [], []\n pos_append, neg_append = pos.append, neg.append\n for elem in iterable:\n if pred(elem):\n pos_append(elem)\n else:\n neg_append(elem)\n return neg, pos"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the powerset of the given iterable.", "response": "def powerset(iterable, *, reverse=False):\n \"\"\"Return the powerset.\n\n Arguments\n ---------\n iterable : iterable\n reverse : boolean\n Indicates whether the powerset should be returned descending by\n size\n\n Returns\n -------\n A generator producing each element of the powerset.\n \"\"\"\n lst = list(iterable)\n if reverse:\n rng = range(len(lst), -1, -1)\n else:\n rng = range(len(lst) + 1)\n return chain.from_iterable(combinations(lst, r) for r in rng)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef multi_map(key, iterable, *, default_dict=False):\n result = collections.defaultdict(list)\n for rec in iterable:\n result[key(rec)].append(rec)\n return result if default_dict else dict(result)", "response": "Collect data into a multi - map."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsplit the iterable into a list of items.", "response": "def split(pred, iterable, *, trailing=True):\n \"\"\"Split the iterable.\n\n Arguments\n ----------\n pred : function\n A function that accepts an element retrieved from the iterable\n and returns a boolean indicating if it is the element on which\n to split\n iterable : iterable\n trailing : boolean\n Indicates whether the split should occur on the leading edge\n of the match on the split or on the trailing edge of the match\n on the split\n\n Returns\n -------\n The generator produces a list for each split.\n\n If *trailing* is True then the element that was identified by the predicate\n will be returned at the end of each split.\n If *trailing* is False then the element that was identified by the\n predicate will be returned at the beginning of the following split.\n\n No guarantee is made regarding the state of the iterable during operation.\n \"\"\"\n result = []\n result_append = result.append\n if trailing:\n for elem in iterable:\n result_append(elem)\n if pred(elem):\n yield result\n result = []\n result_append = result.append\n else:\n for elem in iterable:\n if pred(elem):\n if result:\n yield result\n result = []\n result_append = result.append\n result_append(elem)\n if result:\n yield result"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef chunk(iterable, length):\n if length < 0:\n return ()\n iterable = iter(iterable)\n result = tuple(islice(iterable, length))\n while result:\n yield result\n result = tuple(islice(iterable, length))", "response": "Returns a generator that yields the specified number of elements from the iterable in chunks of specified length."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndivide a count by n.", "response": "def divide_sizes(count, n): # pylint: disable=invalid-name\n \"\"\"Evenly divide a count.\n\n Arguments\n ---------\n count : integer\n The number to be evenly divided\n n : integer\n The number of buckets in which to divide the number\n\n Returns\n -------\n A list of integers indicating what size each bucket should be for an even\n distribution of *count*.\n\n The number of integers returned is always *n* and, thus, may be 0.\n\n Useful for calculating slices for generators that might be too large to\n convert into a list as happens in divide().\n \"\"\"\n if n <= 0:\n return []\n if count < 0:\n return [0] * n\n base, rem = divmod(count, n)\n return [base + 1 if i < rem else base for i in range(n)]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndisplay the tree represented by the cursor", "response": "def dump(cursor):\n \"\"\" Display the AST represented by the cursor\n \"\"\"\n\n def node_children(node):\n return list(node.get_children())\n\n def print_node(node):\n text = node.spelling or node.displayname\n kind = str(node.kind).split('.')[1]\n return '{} {}'.format(kind, text)\n\n return draw_tree(cursor, node_children, print_node)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert something that may be coercible to a pendulum. DateTime or None.", "response": "def coerce_to_pendulum(x: PotentialDatetimeType,\n assume_local: bool = False) -> Optional[DateTime]:\n \"\"\"\n Converts something to a :class:`pendulum.DateTime`.\n\n Args:\n x: something that may be coercible to a datetime\n assume_local: if ``True``, assume local timezone; if ``False``, assume\n UTC\n\n Returns:\n a :class:`pendulum.DateTime`, or ``None``.\n\n Raises:\n pendulum.parsing.exceptions.ParserError: if a string fails to parse\n ValueError: if no conversion possible\n \"\"\"\n if not x: # None and blank string\n return None\n if isinstance(x, DateTime):\n return x\n tz = get_tz_local() if assume_local else get_tz_utc()\n if isinstance(x, datetime.datetime):\n return pendulum.instance(x, tz=tz) # (*)\n elif isinstance(x, datetime.date):\n # BEWARE: datetime subclasses date. The order is crucial here.\n # Can also use: type(x) is datetime.date\n # noinspection PyUnresolvedReferences\n midnight = DateTime.min.time()\n dt = DateTime.combine(x, midnight)\n return pendulum.instance(dt, tz=tz) # (*)\n elif isinstance(x, str):\n return pendulum.parse(x, tz=tz) # (*) # may raise\n else:\n raise ValueError(\"Don't know how to convert to DateTime: \"\n \"{!r}\".format(x))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef coerce_to_pendulum_date(x: PotentialDatetimeType,\n assume_local: bool = False) -> Optional[Date]:\n \"\"\"\n Converts something to a :class:`pendulum.Date`.\n\n Args:\n x: something that may be coercible to a date\n assume_local: if ``True``, assume local timezone; if ``False``, assume\n UTC\n\n Returns:\n a :class:`pendulum.Date`, or ``None``.\n\n Raises:\n pendulum.parsing.exceptions.ParserError: if a string fails to parse\n ValueError: if no conversion possible\n \"\"\"\n p = coerce_to_pendulum(x, assume_local=assume_local)\n return None if p is None else p.date()", "response": "Converts something to a pendulum. Date."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef pendulum_to_datetime(x: DateTime) -> datetime.datetime:\n return datetime.datetime(\n x.year, x.month, x.day,\n x.hour, x.minute, x.second, x.microsecond,\n tzinfo=x.tzinfo\n )", "response": "Converts a pendulum. datetime. DateTime object to a datetime object."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting a Pendulum datetime to a UTC datetime.", "response": "def pendulum_to_utc_datetime_without_tz(x: DateTime) -> datetime.datetime:\n \"\"\"\n Converts a Pendulum ``DateTime`` (which will have timezone information) to\n a ``datetime.datetime`` that (a) has no timezone information, and (b) is\n in UTC.\n\n Example:\n\n .. code-block:: python\n\n import pendulum\n from cardinal_pythonlib.datetimefunc import *\n in_moscow = pendulum.parse(\"2018-01-01T09:00+0300\") # 9am in Moscow\n in_london = pendulum.UTC.convert(in_moscow) # 6am in UTC\n dt_utc_from_moscow = pendulum_to_utc_datetime_without_tz(in_moscow) # 6am, no timezone info\n dt_utc_from_london = pendulum_to_utc_datetime_without_tz(in_london) # 6am, no timezone info\n\n \"\"\" # noqa\n pendulum_in_utc = pendulum.UTC.convert(x)\n return pendulum_to_datetime_stripping_tz(pendulum_in_utc)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef pendulum_date_to_datetime_date(x: Date) -> datetime.date:\n return datetime.date(year=x.year, month=x.month, day=x.day)", "response": "Takes a pendulum. Date and returns a datetime. date."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ntake a pendulum. Time and returns a datetime. time.", "response": "def pendulum_time_to_datetime_time(x: Time) -> datetime.time:\n \"\"\"\n Takes a :class:`pendulum.Time` and returns a :class:`datetime.time`.\n Used, for example, where a database backend insists on\n :class:`datetime.time`.\n \"\"\"\n return datetime.time(\n hour=x.hour, minute=x.minute, second=x.second,\n microsecond=x.microsecond,\n tzinfo=x.tzinfo\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nformats a datetime with a strftime format specification string.", "response": "def format_datetime(d: PotentialDatetimeType,\n fmt: str,\n default: str = None) -> Optional[str]:\n \"\"\"\n Format a datetime with a ``strftime`` format specification string, or\n return ``default`` if the input is ``None``.\n \"\"\"\n d = coerce_to_pendulum(d)\n if d is None:\n return default\n return d.strftime(fmt)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef strfdelta(tdelta: Union[datetime.timedelta, int, float, str],\n fmt='{D:02}d {H:02}h {M:02}m {S:02}s',\n inputtype='timedelta'):\n \"\"\"\n Convert a ``datetime.timedelta`` object or a regular number to a custom-\n formatted string, just like the ``strftime()`` method does for\n ``datetime.datetime`` objects.\n\n The ``fmt`` argument allows custom formatting to be specified. Fields can\n include ``seconds``, ``minutes``, ``hours``, ``days``, and ``weeks``. Each\n field is optional.\n\n Some examples:\n\n .. code-block:: none\n\n '{D:02}d {H:02}h {M:02}m {S:02}s' --> '05d 08h 04m 02s' (default)\n '{W}w {D}d {H}:{M:02}:{S:02}' --> '4w 5d 8:04:02'\n '{D:2}d {H:2}:{M:02}:{S:02}' --> ' 5d 8:04:02'\n '{H}h {S}s' --> '72h 800s'\n\n The ``inputtype`` argument allows ``tdelta`` to be a regular number,\n instead of the default behaviour of treating it as a ``datetime.timedelta``\n object. Valid ``inputtype`` strings:\n\n .. code-block:: none\n\n 'timedelta', # treats input as a datetime.timedelta\n 's', 'seconds',\n 'm', 'minutes',\n 'h', 'hours',\n 'd', 'days',\n 'w', 'weeks'\n\n Modified from\n https://stackoverflow.com/questions/538666/python-format-timedelta-to-string\n \"\"\" # noqa\n\n # Convert tdelta to integer seconds.\n if inputtype == 'timedelta':\n remainder = int(tdelta.total_seconds())\n elif inputtype in ['s', 'seconds']:\n remainder = int(tdelta)\n elif inputtype in ['m', 'minutes']:\n remainder = int(tdelta) * 60\n elif inputtype in ['h', 'hours']:\n remainder = int(tdelta) * 3600\n elif inputtype in ['d', 'days']:\n remainder = int(tdelta) * 86400\n elif inputtype in ['w', 'weeks']:\n remainder = int(tdelta) * 604800\n else:\n raise ValueError(\"Bad inputtype: {}\".format(inputtype))\n\n f = Formatter()\n desired_fields = [field_tuple[1] for field_tuple in f.parse(fmt)]\n possible_fields = ('W', 'D', 'H', 'M', 'S')\n constants = {'W': 604800, 'D': 86400, 'H': 3600, 'M': 60, 'S': 1}\n values = {}\n for field in possible_fields:\n if field in desired_fields and field in constants:\n values[field], remainder = divmod(remainder, constants[field])\n return f.format(fmt, **values)", "response": "Returns a string representation of a datetime. timedelta object or a regular number."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef convert_datetime_to_utc(dt: PotentialDatetimeType) -> DateTime:\n dt = coerce_to_pendulum(dt)\n tz = get_tz_utc()\n return dt.in_tz(tz)", "response": "Convert date or time to UTC."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting date or time with timezone to local timezone.", "response": "def convert_datetime_to_local(dt: PotentialDatetimeType) -> DateTime:\n \"\"\"\n Convert date/time with timezone to local timezone.\n \"\"\"\n dt = coerce_to_pendulum(dt)\n tz = get_tz_local()\n return dt.in_tz(tz)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_duration_h_m(start: Union[str, DateTime],\n end: Union[str, DateTime],\n default: str = \"N/A\") -> str:\n \"\"\"\n Calculate the time between two dates/times expressed as strings.\n\n Args:\n start: start date/time\n end: end date/time\n default: string value to return in case either of the inputs is\n ``None``\n\n Returns:\n a string that is one of\n\n .. code-block:\n\n 'hh:mm'\n '-hh:mm'\n default\n\n \"\"\"\n start = coerce_to_pendulum(start)\n end = coerce_to_pendulum(end)\n if start is None or end is None:\n return default\n duration = end - start\n minutes = duration.in_minutes()\n (hours, minutes) = divmod(minutes, 60)\n if hours < 0:\n # negative... trickier\n # Python's divmod does interesting things with negative numbers:\n # Hours will be negative, and minutes always positive\n hours += 1\n minutes = 60 - minutes\n return \"-{}:{}\".format(hours, \"00\" if minutes == 0 else minutes)\n else:\n return \"{}:{}\".format(hours, \"00\" if minutes == 0 else minutes)", "response": "Returns the time between two dates or times expressed as strings."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the age of the node in whole years.", "response": "def get_age(dob: PotentialDatetimeType,\n when: PotentialDatetimeType,\n default: str = \"\") -> Union[int, str]:\n \"\"\"\n Age (in whole years) at a particular date, or ``default``.\n\n Args:\n dob: date of birth\n when: date/time at which to calculate age\n default: value to return if either input is ``None``\n\n Returns:\n age in whole years (rounded down), or ``default``\n\n \"\"\"\n dob = coerce_to_pendulum_date(dob)\n when = coerce_to_pendulum_date(when)\n if dob is None or when is None:\n return default\n return (when - dob).years"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef truncate_date_to_first_of_month(\n dt: Optional[DateLikeType]) -> Optional[DateLikeType]:\n \"\"\"\n Change the day to the first of the month.\n \"\"\"\n if dt is None:\n return None\n return dt.replace(day=1)", "response": "Truncate a date to the first of the month."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_now_utc_notz_datetime() -> datetime.datetime:\n now = datetime.datetime.utcnow()\n return now.replace(tzinfo=None)", "response": "Get the UTC time now but without timezone information."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncoerces an object to a datetime. datetime.", "response": "def coerce_to_datetime(x: Any) -> Optional[datetime.datetime]:\n \"\"\"\n Ensure an object is a :class:`datetime.datetime`, or coerce to one, or\n raise :exc:`ValueError` or :exc:`OverflowError` (as per\n http://dateutil.readthedocs.org/en/latest/parser.html).\n \"\"\"\n if x is None:\n return None\n elif isinstance(x, DateTime):\n return pendulum_to_datetime(x)\n elif isinstance(x, datetime.datetime):\n return x\n elif isinstance(x, datetime.date):\n return datetime.datetime(x.year, x.month, x.day)\n else:\n return dateutil.parser.parse(x)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndo http requests from context.", "response": "def request_from_context(context):\n \"\"\"\n Do http requests from context.\n\n :param context: request context.\n \"\"\"\n new_context = copy.deepcopy(context)\n assert new_context.method in ALLOWED_METHODS\n\n new_context.url_path = generate_url_path(\n new_context.url_path,\n prefix=new_context.prefix_url_path,\n format_suffix=new_context.url_path_format,\n **new_context.url_path_params\n )\n\n if new_context.body_params or new_context.files:\n body, content_type = new_context.renderer.encode_params(new_context.body_params, files=new_context.files)\n if new_context.update_content_type and HttpSdk.CONTENT_TYPE_HEADER_NAME not in new_context.headers:\n new_context.headers[HttpSdk.CONTENT_TYPE_HEADER_NAME] = content_type\n else:\n body = None\n\n authentication_instances = new_context.authentication_instances\n for auth_obj in authentication_instances:\n new_context = auth_obj.apply_authentication(new_context)\n\n if HttpSdk.COOKIE_HEADER_NAME not in new_context.headers and not new_context.cookie.is_empty():\n new_context.headers[HttpSdk.COOKIE_HEADER_NAME] = new_context.cookie.as_cookie_header_value()\n\n url = \"%s%s\" % (new_context.host, new_context.url_path)\n if new_context.query_params:\n url += \"?%s\" % (urlencode(new_context.query_params))\n\n log_print_request(new_context.method, url, new_context.query_params, new_context.headers, body)\n # ensure method and url are native str\n r = HttpSdk.get_pool_manager(new_context.proxy).request(\n convert_unicode_to_native_str(new_context.method),\n convert_unicode_to_native_str(url),\n body=body,\n headers=HttpSdk.convert_headers_to_native_str(new_context.headers),\n redirect=new_context.redirect,\n timeout=new_context.timeout\n )\n log_print_response(r.status, r.data, r.headers)\n r = new_context.response_class(r)\n return r"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef clear(self, *args):\n for field in self.fields_to_clear + list(args):\n setattr(self, field, None)", "response": "Clear the default values of the related objects."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef host(self, value):\n scheme, host, port = get_hostname_parameters_from_url(value)\n self._host = \"%s://%s:%s\" % (scheme, host, port)", "response": "Sets the host for the cache entry."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsetting cookie. :param value:", "response": "def cookie(self, value):\n \"\"\"\n Set cookie.\n :param value:\n \"\"\"\n if value and not value.is_empty():\n self._cookie = value\n else:\n self._cookie = Cookie()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef set_default_host(cls, value):\n if value is None:\n cls.DEFAULT_HOST = \"http://127.0.0.1:80\"\n else:\n scheme, host, port = get_hostname_parameters_from_url(value)\n cls.DEFAULT_HOST = \"%s://%s:%s\" % (scheme, host, port)", "response": "Sets the default host for the class."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef set_default_proxy(cls, value):\n if value is None:\n cls.DEFAULT_PROXY = None\n else:\n scheme, host, port = get_hostname_parameters_from_url(value)\n cls.DEFAULT_PROXY = \"%s://%s:%s\" % (scheme, host, port)", "response": "Sets the default proxy for the current object."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef login(self, **kwargs):\n assert self.LOGIN_URL_PATH is not None\n\n render_name = kwargs.pop(\"render\", \"json\")\n render = get_renderer(render_name)\n params = parse_args(**kwargs)\n return self.post(self.LOGIN_URL_PATH, body_params=params, render=render)", "response": "Login abstract method with default implementation."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncalculates a check digit for the given nine digits.", "response": "def nhs_check_digit(ninedigits: Union[str, List[Union[str, int]]]) -> int:\n \"\"\"\n Calculates an NHS number check digit.\n\n Args:\n ninedigits: string or list\n\n Returns:\n check digit\n\n Method:\n\n 1. Multiply each of the first nine digits by the corresponding\n digit weighting (see :const:`NHS_DIGIT_WEIGHTINGS`).\n 2. Sum the results.\n 3. Take remainder after division by 11.\n 4. Subtract the remainder from 11\n 5. If this is 11, use 0 instead\n If it's 10, the number is invalid\n If it doesn't match the actual check digit, the number is invalid\n\n \"\"\"\n if len(ninedigits) != 9 or not all(str(x).isdigit() for x in ninedigits):\n raise ValueError(\"bad string to nhs_check_digit\")\n check_digit = 11 - (sum([\n int(d) * f\n for (d, f) in zip(ninedigits, NHS_DIGIT_WEIGHTINGS)\n ]) % 11)\n # ... % 11 yields something in the range 0-10\n # ... 11 - that yields something in the range 1-11\n if check_digit == 11:\n check_digit = 0\n return check_digit"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef is_valid_nhs_number(n: int) -> bool:\n if not isinstance(n, int):\n log.debug(\"is_valid_nhs_number: parameter was not of integer type\")\n return False\n\n s = str(n)\n # Not 10 digits long?\n if len(s) != 10:\n log.debug(\"is_valid_nhs_number: not 10 digits\")\n return False\n\n main_digits = [int(s[i]) for i in range(9)]\n actual_check_digit = int(s[9]) # tenth digit\n expected_check_digit = nhs_check_digit(main_digits)\n if expected_check_digit == 10:\n log.debug(\"is_valid_nhs_number: calculated check digit invalid\")\n return False\n if expected_check_digit != actual_check_digit:\n log.debug(\"is_valid_nhs_number: check digit mismatch\")\n return False\n # Hooray!\n return True", "response": "Validates an integer as an NHS number."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngenerates a random valid NHS number as an integer.", "response": "def generate_random_nhs_number() -> int:\n \"\"\"\n Returns a random valid NHS number, as an ``int``.\n \"\"\"\n check_digit = 10 # NHS numbers with this check digit are all invalid\n while check_digit == 10:\n digits = [random.randint(1, 9)] # don't start with a zero\n digits.extend([random.randint(0, 9) for _ in range(8)])\n # ... length now 9\n check_digit = nhs_check_digit(digits)\n # noinspection PyUnboundLocalVariable\n digits.append(check_digit)\n return int(\"\".join([str(d) for d in digits]))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef generate_nhs_number_from_first_9_digits(first9digits: str) -> Optional[int]:\n if len(first9digits) != 9:\n log.warning(\"Not 9 digits\")\n return None\n try:\n first9int = int(first9digits)\n except (TypeError, ValueError):\n log.warning(\"Not an integer\")\n return None # not an int\n if len(str(first9int)) != len(first9digits):\n # e.g. leading zeros, or some such\n log.warning(\"Leading zeros?\")\n return None\n check_digit = nhs_check_digit(first9digits)\n if check_digit == 10: # NHS numbers with this check digit are all invalid\n log.warning(\"Can't have check digit of 10\")\n return None\n return int(first9digits + str(check_digit))", "response": "Generates a valid NHS number given the first 9 digits."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef nhs_number_from_text_or_none(s: str) -> Optional[int]:\n # None in, None out.\n funcname = \"nhs_number_from_text_or_none: \"\n if not s:\n log.debug(funcname + \"incoming parameter was None\")\n return None\n\n # (a) If it's not a 10-digit number, bye-bye.\n\n # Remove whitespace\n s = WHITESPACE_REGEX.sub(\"\", s) # replaces all instances\n # Contains non-numeric characters?\n if NON_NUMERIC_REGEX.search(s):\n log.debug(funcname + \"contains non-numeric characters\")\n return None\n # Not 10 digits long?\n if len(s) != 10:\n log.debug(funcname + \"not 10 digits long\")\n return None\n\n # (b) Validation\n n = int(s)\n if not is_valid_nhs_number(n):\n log.debug(funcname + \"failed validation\")\n return None\n\n # Happy!\n return n", "response": "Returns a validated NHS number from a string or None if the string is not a 10 - digit number."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef make_email(from_addr: str,\n date: str = None,\n sender: str = \"\",\n reply_to: Union[str, List[str]] = \"\",\n to: Union[str, List[str]] = \"\",\n cc: Union[str, List[str]] = \"\",\n bcc: Union[str, List[str]] = \"\",\n subject: str = \"\",\n body: str = \"\",\n content_type: str = CONTENT_TYPE_TEXT,\n charset: str = \"utf8\",\n attachment_filenames: Sequence[str] = None,\n attachment_binaries: Sequence[bytes] = None,\n attachment_binary_filenames: Sequence[str] = None,\n verbose: bool = False) -> email.mime.multipart.MIMEMultipart:\n \"\"\"\n Makes an e-mail message.\n\n Arguments that can be multiple e-mail addresses are (a) a single e-mail\n address as a string, or (b) a list of strings (each a single e-mail\n address), or (c) a comma-separated list of multiple e-mail addresses.\n\n Args:\n from_addr: name of the sender for the \"From:\" field\n date: e-mail date in RFC 2822 format, or ``None`` for \"now\"\n sender: name of the sender for the \"Sender:\" field\n reply_to: name of the sender for the \"Reply-To:\" field\n\n to: e-mail address(es) of the recipients for \"To:\" field\n cc: e-mail address(es) of the recipients for \"Cc:\" field\n bcc: e-mail address(es) of the recipients for \"Bcc:\" field\n\n subject: e-mail subject\n body: e-mail body\n content_type: MIME type for body content, default ``text/plain``\n charset: character set for body; default ``utf8``\n\n attachment_filenames: filenames of attachments to add\n attachment_binaries: binary objects to add as attachments\n attachment_binary_filenames: filenames corresponding to\n ``attachment_binaries``\n verbose: be verbose?\n\n Returns:\n a :class:`email.mime.multipart.MIMEMultipart`\n\n Raises:\n :exc:`AssertionError`, :exc:`ValueError`\n\n \"\"\"\n def _csv_list_to_list(x: str) -> List[str]:\n stripped = [item.strip() for item in x.split(COMMA)]\n return [item for item in stripped if item]\n\n def _assert_nocomma(x: Union[str, List[str]]) -> None:\n if isinstance(x, str):\n x = [x]\n for _addr in x:\n assert COMMA not in _addr, (\n \"Commas not allowed in e-mail addresses: {!r}\".format(_addr)\n )\n\n # -------------------------------------------------------------------------\n # Arguments\n # -------------------------------------------------------------------------\n if not date:\n date = email.utils.formatdate(localtime=True)\n assert isinstance(from_addr, str), (\n \"'From:' can only be a single address \"\n \"(for Python sendmail, not RFC 2822); was {!r}\".format(from_addr)\n )\n _assert_nocomma(from_addr)\n assert isinstance(sender, str), (\n \"'Sender:' can only be a single address; was {!r}\".format(sender)\n )\n _assert_nocomma(sender)\n if isinstance(reply_to, str):\n reply_to = [reply_to] if reply_to else [] # type: List[str]\n _assert_nocomma(reply_to)\n if isinstance(to, str):\n to = _csv_list_to_list(to)\n if isinstance(cc, str):\n cc = _csv_list_to_list(cc)\n if isinstance(bcc, str):\n bcc = _csv_list_to_list(bcc)\n assert to or cc or bcc, \"No recipients (must have some of: To, Cc, Bcc)\"\n _assert_nocomma(to)\n _assert_nocomma(cc)\n _assert_nocomma(bcc)\n attachment_filenames = attachment_filenames or [] # type: List[str]\n assert all(attachment_filenames), (\n \"Missing attachment filenames: {!r}\".format(attachment_filenames)\n )\n attachment_binaries = attachment_binaries or [] # type: List[bytes]\n attachment_binary_filenames = attachment_binary_filenames or [] # type: List[str] # noqa\n assert len(attachment_binaries) == len(attachment_binary_filenames), (\n \"If you specify attachment_binaries or attachment_binary_filenames, \"\n \"they must be iterables of the same length.\"\n )\n assert all(attachment_binary_filenames), (\n \"Missing filenames for attached binaries: {!r}\".format(\n attachment_binary_filenames)\n )\n\n # -------------------------------------------------------------------------\n # Make message\n # -------------------------------------------------------------------------\n msg = email.mime.multipart.MIMEMultipart()\n\n # Headers: mandatory\n msg[\"From\"] = from_addr\n msg[\"Date\"] = date\n msg[\"Subject\"] = subject\n\n # Headers: optional\n if sender:\n msg[\"Sender\"] = sender # Single only, not a list\n if reply_to:\n msg[\"Reply-To\"] = COMMASPACE.join(reply_to)\n if to:\n msg[\"To\"] = COMMASPACE.join(to)\n if cc:\n msg[\"Cc\"] = COMMASPACE.join(cc)\n if bcc:\n msg[\"Bcc\"] = COMMASPACE.join(bcc)\n\n # Body\n if content_type == CONTENT_TYPE_TEXT:\n msgbody = email.mime.text.MIMEText(body, \"plain\", charset)\n elif content_type == CONTENT_TYPE_HTML:\n msgbody = email.mime.text.MIMEText(body, \"html\", charset)\n else:\n raise ValueError(\"unknown content_type\")\n msg.attach(msgbody)\n\n # Attachments\n # noinspection PyPep8,PyBroadException\n try:\n if attachment_filenames:\n # -----------------------------------------------------------------\n # Attach things by filename\n # -----------------------------------------------------------------\n if verbose:\n log.debug(\"attachment_filenames: {}\", attachment_filenames)\n # noinspection PyTypeChecker\n for f in attachment_filenames:\n part = email.mime.base.MIMEBase(\"application\", \"octet-stream\")\n part.set_payload(open(f, \"rb\").read())\n email.encoders.encode_base64(part)\n part.add_header(\n 'Content-Disposition',\n 'attachment; filename=\"%s\"' % os.path.basename(f)\n )\n msg.attach(part)\n if attachment_binaries:\n # -----------------------------------------------------------------\n # Binary attachments, which have a notional filename\n # -----------------------------------------------------------------\n if verbose:\n log.debug(\"attachment_binary_filenames: {}\",\n attachment_binary_filenames)\n for i in range(len(attachment_binaries)):\n blob = attachment_binaries[i]\n filename = attachment_binary_filenames[i]\n part = email.mime.base.MIMEBase(\"application\", \"octet-stream\")\n part.set_payload(blob)\n email.encoders.encode_base64(part)\n part.add_header(\n 'Content-Disposition',\n 'attachment; filename=\"%s\"' % filename)\n msg.attach(part)\n except Exception as e:\n raise ValueError(\"send_email: Failed to attach files: {}\".format(e))\n\n return msg", "response": "Returns an e - mail message."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsend an e - mail message to the specified addresses.", "response": "def send_msg(from_addr: str,\n to_addrs: Union[str, List[str]],\n host: str,\n user: str,\n password: str,\n port: int = None,\n use_tls: bool = True,\n msg: email.mime.multipart.MIMEMultipart = None,\n msg_string: str = None) -> None:\n \"\"\"\n Sends a pre-built e-mail message.\n\n Args:\n from_addr: e-mail address for 'From:' field\n to_addrs: address or list of addresses to transmit to\n\n host: mail server host\n user: username on mail server\n password: password for username on mail server\n port: port to use, or ``None`` for protocol default\n use_tls: use TLS, rather than plain SMTP?\n\n msg: a :class:`email.mime.multipart.MIMEMultipart`\n msg_string: alternative: specify the message as a raw string\n\n Raises:\n :exc:`RuntimeError`\n\n See also:\n\n - https://tools.ietf.org/html/rfc3207\n\n \"\"\"\n assert bool(msg) != bool(msg_string), \"Specify either msg or msg_string\"\n # Connect\n try:\n session = smtplib.SMTP(host, port)\n except smtplib.SMTPException as e:\n raise RuntimeError(\n \"send_msg: Failed to connect to host {}, port {}: {}\".format(\n host, port, e))\n try:\n session.ehlo()\n except smtplib.SMTPException as e:\n raise RuntimeError(\"send_msg: Failed to issue EHLO: {}\".format(e))\n\n if use_tls:\n try:\n session.starttls()\n session.ehlo()\n except smtplib.SMTPException as e:\n raise RuntimeError(\n \"send_msg: Failed to initiate TLS: {}\".format(e))\n\n # Log in\n if user:\n try:\n session.login(user, password)\n except smtplib.SMTPException as e:\n raise RuntimeError(\n \"send_msg: Failed to login as user {}: {}\".format(user, e))\n else:\n log.debug(\"Not using SMTP AUTH; no user specified\")\n # For systems with... lax... security requirements\n\n # Send\n try:\n session.sendmail(from_addr, to_addrs, msg.as_string())\n except smtplib.SMTPException as e:\n raise RuntimeError(\"send_msg: Failed to send e-mail: {}\".format(e))\n\n # Log out\n session.quit()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef send_email(from_addr: str,\n host: str,\n user: str,\n password: str,\n port: int = None,\n use_tls: bool = True,\n date: str = None,\n sender: str = \"\",\n reply_to: Union[str, List[str]] = \"\",\n to: Union[str, List[str]] = \"\",\n cc: Union[str, List[str]] = \"\",\n bcc: Union[str, List[str]] = \"\",\n subject: str = \"\",\n body: str = \"\",\n content_type: str = CONTENT_TYPE_TEXT,\n charset: str = \"utf8\",\n attachment_filenames: Sequence[str] = None,\n attachment_binaries: Sequence[bytes] = None,\n attachment_binary_filenames: Sequence[str] = None,\n verbose: bool = False) -> Tuple[bool, str]:\n \"\"\"\n Sends an e-mail in text/html format using SMTP via TLS.\n\n Args:\n host: mail server host\n user: username on mail server\n password: password for username on mail server\n port: port to use, or ``None`` for protocol default\n use_tls: use TLS, rather than plain SMTP?\n \n date: e-mail date in RFC 2822 format, or ``None`` for \"now\"\n \n from_addr: name of the sender for the \"From:\" field\n sender: name of the sender for the \"Sender:\" field\n reply_to: name of the sender for the \"Reply-To:\" field\n \n to: e-mail address(es) of the recipients for \"To:\" field\n cc: e-mail address(es) of the recipients for \"Cc:\" field\n bcc: e-mail address(es) of the recipients for \"Bcc:\" field\n \n subject: e-mail subject\n body: e-mail body\n content_type: MIME type for body content, default ``text/plain``\n charset: character set for body; default ``utf8``\n \n attachment_filenames: filenames of attachments to add\n attachment_binaries: binary objects to add as attachments\n attachment_binary_filenames: filenames corresponding to\n ``attachment_binaries``\n verbose: be verbose?\n\n Returns:\n tuple: ``(success, error_or_success_message)``\n\n See\n\n - https://tools.ietf.org/html/rfc2822\n - https://tools.ietf.org/html/rfc5322\n - http://segfault.in/2010/12/sending-gmail-from-python/\n - http://stackoverflow.com/questions/64505\n - http://stackoverflow.com/questions/3362600\n\n Re security:\n\n - TLS supersedes SSL:\n https://en.wikipedia.org/wiki/Transport_Layer_Security\n \n - https://en.wikipedia.org/wiki/Email_encryption\n \n - SMTP connections on ports 25 and 587 are commonly secured via TLS using\n the ``STARTTLS`` command:\n https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol\n \n - https://tools.ietf.org/html/rfc8314\n \n - \"STARTTLS on port 587\" is one common method. Django refers to this as\n \"explicit TLS\" (its ``E_MAIL_USE_TLS`` setting; see\n https://docs.djangoproject.com/en/2.1/ref/settings/#std:setting-EMAIL_USE_TLS).\n \n - Port 465 is also used for \"implicit TLS\" (3.3 in\n https://tools.ietf.org/html/rfc8314). Django refers to this as \"implicit\n TLS\" too, or SSL; see its ``EMAIL_USE_SSL`` setting at\n https://docs.djangoproject.com/en/2.1/ref/settings/#email-use-ssl). We\n don't support that here.\n\n \"\"\" # noqa\n if isinstance(to, str):\n to = [to]\n if isinstance(cc, str):\n cc = [cc]\n if isinstance(bcc, str):\n bcc = [bcc]\n\n # -------------------------------------------------------------------------\n # Make it\n # -------------------------------------------------------------------------\n try:\n msg = make_email(\n from_addr=from_addr,\n date=date,\n sender=sender,\n reply_to=reply_to,\n to=to,\n cc=cc,\n bcc=bcc,\n subject=subject,\n body=body,\n content_type=content_type,\n charset=charset,\n attachment_filenames=attachment_filenames,\n attachment_binaries=attachment_binaries,\n attachment_binary_filenames=attachment_binary_filenames,\n verbose=verbose,\n )\n except (AssertionError, ValueError) as e:\n errmsg = str(e)\n log.error(\"{}\", errmsg)\n return False, errmsg\n\n # -------------------------------------------------------------------------\n # Send it\n # -------------------------------------------------------------------------\n\n to_addrs = to + cc + bcc\n try:\n send_msg(\n msg=msg,\n from_addr=from_addr,\n to_addrs=to_addrs,\n host=host,\n user=user,\n password=password,\n port=port,\n use_tls=use_tls,\n )\n except RuntimeError as e:\n errmsg = str(e)\n log.error(\"{}\", e)\n return False, errmsg\n\n return True, \"Success\"", "response": "Sends an e - mail using the SMTP standard."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef main() -> None:\n logging.basicConfig()\n log.setLevel(logging.DEBUG)\n parser = argparse.ArgumentParser(\n description=\"Send an e-mail from the command line.\")\n parser.add_argument(\"sender\", action=\"store\",\n help=\"Sender's e-mail address\")\n parser.add_argument(\"host\", action=\"store\",\n help=\"SMTP server hostname\")\n parser.add_argument(\"user\", action=\"store\",\n help=\"SMTP username\")\n parser.add_argument(\"password\", action=\"store\",\n help=\"SMTP password\")\n parser.add_argument(\"recipient\", action=\"append\",\n help=\"Recipient e-mail address(es)\")\n parser.add_argument(\"subject\", action=\"store\",\n help=\"Message subject\")\n parser.add_argument(\"body\", action=\"store\",\n help=\"Message body\")\n parser.add_argument(\"--attach\", nargs=\"*\",\n help=\"Filename(s) to attach\")\n parser.add_argument(\"--tls\", action=\"store_false\",\n help=\"Use TLS connection security\")\n parser.add_argument(\"--verbose\", action=\"store_true\",\n help=\"Be verbose\")\n parser.add_argument(\"-h --help\", action=\"help\",\n help=\"Prints this help\")\n args = parser.parse_args()\n (result, msg) = send_email(\n from_addr=args.sender,\n to=args.recipient,\n subject=args.subject,\n body=args.body,\n host=args.host,\n user=args.user,\n password=args.password,\n use_tls=args.tls,\n attachment_filenames=args.attach,\n verbose=args.verbose,\n )\n if result:\n log.info(\"Success\")\n else:\n log.info(\"Failure\")\n # log.error(msg)\n sys.exit(0 if result else 1)", "response": "Command - line processor."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nparsing a string of C ++ code into a sequence of unsaved C ++ code.", "response": "def parse_string(contents, name='tmp.cpp', **kwargs):\n \"\"\" Parse a string of C/C++ code\n \"\"\"\n idx = clang.cindex.Index.create()\n tu = idx.parse(name, unsaved_files=[(name, contents)], **kwargs)\n return _ensure_parse_valid(tu)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nparse a C ++ file and return a tuple of the first element of the tuple.", "response": "def parse(name, **kwargs):\n \"\"\" Parse a C/C++ file\n \"\"\"\n idx = clang.cindex.Index.create()\n assert os.path.exists(name)\n tu = idx.parse(name, **kwargs)\n return _ensure_parse_valid(tu)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef ip_addresses_from_xff(value: str) -> List[str]:\n if not value:\n return []\n return [x.strip() for x in value.split(\",\")]", "response": "Returns a list of IP addresses as strings given the value of an HTTP X - Forwarded - For header or WSGI HTTP_X_FORWARDED - For header."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nam any special handling necessary for thie config?", "response": "def necessary(self) -> bool:\n \"\"\"\n Is any special handling (e.g. the addition of\n :class:`ReverseProxiedMiddleware`) necessary for thie config?\n \"\"\"\n return any([\n self.trusted_proxy_headers,\n self.http_host,\n self.remote_addr,\n self.script_name,\n self.server_name,\n self.server_port,\n self.url_scheme,\n self.rewrite_path_info,\n ])"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef queue(self, queue, message, params={}, uids=[]):\n d = {\n 'id': uuid.uuid1().get_hex(),\n 'version': 1,\n 'message': message,\n 'params': params\n }\n\n if uids:\n for uid in uids:\n yield self.client.lpush('rhumba.dq.%s.%s' % (\n uid, queue), json.dumps(d))\n else:\n yield self.client.lpush('rhumba.q.%s' % queue, json.dumps(d))\n\n defer.returnValue(d['id'])", "response": "Queue a job in Rhumba"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nretrieve the result of a job from its ID.", "response": "def getResult(self, queue, uid, suid=None):\n \"\"\"\n Retrieve the result of a job from its ID\n \"\"\"\n\n if suid:\n r = yield self.client.get('rhumba.dq.%s.%s.%s' % (suid, queue, uid))\n else:\n r = yield self.client.get('rhumba.q.%s.%s' % (queue, uid))\n\n if r:\n defer.returnValue(json.loads(r))\n else:\n defer.returnValue(None)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef clusterStatus(self):\n servers = yield self.client.keys('rhumba.server.*.heartbeat')\n \n d = {}\n\n now = time.time()\n\n for s in servers:\n sname = s.split('.', 2)[-1].rsplit('.', 1)[0]\n \n last = yield self.client.get('rhumba.server.%s.heartbeat' % sname)\n\n if not last:\n last = 0\n\n status = yield self.client.get('rhumba.server.%s.status' % sname)\n\n if (status == 'ready') and (now - last > 5):\n status = 'offline'\n\n d[sname] = {\n 'lastseen': last,\n 'status': status\n }\n\n defer.returnValue(d)", "response": "Returns a dict of cluster nodes and their status information"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nqueues a job in Rhumba", "response": "def queue(self, queue, message, params={}, uids=[]):\n \"\"\"\n Queue a job in Rhumba\n \"\"\"\n d = {\n 'id': uuid.uuid1().get_hex(),\n 'version': 1,\n 'message': message,\n 'params': params\n }\n\n if uids:\n for uid in uids:\n self._get_client().lpush('rhumba.dq.%s.%s' % (\n uid, queue), json.dumps(d))\n else:\n self._get_client().lpush('rhumba.q.%s' % queue, json.dumps(d))\n\n return d['id']"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef getResult(self, queue, uid, suid=None):\n if suid:\n r = self._get_client().get('rhumba.dq.%s.%s.%s' % (suid, queue, uid))\n else:\n r = self._get_client().get('rhumba.q.%s.%s' % (queue, uid))\n\n if r:\n return json.loads(r)\n else:\n return None", "response": "Retrieve the result of a job from its ID"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a dict of cluster nodes and their status information", "response": "def clusterStatus(self):\n \"\"\"\n Returns a dict of cluster nodes and their status information\n \"\"\"\n c = self._get_client()\n servers = c.keys('rhumba.server.*.heartbeat')\n \n d = {}\n\n now = time.time()\n\n for s in servers:\n sname = s.split('.', 2)[-1].rsplit('.', 1)[0]\n\n last = float(c.get('rhumba.server.%s.heartbeat' % sname))\n status = c.get('rhumba.server.%s.status' % sname)\n\n if (status == 'ready') and (now - last > 5):\n status = 'offline'\n\n d[sname] = {\n 'lastseen': last,\n 'status': status\n }\n\n return d"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting geojson and simplify topology.", "response": "def toposimplify(geojson, p):\n \"\"\"\n Convert geojson and simplify topology.\n\n geojson is a dict representing geojson.\n p is a simplification threshold value between 0 and 1.\n \"\"\"\n proc_out = subprocess.run(\n ['geo2topo'],\n input=bytes(\n json.dumps(geojson),\n 'utf-8'),\n stdout=subprocess.PIPE\n )\n proc_out = subprocess.run(\n ['toposimplify', '-P', p],\n input=proc_out.stdout,\n stdout=subprocess.PIPE,\n stderr=subprocess.DEVNULL\n )\n topojson = json.loads(proc_out.stdout)\n # Standardize object name\n topojson['objects']['divisions'] = topojson['objects'].pop('-')\n return topojson"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_sqlserver_product_version(engine: \"Engine\") -> Tuple[int]:\n assert is_sqlserver(engine), (\n \"Only call get_sqlserver_product_version() for Microsoft SQL Server \"\n \"instances.\"\n )\n sql = \"SELECT CAST(SERVERPROPERTY('ProductVersion') AS VARCHAR)\"\n rp = engine.execute(sql) # type: ResultProxy\n row = rp.fetchone()\n dotted_version = row[0] # type: str # e.g. '12.0.5203.0'\n return tuple(int(x) for x in dotted_version.split(\".\"))", "response": "Returns the SQL Server product version of the current SQL Server instance."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nis the SQLAlchemy Engine an instance of Microsoft SQL Server 2008 or later?", "response": "def is_sqlserver_2008_or_later(engine: \"Engine\") -> bool:\n \"\"\"\n Is the SQLAlchemy :class:`Engine` an instance of Microsoft SQL Server,\n version 2008 or later?\n \"\"\"\n if not is_sqlserver(engine):\n return False\n version_tuple = get_sqlserver_product_version(engine)\n return version_tuple >= (SQLSERVER_MAJOR_VERSION_2008, )"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nadds a column of 1s at the end of the array", "response": "def add_ones(a):\n \"\"\"Adds a column of 1s at the end of the array\"\"\"\n arr = N.ones((a.shape[0],a.shape[1]+1))\n arr[:,:-1] = a\n return arr"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _uninstall(cls):\n if cls._hook:\n sys.meta_path.remove(cls._hook)\n cls._hook = None", "response": "uninstall the hook if installed"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef add(cls, module_name, fsts, composer):\n '''\n add a couple of fsts to be superimposed on the module given\n by module_name as soon as it is imported.\n\n internal - use featuremonkey.compose_later\n '''\n cls._to_compose.setdefault(module_name, [])\n cls._to_compose[module_name].append(\n (list(fsts), composer)\n )\n cls._install()", "response": "add a couple of fsts to be superimposed on the given module"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add(cls, module_name, msg=''):\n '''\n Until the guard is dropped again,\n disallow imports of the module given by ``module_name``.\n\n If the module is imported while the guard is in place\n an ``ImportGuard`` is raised. An additional message on why\n the module cannot be imported can optionally be specified\n using the parameter ``msg``.\n\n If multiple guards are placed on the same module, all these guards\n have to be dropped before the module can be imported again.\n '''\n if module_name in sys.modules:\n raise ImportGuard(\n 'Module to guard has already been imported: '\n + module_name\n )\n cls._guards.setdefault(module_name, [])\n cls._guards[module_name].append(msg)\n cls._num_entries += 1\n cls._install()", "response": "Add a guard to the list of guards that can be imported for the given module."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef remove(cls, module_name):\n module_guards = cls._guards.get(module_name, False)\n if module_guards:\n module_guards.pop()\n cls._num_entries -= 1\n if cls._num_entries < 1:\n if cls._num_entries < 0:\n raise Exception(\n 'Bug: ImportGuardHook._num_entries became negative!'\n )\n cls._uninstall()", "response": "Removes a previously created guard on module_name."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef random_product(*args, repeat=1):\n \"Random selection from itertools.product(*args, **kwds)\"\n pools = [tuple(pool) for pool in args] * repeat\n return tuple(random.choice(pool) for pool in pools)", "response": "Random selection from itertools. product(*args **kwds )"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates a copy of this feature.", "response": "def copy(\n self,\n extractor=None,\n needs=None,\n store=None,\n data_writer=None,\n persistence=None,\n extractor_args=None):\n \"\"\"\n Use self as a template to build a new feature, replacing\n values in kwargs\n \"\"\"\n f = Feature(\n extractor or self.extractor,\n needs=needs,\n store=self.store if store is None else store,\n encoder=self.encoder,\n decoder=self.decoder,\n key=self.key,\n data_writer=data_writer,\n persistence=persistence,\n **(extractor_args or self.extractor_args))\n f._fixup_needs()\n return f"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns True if this feature can be computed from stored dependencies.", "response": "def _can_compute(self, _id, persistence):\n \"\"\"\n Return true if this feature stored, or is unstored, but can be computed\n from stored dependencies\n \"\"\"\n if self.store and self._stored(_id, persistence):\n return True\n\n if self.is_root:\n return False\n\n return all(\n [n._can_compute(_id, persistence) for n in self.dependencies])"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _partial(self, _id, features=None, persistence=None):\n\n root = features is None\n\n stored = self._stored(_id, persistence)\n is_cached = self.store and stored\n\n if self.store and not stored:\n data_writer = None\n elif root:\n data_writer = BytesIODataWriter\n else:\n data_writer = None\n\n should_store = self.store and not stored\n nf = self.copy(\n extractor=DecoderNode if is_cached else self.extractor,\n store=root or should_store,\n needs=dict(),\n data_writer=data_writer,\n persistence=self.persistence,\n extractor_args=dict(decodifier=self.decoder, version=self.version) \\\n if is_cached else self.extractor_args)\n\n if root:\n features = dict()\n\n features[self.key] = nf\n\n if not is_cached:\n for k, v in list(self.needs.items()):\n v._partial(_id, features=features, persistence=persistence)\n nf.needs[k] = features[v.key]\n\n return features", "response": "This method is used to build a partial version of the current node."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef genrows(cursor: Cursor, arraysize: int = 1000) \\\n -> Generator[List[Any], None, None]:\n \"\"\"\n Generate all rows from a cursor.\n\n Args:\n cursor: the cursor\n arraysize: split fetches into chunks of this many records\n\n Yields:\n each row\n \"\"\"\n # http://code.activestate.com/recipes/137270-use-generators-for-fetching-large-db-record-sets/ # noqa\n while True:\n results = cursor.fetchmany(arraysize)\n if not results:\n break\n for result in results:\n yield result", "response": "Generator function that generates all rows from a cursor."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef genfirstvalues(cursor: Cursor, arraysize: int = 1000) \\\n -> Generator[Any, None, None]:\n \"\"\"\n Generate the first value in each row.\n\n Args:\n cursor: the cursor\n arraysize: split fetches into chunks of this many records\n\n Yields:\n the first value of each row\n \"\"\"\n return (row[0] for row in genrows(cursor, arraysize))", "response": "Generate the first value in each row."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef gendicts(cursor: Cursor, arraysize: int = 1000) \\\n -> Generator[Dict[str, Any], None, None]:\n \"\"\"\n Generate all rows from a cursor as :class:`OrderedDict` objects.\n\n Args:\n cursor: the cursor\n arraysize: split fetches into chunks of this many records\n\n Yields:\n each row, as an :class:`OrderedDict` whose key are column names\n and whose values are the row values\n \"\"\"\n columns = get_fieldnames_from_cursor(cursor)\n return (\n OrderedDict(zip(columns, row))\n for row in genrows(cursor, arraysize)\n )", "response": "Generate all rows from a cursor as an ordered dict."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef dictfetchall(cursor: Cursor) -> List[Dict[str, Any]]:\n columns = get_fieldnames_from_cursor(cursor)\n return [\n OrderedDict(zip(columns, row))\n for row in cursor.fetchall()\n ]", "response": "Returns all rows from a cursor as a list of OrderedDicts."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the next row from a cursor as an OrderedDict or None.", "response": "def dictfetchone(cursor: Cursor) -> Optional[Dict[str, Any]]:\n \"\"\"\n Return the next row from a cursor as an :class:`OrderedDict`, or ``None``.\n \"\"\"\n columns = get_fieldnames_from_cursor(cursor)\n row = cursor.fetchone()\n if not row:\n return None\n return OrderedDict(zip(columns, row))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_values_and_permissible(values: Iterable[Tuple[Any, str]],\n add_none: bool = False,\n none_description: str = \"[None]\") \\\n -> Tuple[List[Tuple[Any, str]], List[Any]]:\n \"\"\"\n Used when building Colander nodes.\n\n Args:\n values: an iterable of tuples like ``(value, description)`` used in\n HTML forms\n\n add_none: add a tuple ``(None, none_description)`` at the start of\n ``values`` in the result?\n\n none_description: the description used for ``None`` if ``add_none``\n is set\n\n Returns:\n a tuple ``(values, permissible_values)``, where\n\n - ``values`` is what was passed in (perhaps with the addition of the\n \"None\" tuple at the start)\n - ``permissible_values`` is a list of all the ``value`` elements of\n the original ``values``\n\n \"\"\"\n permissible_values = list(x[0] for x in values)\n # ... does not include the None value; those do not go to the validator\n if add_none:\n none_tuple = (SERIALIZED_NONE, none_description)\n values = [none_tuple] + list(values)\n return values, permissible_values", "response": "Returns the values and permissible values for the current node."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef serialize(self,\n node: SchemaNode,\n appstruct: Union[PotentialDatetimeType,\n ColanderNullType]) \\\n -> Union[str, ColanderNullType]:\n \"\"\"\n Serializes Python object to string representation.\n \"\"\"\n if not appstruct:\n return colander.null\n try:\n appstruct = coerce_to_pendulum(appstruct,\n assume_local=self.use_local_tz)\n except (ValueError, ParserError) as e:\n raise Invalid(\n node,\n \"{!r} is not a pendulum.DateTime object; error was \"\n \"{!r}\".format(appstruct, e))\n return appstruct.isoformat()", "response": "Serializes a datetime object to a string representation."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef serialize(self, node: SchemaNode,\n value: Any) -> Union[str, ColanderNullType]:\n \"\"\"\n Serializes Python object to string representation.\n \"\"\"\n if value is None:\n retval = ''\n else:\n # noinspection PyUnresolvedReferences\n retval = self.type_.serialize(node, value)\n # log.debug(\"AllowNoneType.serialize: {!r} -> {!r}\", value, retval)\n return retval", "response": "Serializes the given value into a string representation."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_features_from_equation_file(filename):\n features = []\n for line in open(filename):\n line = line.split('#')[0].strip()\n if line:\n features.append(line)\n return features", "response": "returns list of feature names read from equation file given by filename"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _create_refinement_wrapper(transformation, baseattr, base, target_attrname):\n # first step: extract the original\n special_refinement_type=None\n instance_refinement = _is_class_instance(base)\n\n if instance_refinement:\n dictelem = base.__class__.__dict__.get(target_attrname, None)\n else:\n dictelem = base.__dict__.get(target_attrname, None)\n\n if isinstance(dictelem, staticmethod):\n special_refinement_type = 'staticmethod'\n original = _extract_staticmethod(dictelem)\n elif isinstance(dictelem, classmethod):\n special_refinement_type = 'classmethod'\n original = _extract_classmethod(dictelem)\n else:\n if instance_refinement:\n # methods need a delegator\n original = _delegate(baseattr)\n # TODO: evaluate this:\n # original = base.__class__.__dict__[target_attrname]\n else:\n # default handling\n original = baseattr\n\n # step two: call the refinement passing it the original\n # the result is the wrapper\n wrapper = transformation(original)\n\n # rescue docstring\n if not wrapper.__doc__:\n wrapper.__doc__ = baseattr.__doc__\n\n # step three: make wrapper ready for injection\n if special_refinement_type == 'staticmethod':\n wrapper = staticmethod(wrapper)\n elif special_refinement_type == 'classmethod':\n wrapper = classmethod(wrapper)\n\n if instance_refinement:\n wrapper = wrapper.__get__(base, base.__class__)\n\n return wrapper", "response": "This function creates a new refinement wrapper for the base class attribute."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _compose_pair(self, role, base):\n '''\n composes onto base by applying the role\n '''\n # apply transformations in role to base\n for attrname in dir(role):\n transformation = getattr(role, attrname)\n self._apply_transformation(role, base, transformation, attrname)\n\n return base", "response": "compose a pair of attributes on base by applying the role to base by applying the role s attributes to base."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncomposing returns a new object with the given list of things", "response": "def compose(self, *things):\n '''\n compose applies multiple fsts onto a base implementation.\n Pass the base implementation as last parameter.\n fsts are merged from RIGHT TO LEFT (like function application)\n e.g.:\n\n class MyFST(object):\n #place introductions and refinements here\n introduce_foo = 'bar'\n\n compose(MyFST(), MyClass)\n '''\n if not len(things):\n raise CompositionError('nothing to compose')\n if len(things) == 1:\n # composing one element is simple\n return things[0]\n else:\n # recurse after applying last role to object\n return self.compose(*(\n list(things[:-2]) # all but the last two\n # plus the composition of the last two\n + [self._compose_pair(things[-2], things[-1])]\n ))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nregister list of things for composition later", "response": "def compose_later(self, *things):\n \"\"\"\n register list of things for composition using compose()\n\n compose_later takes a list of fsts.\n The last element specifies the base module as string\n things are composed directly after the base module\n is imported by application code\n \"\"\"\n if len(things) == 1:\n return things[0]\n module_name = things[-1]\n if module_name in sys.modules:\n raise CompositionError(\n 'compose_later call after module has been imported: '\n + module_name\n )\n LazyComposerHook.add(module_name, things[:-1], self)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef select(self, *features):\n for feature_name in features:\n feature_module = importlib.import_module(feature_name)\n # if available, import feature.py and select the feature\n try:\n feature_spec_module = importlib.import_module(\n feature_name + '.feature'\n )\n if not hasattr(feature_spec_module, 'select'):\n raise CompositionError(\n 'Function %s.feature.select not found!\\n '\n 'Feature modules need to specify a function'\n ' select(composer).' % (\n feature_name\n )\n )\n args, varargs, keywords, defaults = inspect.getargspec(\n feature_spec_module.select\n )\n if varargs or keywords or defaults or len(args) != 1:\n raise CompositionError(\n 'invalid signature: %s.feature.select must '\n 'have the signature select(composer)' % (\n feature_name\n )\n )\n # call the feature`s select function\n feature_spec_module.select(self)\n except ImportError:\n # Unfortunately, python makes it really hard\n # to distinguish missing modules from modules\n # that contain errors.\n # Hacks like parsing the exception message will\n # not work reliably due to import hooks and such.\n # Conclusion: features must contain a feature.py for now\n\n # re-raise\n raise", "response": "Select the given features in the current locale."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef plot_aligned(pca, sparse=True, **kwargs):\n colormap = kwargs.pop('colormap',None)\n\n A = pca.rotated()\n # Flip the result if upside down\n if pca.normal[2] < 0:\n A[:,-1] *= -1\n minmax = [(A[:,i].min(),A[:,i].max()) for i in range(3)]\n lengths = [j-i for i,j in minmax]\n\n if sparse:\n i = 1\n l = len(A)\n if l > 10000:\n i = N.ceil(l/10000)\n A = A[::int(i)]\n\n log.info(\"Plotting with {} points\".format(len(A)))\n\n w = 8\n ratio = (lengths[2]*2+lengths[1])/lengths[0]*1.5\n h = w*ratio\n if h < 3: h = 3\n\n r = (lengths[1]+5)/(lengths[2]+5)\n if r > 5:\n r = 5\n\n fig = Figure(figsize=(w,h))\n fig.canvas = FigureCanvas(fig)\n\n def setup_axes():\n gs = GridSpec(3,1, height_ratios=[r,1,1])\n kwargs = dict()\n axes = []\n for g in gs:\n ax = fig.add_subplot(g,**kwargs)\n kwargs['sharex'] = ax\n yield ax\n\n axes = list(setup_axes())\n\n fig.subplots_adjust(hspace=0, wspace=0.1)\n\n #lengths = attitude.pca.singular_values[::-1]\n\n titles = (\n \"Plan view\",\n \"Long cross-section\",\n \"Short cross-section\")\n\n ylabels = (\n \"Meters\",\n \"Residuals (m)\",\n \"Residuals (m)\")\n\n colors = ['cornflowerblue','red']\n\n hyp = sampling_axes(pca)\n vertical = vector(0,0,1)\n\n for title,ax,(a,b),ylabel in zip(titles,axes,\n [(0,1),(0,2),(1,2)],ylabels):\n\n kw = dict(linewidth=2, alpha=0.5)\n bounds = minmax[a]\n if b != 2:\n ax.plot(bounds,(0,0), c=colors[a], **kw)\n ax.plot((0,0),minmax[b], c=colors[b], **kw)\n else:\n ax.plot(bounds,(-10,-10), c=colors[a], **kw)\n v0 = N.zeros(3)\n v0[a] = 1\n axes = N.array([v0,vertical])\n\n ax_ = (axes@N.diag(hyp)@axes.T).T\n ax_ = N.sqrt(ax_)\n l1 = N.linalg.norm(ax_[:-1])\n l2 = N.linalg.norm(ax_[-1])\n ang_error = 2*N.degrees(N.arctan2(l2,l1))\n\n title += \": {:.0f} m long, angular error (95% CI): {:.2f}\u00ba\".format(lengths[a], ang_error)\n\n bounds = minmax[0]\n x_ = N.linspace(bounds[0]*1.2,bounds[1]*1.2,100)\n\n err = HyperbolicErrors(hyp,x_,axes=axes)\n err.plot(ax, fc='#cccccc', alpha=0.3)\n\n x,y = A[:,a], A[:,b]\n kw = dict(alpha=0.5, zorder=5)\n\n if colormap is None:\n ax.plot(x,y,c=\"#555555\", linestyle='None', marker='.',**kw)\n else:\n ax.scatter(x,y,c=A[:,-1], cmap=colormap, **kw)\n\n #ax.set_aspect(\"equal\")\n\n ax.text(0.01,.99,title,\n verticalalignment='top',\n transform=ax.transAxes)\n #ax.autoscale(tight=True)\n ax.yaxis.set_ticks([])\n ax.xaxis.set_ticks_position('bottom')\n if a != 1:\n pass\n #ax.xaxis.set_ticks([])\n #ax.spines['bottom'].set_color('none')\n for spine in ax.spines.values():\n spine.set_visible(False)\n\n\n ax.text(0.99,0.99,\"Max residual: {:.1f} m\".format(lengths[2]),\n verticalalignment='bottom',\n ha='right',\n transform=ax.transAxes)\n\n\n ax.set_xlabel(\"Meters\")\n return fig", "response": "Plots the residuals of a principal component in the same direction as the given principal component."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nmap each osm POI and geotweets based on spatial lookup of metro area", "response": "def mapper_metro(self, _, data):\n \"\"\" map each osm POI and geotweets based on spatial lookup of metro area \"\"\"\n # OSM POI record\n if 'tags' in data:\n type_tag = 1\n lonlat = data['coordinates']\n payload = data['tags']\n # Tweet with coordinates from Streaming API\n elif 'user_id' in data:\n type_tag = 2\n # only allow tweets from the listed domains to try and filter out\n # noise such as HR tweets, Weather reports and news updates\n accept = [\n \"twitter\\.com\",\n \"foursquare\\.com\",\n \"instagram\\.com\",\n \"untappd\\.com\"\n ]\n expr = \"|\".join(accept)\n if not re.findall(expr, data['source']):\n return\n lonlat = data['lonlat']\n payload = None\n # spatial lookup using Rtree with cached results\n metro = self.lookup.get(lonlat, METRO_DISTANCE)\n if not metro:\n return\n yield metro, (type_tag, lonlat, payload)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\noutputs tags of POI locations nearby tweet locations Values will be sorted coming into reducer. First element in each value tuple will be either 1 (osm POI) or 2 (geotweet). Build a spatial index with POI records. For each tweet lookup nearby POI, and emit tag values for predefined tags.", "response": "def reducer_metro(self, metro, values):\n \"\"\"\n Output tags of POI locations nearby tweet locations\n\n Values will be sorted coming into reducer.\n First element in each value tuple will be either 1 (osm POI) or 2 (geotweet).\n Build a spatial index with POI records.\n For each tweet lookup nearby POI, and emit tag values for predefined tags.\n \n \"\"\"\n lookup = CachedLookup(precision=POI_GEOHASH_PRECISION)\n for i, value in enumerate(values):\n type_tag, lonlat, data = value\n if type_tag == 1:\n # OSM POI node, construct geojson and add to Rtree index\n lookup.insert(i, dict(\n geometry=dict(type='Point', coordinates=project(lonlat)),\n properties=dict(tags=data)\n ))\n else:\n # geotweet, lookup nearest POI from index\n if not lookup.data_store:\n return\n poi_names = []\n kwargs = dict(buffer_size=POI_DISTANCE, multiple=True)\n # lookup nearby POI from Rtree index (caching results)\n # for any tags we care about emit the tags value and 1\n for poi in lookup.get(lonlat, **kwargs):\n has_tag = [ tag in poi['tags'] for tag in POI_TAGS ]\n if any(has_tag) and 'name' in poi['tags']:\n poi_names.append(poi['tags']['name'])\n for poi in set(poi_names):\n yield (metro, poi), 1"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef reducer_count(self, key, values):\n total = sum(values)\n metro, poi = key\n # group data by metro areas for final output \n yield metro, (total, poi)", "response": "count occurences for each metro record"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef reducer_output(self, metro, values):\n records = []\n # build up list of data for each metro area and submit as one network\n # call instead of individually \n for value in values:\n total, poi = value\n records.append(dict(\n metro_area=metro,\n poi=poi,\n count=total\n ))\n output = \"{0}\\t{1}\\t{2}\"\n output = output.format(metro.encode('utf-8'), total, poi.encode('utf-8'))\n yield None, output\n if self.mongo:\n self.mongo.insert_many(records)", "response": "store each record in MongoDB and output tab delimited lines"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nraise an exception if the SQLCompiler is not bound to an unknown dialect.", "response": "def fail_unknown_dialect(compiler: \"SQLCompiler\", task: str) -> None:\n \"\"\"\n Raise :exc:`NotImplementedError` in relation to a dialect for which a\n function hasn't been implemented (with a helpful error message).\n \"\"\"\n raise NotImplementedError(\n \"Don't know how to {task} on dialect {dialect!r}. \"\n \"(Check also: if you printed the SQL before it was bound to an \"\n \"engine, you will be trying to use a dialect like StrSQLCompiler, \"\n \"which could be a reason for failure.)\".format(\n task=task,\n dialect=compiler.dialect\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef fetch_processed_single_clause(element: \"ClauseElement\",\n compiler: \"SQLCompiler\") -> str:\n \"\"\"\n Takes a clause element that must have a single clause, and converts it\n to raw SQL text.\n \"\"\"\n if len(element.clauses) != 1:\n raise TypeError(\"Only one argument supported; {} were passed\".format(\n len(element.clauses)))\n clauselist = element.clauses # type: ClauseList\n first = clauselist.get_children()[0]\n return compiler.process(first)", "response": "Takes a clause element that must have a single clause and converts it to raw SQL text."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting a parameter to an integer or raises a forms. ValidationError.", "response": "def clean_int(x) -> int:\n \"\"\"\n Returns its parameter as an integer, or raises\n ``django.forms.ValidationError``.\n \"\"\"\n try:\n return int(x)\n except ValueError:\n raise forms.ValidationError(\n \"Cannot convert to integer: {}\".format(repr(x)))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef clean_nhs_number(x) -> int:\n try:\n x = int(x)\n if not is_valid_nhs_number(x):\n raise ValueError\n return x\n except ValueError:\n raise forms.ValidationError(\n \"Not a valid NHS number: {}\".format(repr(x)))", "response": "Returns its parameter as a valid integer NHS number or raises a forms. ValidationError."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef coltype_as_typeengine(coltype: Union[VisitableType,\n TypeEngine]) -> TypeEngine:\n \"\"\"\n Instances of SQLAlchemy column types are subclasses of ``TypeEngine``.\n It's possible to specify column types either as such instances, or as the\n class type. This function ensures that such classes are converted to\n instances.\n\n To explain: you can specify columns like\n\n .. code-block:: python\n\n a = Column(\"a\", Integer)\n b = Column(\"b\", Integer())\n c = Column(\"c\", String(length=50))\n\n isinstance(Integer, TypeEngine) # False\n isinstance(Integer(), TypeEngine) # True\n isinstance(String(length=50), TypeEngine) # True\n\n type(Integer) # <class 'sqlalchemy.sql.visitors.VisitableType'>\n type(Integer()) # <class 'sqlalchemy.sql.sqltypes.Integer'>\n type(String) # <class 'sqlalchemy.sql.visitors.VisitableType'>\n type(String(length=50)) # <class 'sqlalchemy.sql.sqltypes.String'>\n\n This function coerces things to a :class:`TypeEngine`.\n \"\"\"\n if isinstance(coltype, TypeEngine):\n return coltype\n return coltype()", "response": "Returns the given column type as a typeengine."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef walk_orm_tree(obj,\n debug: bool = False,\n seen: Set = None,\n skip_relationships_always: List[str] = None,\n skip_relationships_by_tablename: Dict[str, List[str]] = None,\n skip_all_relationships_for_tablenames: List[str] = None,\n skip_all_objects_for_tablenames: List[str] = None) \\\n -> Generator[object, None, None]:\n \"\"\"\n Starting with a SQLAlchemy ORM object, this function walks a\n relationship tree, yielding each of the objects once.\n\n To skip attributes by name, put the attribute name(s) in\n ``skip_attrs_always``. To skip by table name, pass\n ``skip_attrs_by_tablename`` as e.g.\n\n .. code-block:: python\n\n {'sometable': ['attr1_to_skip', 'attr2_to_skip']}\n\n\n Args:\n obj: the SQLAlchemy ORM object to walk\n\n debug: be verbose\n\n seen: usually ``None``, but can be a set of objects marked as \"already\n seen\"; if an object is in this set, it is skipped\n\n skip_relationships_always: relationships are skipped if the\n relationship has a name in this (optional) list\n\n skip_relationships_by_tablename: optional dictionary mapping table\n names (keys) to relationship attribute names (values); if the\n \"related table\"/\"relationship attribute\" pair are in this\n dictionary, the relationship is skipped\n\n skip_all_relationships_for_tablenames: relationships are skipped if the\n the related table has a name in this (optional) list\n\n skip_all_objects_for_tablenames: if the object belongs to a table whose\n name is in this (optional) list, the object is skipped\n\n Yields:\n SQLAlchemy ORM objects (including the starting object)\n\n \"\"\"\n # http://docs.sqlalchemy.org/en/latest/faq/sessions.html#faq-walk-objects\n skip_relationships_always = skip_relationships_always or [] # type: List[str] # noqa\n skip_relationships_by_tablename = skip_relationships_by_tablename or {} # type: Dict[str, List[str]] # noqa\n skip_all_relationships_for_tablenames = skip_all_relationships_for_tablenames or [] # type: List[str] # noqa\n skip_all_objects_for_tablenames = skip_all_objects_for_tablenames or [] # type: List[str] # noqa\n stack = [obj]\n if seen is None:\n seen = set()\n while stack:\n obj = stack.pop(0)\n if obj in seen:\n continue\n tablename = obj.__tablename__\n if tablename in skip_all_objects_for_tablenames:\n continue\n seen.add(obj)\n if debug:\n log.debug(\"walk: yielding {!r}\", obj)\n yield obj\n insp = inspect(obj) # type: InstanceState\n for relationship in insp.mapper.relationships: # type: RelationshipProperty # noqa\n attrname = relationship.key\n # Skip?\n if attrname in skip_relationships_always:\n continue\n if tablename in skip_all_relationships_for_tablenames:\n continue\n if (tablename in skip_relationships_by_tablename and\n attrname in skip_relationships_by_tablename[tablename]):\n continue\n # Process relationship\n if debug:\n log.debug(\"walk: following relationship {}\", relationship)\n related = getattr(obj, attrname)\n if debug and related:\n log.debug(\"walk: queueing {!r}\", related)\n if relationship.uselist:\n stack.extend(related)\n elif related is not None:\n stack.append(related)", "response": "This function walks the SQLAlchemy ORM object tree and yields each of the objects in the order they appear in the SQLAlchemy ORM object tree."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngives an SQLAlchemy object creates a new object and copies all properties of the object into a new object.", "response": "def copy_sqla_object(obj: object,\n omit_fk: bool = True,\n omit_pk: bool = True,\n omit_attrs: List[str] = None,\n debug: bool = False) -> object:\n \"\"\"\n Given an SQLAlchemy object, creates a new object (FOR WHICH THE OBJECT\n MUST SUPPORT CREATION USING ``__init__()`` WITH NO PARAMETERS), and copies\n across all attributes, omitting PKs (by default), FKs (by default), and\n relationship attributes (always omitted).\n\n Args:\n obj: the object to copy\n omit_fk: omit foreign keys (FKs)?\n omit_pk: omit primary keys (PKs)?\n omit_attrs: attributes (by name) not to copy\n debug: be verbose\n \n Returns:\n a new copy of the object\n \"\"\"\n omit_attrs = omit_attrs or [] # type: List[str]\n cls = type(obj)\n mapper = class_mapper(cls)\n newobj = cls() # not: cls.__new__(cls)\n rel_keys = set([c.key for c in mapper.relationships])\n prohibited = rel_keys\n if omit_pk:\n pk_keys = set([c.key for c in mapper.primary_key])\n prohibited |= pk_keys\n if omit_fk:\n fk_keys = set([c.key for c in mapper.columns if c.foreign_keys])\n prohibited |= fk_keys\n prohibited |= set(omit_attrs)\n if debug:\n log.debug(\"copy_sqla_object: skipping: {}\", prohibited)\n for k in [p.key for p in mapper.iterate_properties\n if p.key not in prohibited]:\n try:\n value = getattr(obj, k)\n if debug:\n log.debug(\"copy_sqla_object: processing attribute {} = {}\",\n k, value)\n setattr(newobj, k, value)\n except AttributeError:\n if debug:\n log.debug(\"copy_sqla_object: failed attribute {}\", k)\n pass\n return newobj"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef deepcopy_sqla_objects(\n startobjs: List[object],\n session: Session,\n flush: bool = True,\n debug: bool = False,\n debug_walk: bool = True,\n debug_rewrite_rel: bool = False,\n objmap: Dict[object, object] = None) -> None:\n \"\"\"\n Makes a copy of the specified SQLAlchemy ORM objects, inserting them into a\n new session.\n\n This function operates in several passes:\n\n 1. Walk the ORM tree through all objects and their relationships, copying\n every object thus found (via :func:`copy_sqla_object`, without their\n relationships), and building a map from each source-session object to\n its equivalent destination-session object.\n\n 2. Work through all the destination objects, rewriting their relationships\n (via :func:`rewrite_relationships`) so they relate to each other (rather\n than their source-session brethren).\n\n 3. Insert all the destination-session objects into the destination session.\n\n For this to succeed, every object must take an ``__init__`` call with no\n arguments (see :func:`copy_sqla_object`). (We can't specify the required\n ``args``/``kwargs``, since we are copying a tree of arbitrary objects.)\n\n Args:\n startobjs: SQLAlchemy ORM objects to copy\n session: destination SQLAlchemy :class:`Session` into which to insert\n the copies\n flush: flush the session when we've finished?\n debug: be verbose?\n debug_walk: be extra verbose when walking the ORM tree?\n debug_rewrite_rel: be extra verbose when rewriting relationships?\n objmap: starting object map from source-session to destination-session\n objects (see :func:`rewrite_relationships` for more detail);\n usually ``None`` to begin with.\n \"\"\"\n if objmap is None:\n objmap = {} # keys = old objects, values = new objects\n if debug:\n log.debug(\"deepcopy_sqla_objects: pass 1: create new objects\")\n\n # Pass 1: iterate through all objects. (Can't guarantee to get\n # relationships correct until we've done this, since we don't know whether\n # or where the \"root\" of the PK tree is.)\n seen = set()\n for startobj in startobjs:\n for oldobj in walk_orm_tree(startobj, seen=seen, debug=debug_walk):\n if debug:\n log.debug(\"deepcopy_sqla_objects: copying {}\", oldobj)\n newobj = copy_sqla_object(oldobj, omit_pk=True, omit_fk=True)\n # Don't insert the new object into the session here; it may trigger\n # an autoflush as the relationships are queried, and the new\n # objects are not ready for insertion yet (as their relationships\n # aren't set).\n # Note also the session.no_autoflush option:\n # \"sqlalchemy.exc.OperationalError: (raised as a result of Query-\n # invoked autoflush; consider using a session.no_autoflush block if\n # this flush is occurring prematurely)...\"\n objmap[oldobj] = newobj\n\n # Pass 2: set all relationship properties.\n if debug:\n log.debug(\"deepcopy_sqla_objects: pass 2: set relationships\")\n for oldobj, newobj in objmap.items():\n if debug:\n log.debug(\"deepcopy_sqla_objects: newobj: {}\", newobj)\n rewrite_relationships(oldobj, newobj, objmap, debug=debug_rewrite_rel)\n\n # Now we can do session insert.\n if debug:\n log.debug(\"deepcopy_sqla_objects: pass 3: insert into session\")\n for newobj in objmap.values():\n session.add(newobj)\n\n # Done\n if debug:\n log.debug(\"deepcopy_sqla_objects: done\")\n if flush:\n session.flush()", "response": "This function copies SQLAlchemy ORM objects into a new SQLAlchemy ORM object tree."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef deepcopy_sqla_object(startobj: object,\n session: Session,\n flush: bool = True,\n debug: bool = False,\n debug_walk: bool = False,\n debug_rewrite_rel: bool = False,\n objmap: Dict[object, object] = None) -> object:\n \"\"\"\n Makes a copy of the object, inserting it into ``session``.\n\n Uses :func:`deepcopy_sqla_objects` (q.v.).\n\n A problem is the creation of duplicate dependency objects if you call it\n repeatedly.\n\n Optionally, if you pass the objmap in (which maps old to new objects), you\n can call this function repeatedly to clone a related set of objects...\n ... no, that doesn't really work, as it doesn't visit parents before\n children. The :func:`cardinal_pythonlib.sqlalchemy.merge_db.merge_db`\n function does that properly.\n\n Args:\n startobj: SQLAlchemy ORM object to deep-copy\n session: see :func:`deepcopy_sqla_objects`\n flush: see :func:`deepcopy_sqla_objects`\n debug: see :func:`deepcopy_sqla_objects`\n debug_walk: see :func:`deepcopy_sqla_objects`\n debug_rewrite_rel: see :func:`deepcopy_sqla_objects`\n objmap: see :func:`deepcopy_sqla_objects`\n\n Returns:\n the copied object matching ``startobj``\n\n \"\"\"\n if objmap is None:\n objmap = {} # keys = old objects, values = new objects\n deepcopy_sqla_objects(\n startobjs=[startobj],\n session=session,\n flush=flush,\n debug=debug,\n debug_walk=debug_walk,\n debug_rewrite_rel=debug_rewrite_rel,\n objmap=objmap\n )\n return objmap[startobj]", "response": "This function copies an SQLAlchemy ORM object into a new SQLAlchemy object."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef gen_columns(obj) -> Generator[Tuple[str, Column], None, None]:\n mapper = obj.__mapper__ # type: Mapper\n assert mapper, \"gen_columns called on {!r} which is not an \" \\\n \"SQLAlchemy ORM object\".format(obj)\n colmap = mapper.columns # type: OrderedProperties\n if not colmap:\n return\n for attrname, column in colmap.items():\n # NB: column.name is the SQL column name, not the attribute name\n yield attrname, column", "response": "Asks a SQLAlchemy ORM object : what are your SQLAlchemy columns?"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_pk_attrnames(obj) -> List[str]:\n return [attrname\n for attrname, column in gen_columns(obj)\n if column.primary_key]", "response": "Asks an SQLAlchemy ORM object to be what are your primary key ( s?"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngenerating tuples of names and column objects for the given UNINSTRUMENTED class.", "response": "def gen_columns_for_uninstrumented_class(cls: Type) \\\n -> Generator[Tuple[str, Column], None, None]:\n \"\"\"\n Generate ``(attr_name, Column)`` tuples from an UNINSTRUMENTED class, i.e.\n one that does not inherit from ``declarative_base()``. Use this for mixins\n of that kind.\n\n SUBOPTIMAL. May produce warnings like:\n \n .. code-block:: none\n \n SAWarning: Unmanaged access of declarative attribute id from non-mapped class GenericTabletRecordMixin\n\n Try to use :func:`gen_columns` instead.\n \"\"\" # noqa\n for attrname in dir(cls):\n potential_column = getattr(cls, attrname)\n if isinstance(potential_column, Column):\n yield attrname, potential_column"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef attrname_to_colname_dict(cls) -> Dict[str, str]:\n attr_col = {} # type: Dict[str, str]\n for attrname, column in gen_columns(cls):\n attr_col[attrname] = column.name\n return attr_col", "response": "Asks an SQLAlchemy class how its attribute names correspond to database column names."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef gen_relationships(obj) -> Generator[Tuple[str, RelationshipProperty, Type],\n None, None]:\n \"\"\"\n Yields tuples of ``(attrname, RelationshipProperty, related_class)``\n for all relationships of an ORM object.\n The object 'obj' can be EITHER an instance OR a class.\n \"\"\"\n insp = inspect(obj) # type: InstanceState\n # insp.mapper.relationships is of type\n # sqlalchemy.utils._collections.ImmutableProperties, which is basically\n # a sort of AttrDict.\n for attrname, rel_prop in insp.mapper.relationships.items(): # type: Tuple[str, RelationshipProperty] # noqa\n # noinspection PyUnresolvedReferences\n related_class = rel_prop.mapper.class_\n # log.critical(\"gen_relationships: attrname={!r}, \"\n # \"rel_prop={!r}, related_class={!r}, rel_prop.info={!r}\",\n # attrname, rel_prop, related_class, rel_prop.info)\n yield attrname, rel_prop, related_class", "response": "Yields tuples of attrname rel_prop related_class for all relationships of an ORM object."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets a list of columns from an SQLAlchemy ORM class.", "response": "def get_orm_columns(cls: Type) -> List[Column]:\n \"\"\"\n Gets :class:`Column` objects from an SQLAlchemy ORM class.\n Does not provide their attribute names.\n \"\"\"\n mapper = inspect(cls) # type: Mapper\n # ... returns InstanceState if called with an ORM object\n # http://docs.sqlalchemy.org/en/latest/orm/session_state_management.html#session-object-states # noqa\n # ... returns Mapper if called with an ORM class\n # http://docs.sqlalchemy.org/en/latest/orm/mapping_api.html#sqlalchemy.orm.mapper.Mapper # noqa\n colmap = mapper.columns # type: OrderedProperties\n return colmap.values()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget the column names from an SQLAlchemy ORM class.", "response": "def get_orm_column_names(cls: Type, sort: bool = False) -> List[str]:\n \"\"\"\n Gets column names (that is, database column names) from an SQLAlchemy\n ORM class.\n \"\"\"\n colnames = [col.name for col in get_orm_columns(cls)]\n return sorted(colnames) if sort else colnames"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns all database table names found in an SQLAlchemy MetaData object.", "response": "def get_table_names_from_metadata(metadata: MetaData) -> List[str]:\n \"\"\"\n Returns all database table names found in an SQLAlchemy :class:`MetaData`\n object.\n \"\"\"\n return [table.name for table in metadata.tables.values()]"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nyield all the ORM base classes except those that are abstract.", "response": "def gen_orm_classes_from_base(base: Type) -> Generator[Type, None, None]:\n \"\"\"\n From an SQLAlchemy ORM base class, yield all the subclasses (except those\n that are abstract).\n\n If you begin with the proper :class`Base` class, then this should give all\n ORM classes in use.\n \"\"\"\n for cls in gen_all_subclasses(base):\n if _get_immediate_cls_attr(cls, '__abstract__', strict=True):\n # This is SQLAlchemy's own way of detecting abstract classes; see\n # sqlalchemy.ext.declarative.base\n continue # NOT an ORM class\n yield cls"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_orm_classes_by_table_name_from_base(base: Type) -> Dict[str, Type]:\n # noinspection PyUnresolvedReferences\n return {cls.__tablename__: cls for cls in gen_orm_classes_from_base(base)}", "response": "Given an SQLAlchemy ORM base class returns a dictionary whose keys are SQLAlchemy ORM table names and whose values are ORM classes."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning what looks like a plain object with the values of the SQLAlchemy ORM object.", "response": "def get_attrdict(self) -> OrderedNamespace:\n \"\"\"\n Returns what looks like a plain object with the values of the\n SQLAlchemy ORM object.\n \"\"\"\n # noinspection PyUnresolvedReferences\n columns = self.__table__.columns.keys()\n values = (getattr(self, x) for x in columns)\n zipped = zip(columns, values)\n return OrderedNamespace(zipped)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef from_attrdict(cls, attrdict: OrderedNamespace) -> object:\n dictionary = attrdict.__dict__\n # noinspection PyArgumentList\n return cls(**dictionary)", "response": "Builds an ORM object from an attribute dictionary."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning True if the derived class implements a method of the given name.", "response": "def derived_class_implements_method(derived: Type[T1],\n base: Type[T2],\n method_name: str) -> bool:\n \"\"\"\n Does a derived class implement a method (and not just inherit a base\n class's version)?\n\n Args:\n derived: a derived class\n base: a base class\n method_name: the name of a method\n\n Returns:\n whether the derived class method is (a) present, and (b) different to\n the base class's version of the same method\n\n Note: if C derives from B derives from A, then a check on C versus A will\n succeed if C implements the method, or if C inherits it from B but B has\n re-implemented it compared to A.\n\n \"\"\"\n derived_method = getattr(derived, method_name, None)\n if derived_method is None:\n return False\n base_method = getattr(base, method_name, None)\n # if six.PY2:\n # return derived_method.__func__ != base_method.__func__\n # else:\n # return derived_method is not base_method\n return derived_method is not base_method"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngenerating all subclasses of a class.", "response": "def gen_all_subclasses(cls: Type) -> Generator[Type, None, None]:\n \"\"\"\n Generates all subclasses of a class.\n\n Args:\n cls: a class\n\n Yields:\n all subclasses\n\n \"\"\"\n\n for s1 in cls.__subclasses__():\n yield s1\n for s2 in gen_all_subclasses(s1):\n yield s2"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef augment_tensor(matrix, ndim=None):\n s = matrix.shape\n if ndim is None:\n ndim = s[0]+1\n arr = N.identity(ndim)\n arr[:s[0],:s[1]] = matrix\n return arr", "response": "Augment a tensor with an identity matrix."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef angle(v1,v2, cos=False):\n n = (norm(v1)*norm(v2))\n _ = dot(v1,v2)/n\n return _ if cos else N.arccos(_)", "response": "Find the angle between two vectors."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef perpendicular_vector(n):\n dim = len(n)\n if dim == 2:\n return n[::-1]\n # More complex in 3d\n for ix in range(dim):\n _ = N.zeros(dim)\n # Try to keep axes near the global projection\n # by finding vectors perpendicular to higher-\n # index axes first. This may or may not be worth\n # doing.\n _[dim-ix-1] = 1\n v1 = N.cross(n,_)\n if N.linalg.norm(v1) != 0:\n return v1\n raise ValueError(\"Cannot find perpendicular vector\")", "response": "Get a random vector perpendicular to the given vector"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntraps Ctrl - C and Ctrl - B signals from doing Ctrl - C.", "response": "def trap_ctrl_c_ctrl_break() -> None:\n \"\"\"\n Prevent ``CTRL-C``, ``CTRL-BREAK``, and similar signals from doing\n anything.\n \n See\n \n - https://docs.python.org/3/library/signal.html#signal.SIG_IGN\n - https://msdn.microsoft.com/en-us/library/xdkz3x12.aspx\n - https://msdn.microsoft.com/en-us/library/windows/desktop/ms682541(v=vs.85).aspx\n\n Under Windows, the only options are:\n \n =========== ======================= =====================================\n Signal Meaning Comment\n =========== ======================= =====================================\n SIGABRT abnormal termination\n SIGFPE floating-point error\n SIGILL illegal instruction\n SIGINT CTRL+C signal -- trapped here\n SIGSEGV illegal storage access\n SIGTERM termination request -- trapped here\n SIGBREAK CTRL+BREAK -- trapped here under Windows\n =========== ======================= =====================================\n\n In Linux, you also find:\n\n =========== =============================\n Signal Meaning\n =========== =============================\n SIGBUS bus error / unaligned access\n =========== =============================\n\n To ignore, can do:\n \n .. code-block:: python\n \n signal.signal(signal.SIGINT, signal.SIG_IGN) # SIG_IGN = \"ignore me\"\n \n or pass a specified handler, as in the code here.\n \"\"\" # noqa\n\n signal.signal(signal.SIGINT, ctrl_c_trapper)\n signal.signal(signal.SIGTERM, sigterm_trapper)\n if platform.system() == 'Windows':\n # SIGBREAK isn't in the Linux signal module\n # noinspection PyUnresolvedReferences\n signal.signal(signal.SIGBREAK, ctrl_break_trapper)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef is_openxml_good(filename: str) -> bool:\n try:\n log.debug(\"Trying: {}\", filename)\n with ZipFile(filename, 'r') as zip_ref:\n namelist = zip_ref.namelist() # type: List[str]\n # log.critical(\"\\n{}\", pformat(namelist))\n # -----------------------------------------------------------------\n # Contains key files?\n # -----------------------------------------------------------------\n for mandatory_filename in MANDATORY_FILENAMES:\n if mandatory_filename not in namelist:\n log.debug(\"Bad [missing {!r}]: {}\",\n mandatory_filename, filename)\n return False\n\n infolist = zip_ref.infolist() # type: List[ZipInfo]\n contains_docx = False\n contains_pptx = False\n contains_xlsx = False\n for info in infolist:\n # -------------------------------------------------------------\n # Sensible date check?\n # ... NO: lots of perfectly good files have this date/time.\n # -------------------------------------------------------------\n # if info.date_time == NULL_DATE_TIME:\n # log.debug(\"{!r}: {!r}\", info.filename, info.date_time)\n\n # -------------------------------------------------------------\n # Only one kind of contents?\n # ... YES, I think so. This has 100% reliability on my\n # stash of 34 PPTX, 223 DOCX, 85 XLSX, and labelled none as bad\n # from an HFC collection of 1866 such files. There are lots of\n # files emerging from Scalpel (plus my find_recovered_openxml\n # zip-fixing tool) that fail this test, though.\n # -------------------------------------------------------------\n if (not contains_docx and\n DOCX_CONTENTS_REGEX.search(info.filename)):\n contains_docx = True\n if (not contains_pptx and\n PPTX_CONTENTS_REGEX.search(info.filename)):\n contains_pptx = True\n if (not contains_xlsx and\n XLSX_CONTENTS_REGEX.search(info.filename)):\n contains_xlsx = True\n if sum([contains_docx, contains_pptx, contains_xlsx]) > 1:\n log.debug(\"Bad [>1 of DOCX, PPTX, XLSX content]: {}\",\n filename)\n return False\n\n return True\n except (BadZipFile, OSError) as e:\n # ---------------------------------------------------------------------\n # Duff file. Easy!\n # ---------------------------------------------------------------------\n log.debug(\"Bad [BadZipFile or OSError]: {!r}; error was {!r}\",\n filename, e)\n return False", "response": "Determines whether an OpenXML file appears to be good."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nprocess an OpenXML file based on whether it is good or not.", "response": "def process_openxml_file(filename: str,\n print_good: bool,\n delete_if_bad: bool) -> None:\n \"\"\"\n Prints the filename of, or deletes, an OpenXML file depending on whether\n it is corrupt or not.\n\n Args:\n filename: filename to check\n print_good: if ``True``, then prints the filename if the file\n appears good.\n delete_if_bad: if ``True``, then deletes the file if the file\n appears corrupt.\n \"\"\"\n print_bad = not print_good\n try:\n file_good = is_openxml_good(filename)\n file_bad = not file_good\n if (print_good and file_good) or (print_bad and file_bad):\n print(filename)\n if delete_if_bad and file_bad:\n log.warning(\"Deleting: {}\", filename)\n os.remove(filename)\n except Exception as e:\n # Must explicitly catch and report errors, since otherwise they vanish\n # into the ether.\n log.critical(\"Uncaught error in subprocess: {!r}\\n{}\", e,\n traceback.format_exc())\n raise"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef main() -> None:\n parser = ArgumentParser(\n formatter_class=RawDescriptionHelpFormatter,\n description=\"\"\"\nTool to scan rescued Microsoft Office OpenXML files (produced by the\nfind_recovered_openxml.py tool in this kit; q.v.) and detect bad (corrupted)\nones.\n \"\"\"\n )\n parser.add_argument(\n \"filename\", nargs=\"*\",\n help=\"File(s) to check. You can also specify directores if you use \"\n \"--recursive\"\n )\n parser.add_argument(\n \"--filenames_from_stdin\", \"-x\", action=\"store_true\",\n help=\"Take filenames from stdin instead, one line per filename \"\n \"(useful for chained grep).\"\n )\n parser.add_argument(\n \"--recursive\", action=\"store_true\",\n help=\"Allow search to descend recursively into any directories \"\n \"encountered.\"\n )\n parser.add_argument(\n \"--skip_files\", nargs=\"*\", default=[],\n help=\"File pattern(s) to skip. You can specify wildcards like '*.txt' \"\n \"(but you will have to enclose that pattern in quotes under \"\n \"UNIX-like operating systems). The basename of each file will be \"\n \"tested against these filenames/patterns. Consider including \"\n \"Scalpel's 'audit.txt'.\"\n )\n parser.add_argument(\n \"--good\", action=\"store_true\",\n help=\"List good files, not bad\"\n )\n parser.add_argument(\n \"--delete_if_bad\", action=\"store_true\",\n help=\"If a file is found to be bad, delete it. DANGEROUS.\"\n )\n parser.add_argument(\n \"--run_repeatedly\", type=int,\n help=\"Run the tool repeatedly with a pause of <run_repeatedly> \"\n \"seconds between runs. (For this to work well with the move/\"\n \"delete options, you should specify one or more DIRECTORIES in \"\n \"the 'filename' arguments, not files, and you will need the \"\n \"--recursive option.)\"\n )\n parser.add_argument(\n \"--nprocesses\", type=int, default=multiprocessing.cpu_count(),\n help=\"Specify the number of processes to run in parallel.\"\n )\n parser.add_argument(\n \"--verbose\", action=\"store_true\",\n help=\"Verbose output\"\n )\n args = parser.parse_args()\n main_only_quicksetup_rootlogger(\n level=logging.DEBUG if args.verbose else logging.INFO,\n with_process_id=True\n )\n if bool(args.filenames_from_stdin) == bool(args.filename):\n raise ValueError(\"Specify --filenames_from_stdin or filenames on the \"\n \"command line, but not both\")\n if args.filenames_from_stdin and args.run_repeatedly:\n raise ValueError(\"Can't use both --filenames_from_stdin and \"\n \"--run_repeatedly\")\n\n # Repeated scanning loop\n while True:\n log.debug(\"Starting scan.\")\n log.debug(\"- Scanning files/directories {!r}{}\",\n args.filename,\n \" recursively\" if args.recursive else \"\")\n log.debug(\"- Skipping files matching {!r}\", args.skip_files)\n log.debug(\"- Using {} simultaneous processes\", args.nprocesses)\n log.debug(\"- Reporting {} filenames\", \"good\" if args.good else \"bad\")\n if args.delete_if_bad:\n log.warning(\"- Deleting bad OpenXML files.\")\n\n # Iterate through files\n pool = multiprocessing.Pool(processes=args.nprocesses)\n\n if args.filenames_from_stdin:\n generator = gen_from_stdin()\n else:\n generator = gen_filenames(starting_filenames=args.filename,\n recursive=args.recursive)\n\n for filename in generator:\n src_basename = os.path.basename(filename)\n if any(fnmatch.fnmatch(src_basename, pattern)\n for pattern in args.skip_files):\n log.debug(\"Skipping file as ordered: \" + filename)\n continue\n exists, locked = exists_locked(filename)\n if locked or not exists:\n log.debug(\"Skipping currently inaccessible file: \" + filename)\n continue\n kwargs = {\n 'filename': filename,\n 'print_good': args.good,\n 'delete_if_bad': args.delete_if_bad,\n }\n # log.critical(\"start\")\n pool.apply_async(process_openxml_file, [], kwargs)\n # result = pool.apply_async(process_file, [], kwargs)\n # result.get() # will re-raise any child exceptions\n # ... but it waits for the process to complete! That's no help.\n # log.critical(\"next\")\n # ... https://stackoverflow.com/questions/22094852/how-to-catch-exceptions-in-workers-in-multiprocessing # noqa\n pool.close()\n pool.join()\n\n log.debug(\"Finished scan.\")\n if args.run_repeatedly is None:\n break\n log.info(\"Sleeping for {} s...\", args.run_repeatedly)\n sleep(args.run_repeatedly)", "response": "This function is called by the command line interface to find bad OpenXML files."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef open(self) -> bool:\n if self.connection:\n # Nothing to do if the connection is already open.\n return False\n\n connection_params = {'local_hostname': DNS_NAME.get_fqdn()}\n if self.timeout is not None:\n connection_params['timeout'] = self.timeout\n try:\n self.connection = smtplib.SMTP(self.host, self.port,\n **connection_params)\n\n # TLS\n context = ssl.SSLContext(self._protocol())\n if self.ssl_certfile:\n context.load_cert_chain(certfile=self.ssl_certfile,\n keyfile=self.ssl_keyfile)\n self.connection.ehlo()\n self.connection.starttls(context=context)\n self.connection.ehlo()\n if self.username and self.password:\n self.connection.login(self.username, self.password)\n log.debug(\"Successful SMTP connection/login\")\n else:\n log.debug(\"Successful SMTP connection (without login)\")\n return True\n except smtplib.SMTPException:\n log.debug(\"SMTP connection and/or login failed\")\n if not self.fail_silently:\n raise", "response": "Open the SMTP connection to the email server."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef bhattacharyya_distance(pca1,pca2):\n u1 = pca1.coefficients\n s1 = pca1.covariance_matrix\n u2 = pca2.coefficients\n s2 = pca2.covariance_matrix\n\n sigma = (s1+s2)/2\n\n assert all(u1 > 0)\n assert all(u2 > 0)\n assert all(s1.sum(axis=1) > 0)\n assert all(s2.sum(axis=1) > 0)\n\n _ = 1/8*dot((u1-u2).T, N.linalg.inv(sigma),u1-u2)\n _ += 1/2*N.log(N.linalg.det(sigma)/(N.linalg.det(s1)*N.linalg.det(s2)))\n return _", "response": "A measure of the distance between two probability distributions\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef produce_csv_output(filehandle: TextIO,\n fields: Sequence[str],\n values: Iterable[str]) -> None:\n \"\"\"\n Produce CSV output, without using ``csv.writer``, so the log can be used\n for lots of things.\n\n - ... eh? What was I talking about?\n - POOR; DEPRECATED.\n\n Args:\n filehandle: file to write to\n fields: field names\n values: values\n \"\"\"\n output_csv(filehandle, fields)\n for row in values:\n output_csv(filehandle, row)", "response": "Produce CSV output for the current language."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nwrites a line of CSV. POOR does not escape things properly. DEPRECATED.", "response": "def output_csv(filehandle: TextIO, values: Iterable[str]) -> None:\n \"\"\"\n Write a line of CSV. POOR; does not escape things properly. DEPRECATED.\n\n Args:\n filehandle: file to write to\n values: values\n \"\"\"\n line = \",\".join(values)\n filehandle.write(line + \"\\n\")"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_what_follows_raw(s: str,\n prefix: str,\n onlyatstart: bool = True,\n stripwhitespace: bool = True) -> Tuple[bool, str]:\n \"\"\"\n Find the part of ``s`` that is after ``prefix``.\n\n Args:\n s: string to analyse\n prefix: prefix to find\n onlyatstart: only accept the prefix if it is right at the start of\n ``s``\n stripwhitespace: remove whitespace from the result\n\n Returns:\n tuple: ``(found, result)``\n\n \"\"\"\n prefixstart = s.find(prefix)\n if ((prefixstart == 0 and onlyatstart) or\n (prefixstart != -1 and not onlyatstart)):\n # substring found\n resultstart = prefixstart + len(prefix)\n result = s[resultstart:]\n if stripwhitespace:\n result = result.strip()\n return True, result\n return False, \"\"", "response": "Find the part of s that is after prefix."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_what_follows(strings: Sequence[str],\n prefix: str,\n onlyatstart: bool = True,\n stripwhitespace: bool = True,\n precedingline: str = \"\") -> str:\n \"\"\"\n Find a string in ``strings`` that begins with ``prefix``; return the part\n that's after ``prefix``. Optionally, require that the preceding string\n (line) is ``precedingline``.\n\n Args:\n strings: strings to analyse\n prefix: prefix to find\n onlyatstart: only accept the prefix if it is right at the start of\n ``s``\n stripwhitespace: remove whitespace from the result\n precedingline: if truthy, require that the preceding line be as\n specified here\n\n Returns:\n the line fragment\n\n \"\"\"\n if not precedingline:\n for s in strings:\n (found, result) = get_what_follows_raw(s, prefix, onlyatstart,\n stripwhitespace)\n if found:\n return result\n return \"\"\n else:\n for i in range(1, len(strings)): # i indexes the second of a pair\n if strings[i-1].find(precedingline) == 0:\n # ... if found at the start\n (found, result) = get_what_follows_raw(strings[i], prefix,\n onlyatstart,\n stripwhitespace)\n if found:\n return result\n return \"\"", "response": "Returns the string that follows the prefix."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_string(strings: Sequence[str],\n prefix: str,\n ignoreleadingcolon: bool = False,\n precedingline: str = \"\") -> Optional[str]:\n \"\"\"\n Find a string as per :func:`get_what_follows`.\n\n Args:\n strings: see :func:`get_what_follows`\n prefix: see :func:`get_what_follows`\n ignoreleadingcolon: if ``True``, restrict the result to what comes\n after its first colon (and whitespace-strip that)\n precedingline: see :func:`get_what_follows`\n\n Returns:\n the line fragment\n\n \"\"\"\n s = get_what_follows(strings, prefix, precedingline=precedingline)\n if ignoreleadingcolon:\n f = s.find(\":\")\n if f != -1:\n s = s[f+1:].strip()\n if len(s) == 0:\n return None\n return s", "response": "Find a string as per get_what_follows."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_string_relative(strings: Sequence[str],\n prefix1: str,\n delta: int,\n prefix2: str,\n ignoreleadingcolon: bool = False,\n stripwhitespace: bool = True) -> Optional[str]:\n \"\"\"\n Finds a line (string) in ``strings`` beginning with ``prefix1``. Moves\n ``delta`` lines (strings) further. Returns the end of the line that\n begins with ``prefix2``, if found.\n\n Args:\n strings: as above\n prefix1: as above\n delta: as above\n prefix2: as above\n ignoreleadingcolon: restrict the result to the part after its first\n colon?\n stripwhitespace: strip whitespace from the start/end of the result?\n\n Returns:\n the line fragment\n \"\"\"\n for firstline in range(0, len(strings)):\n if strings[firstline].find(prefix1) == 0: # if found...\n secondline = firstline + delta\n if secondline < 0 or secondline >= len(strings):\n continue\n if strings[secondline].find(prefix2) == 0:\n s = strings[secondline][len(prefix2):]\n if stripwhitespace:\n s = s.strip()\n if ignoreleadingcolon:\n f = s.find(\":\")\n if f != -1:\n s = s[f+1:].strip()\n if stripwhitespace:\n s = s.strip()\n if len(s) == 0:\n return None\n return s\n return None", "response": "Returns the string that is relative to the first line in the sequence of strings."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nretrieves an integer parameter from a sequence of strings.", "response": "def get_int(strings: Sequence[str],\n prefix: str,\n ignoreleadingcolon: bool = False,\n precedingline: str = \"\") -> Optional[int]:\n \"\"\"\n Fetches an integer parameter via :func:`get_string`.\n \"\"\"\n return get_int_raw(get_string(strings, prefix,\n ignoreleadingcolon=ignoreleadingcolon,\n precedingline=precedingline))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_float(strings: Sequence[str],\n prefix: str,\n ignoreleadingcolon: bool = False,\n precedingline: str = \"\") -> Optional[float]:\n \"\"\"\n Fetches a float parameter via :func:`get_string`.\n \"\"\"\n return get_float_raw(get_string(strings, prefix,\n ignoreleadingcolon=ignoreleadingcolon,\n precedingline=precedingline))", "response": "Retrieves a float parameter from a sequence of strings."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_bool_raw(s: str) -> Optional[bool]:\n if s == \"Y\" or s == \"y\":\n return True\n elif s == \"N\" or s == \"n\":\n return False\n return None", "response": "Maps Y to True and N to False."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfetch a boolean parameter from a sequence of strings.", "response": "def get_bool(strings: Sequence[str],\n prefix: str,\n ignoreleadingcolon: bool = False,\n precedingline: str = \"\") -> Optional[bool]:\n \"\"\"\n Fetches a boolean parameter via :func:`get_string`.\n \"\"\"\n return get_bool_raw(get_string(strings, prefix,\n ignoreleadingcolon=ignoreleadingcolon,\n precedingline=precedingline))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_bool_relative(strings: Sequence[str],\n prefix1: str,\n delta: int,\n prefix2: str,\n ignoreleadingcolon: bool = False) -> Optional[bool]:\n \"\"\"\n Fetches a boolean parameter via :func:`get_string_relative`.\n \"\"\"\n return get_bool_raw(get_string_relative(\n strings, prefix1, delta, prefix2,\n ignoreleadingcolon=ignoreleadingcolon))", "response": "Gets a boolean parameter from a sequence of strings relative to a given base."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_float_relative(strings: Sequence[str],\n prefix1: str,\n delta: int,\n prefix2: str,\n ignoreleadingcolon: bool = False) -> Optional[float]:\n \"\"\"\n Fetches a float parameter via :func:`get_string_relative`.\n \"\"\"\n return get_float_raw(get_string_relative(\n strings, prefix1, delta, prefix2,\n ignoreleadingcolon=ignoreleadingcolon))", "response": "Gets a float parameter from a sequence of strings relative to a given prefix."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets an integer parameter from a sequence of strings relative to another string.", "response": "def get_int_relative(strings: Sequence[str],\n prefix1: str,\n delta: int,\n prefix2: str,\n ignoreleadingcolon: bool = False) -> Optional[int]:\n \"\"\"\n Fetches an int parameter via :func:`get_string_relative`.\n \"\"\"\n return get_int_raw(get_string_relative(\n strings, prefix1, delta, prefix2,\n ignoreleadingcolon=ignoreleadingcolon))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nretrieves a datetime. datetime parameter from a sequence of strings.", "response": "def get_datetime(strings: Sequence[str],\n prefix: str,\n datetime_format_string: str,\n ignoreleadingcolon: bool = False,\n precedingline: str = \"\") -> Optional[datetime.datetime]:\n \"\"\"\n Fetches a ``datetime.datetime`` parameter via :func:`get_string`.\n \"\"\"\n x = get_string(strings, prefix, ignoreleadingcolon=ignoreleadingcolon,\n precedingline=precedingline)\n if len(x) == 0:\n return None\n # For the format strings you can pass to datetime.datetime.strptime, see\n # http://docs.python.org/library/datetime.html\n # A typical one is \"%d-%b-%Y (%H:%M:%S)\"\n d = datetime.datetime.strptime(x, datetime_format_string)\n return d"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfind the index of the line in strings that begins with linestart.", "response": "def find_line_beginning(strings: Sequence[str],\n linestart: Optional[str]) -> int:\n \"\"\"\n Finds the index of the line in ``strings`` that begins with ``linestart``,\n or ``-1`` if none is found.\n\n If ``linestart is None``, match an empty line.\n \"\"\"\n if linestart is None: # match an empty line\n for i in range(len(strings)):\n if is_empty_string(strings[i]):\n return i\n return -1\n for i in range(len(strings)):\n if strings[i].find(linestart) == 0:\n return i\n return -1"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfinds the index of the line in strings that contains contents.", "response": "def find_line_containing(strings: Sequence[str], contents: str) -> int:\n \"\"\"\n Finds the index of the line in ``strings`` that contains ``contents``,\n or ``-1`` if none is found.\n \"\"\"\n for i in range(len(strings)):\n if strings[i].find(contents) != -1:\n return i\n return -1"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_lines_from_to(strings: List[str],\n firstlinestart: str,\n list_of_lastline_starts: Iterable[Optional[str]]) \\\n -> List[str]:\n \"\"\"\n Takes a list of ``strings``. Returns a list of strings FROM\n ``firstlinestart`` (inclusive) TO the first of ``list_of_lastline_starts``\n (exclusive).\n\n To search to the end of the list, use ``list_of_lastline_starts = []``.\n\n To search to a blank line, use ``list_of_lastline_starts = [None]``\n \"\"\"\n start_index = find_line_beginning(strings, firstlinestart)\n # log.debug(\"start_index: {}\", start_index)\n if start_index == -1:\n return []\n end_offset = None # itself a valid slice index\n for lls in list_of_lastline_starts:\n possible_end_offset = find_line_beginning(strings[start_index:], lls)\n # log.debug(\"lls {!r} -> possible_end_offset {}\",\n # lls, possible_end_offset)\n if possible_end_offset != -1: # found one\n if end_offset is None or possible_end_offset < end_offset:\n end_offset = possible_end_offset\n end_index = None if end_offset is None else (start_index + end_offset)\n # log.debug(\"end_index: {}\", end_index)\n return strings[start_index:end_index]", "response": "Takes a list of strings and returns a list of strings FROM firstlinestart TO lastline_starts."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a list of fields from a CSV file.", "response": "def csv_to_list_of_fields(lines: List[str],\n csvheader: str,\n quotechar: str = '\"') -> List[str]:\n \"\"\"\n Extracts data from a list of CSV lines (starting with a defined header\n line) embedded in a longer text block but ending with a blank line.\n \n Used for processing e.g. MonkeyCantab rescue text output.\n\n Args:\n lines: CSV lines\n csvheader: CSV header line\n quotechar: ``quotechar`` parameter passed to :func:`csv.reader`\n\n Returns:\n list (by row) of lists (by value); see example\n\n Test code:\n\n .. code-block:: python\n\n import logging\n from cardinal_pythonlib.rnc_text import *\n logging.basicConfig(level=logging.DEBUG)\n \n myheader = \"field1,field2,field3\"\n mycsvlines = [\n \"irrelevant line\",\n myheader, # header: START\n \"row1value1,row1value2,row1value3\",\n \"row2value1,row2value2,row2value3\",\n \"\", # terminating blank line: END\n \"other irrelevant line\",\n ]\n csv_to_list_of_fields(mycsvlines, myheader)\n # [['row1value1', 'row1value2', 'row1value3'], ['row2value1', 'row2value2', 'row2value3']]\n\n \"\"\" # noqa\n data = [] # type: List[str]\n # an empty line marks the end of the block\n csvlines = get_lines_from_to(lines, csvheader, [None])[1:]\n # ... remove the CSV header\n reader = csv.reader(csvlines, quotechar=quotechar)\n for fields in reader:\n data.append(fields)\n return data"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef csv_to_list_of_dicts(lines: List[str],\n csvheader: str,\n quotechar: str = '\"') -> List[Dict[str, str]]:\n \"\"\"\n Extracts data from a list of CSV lines (starting with a defined header\n line) embedded in a longer text block but ending with a blank line.\n\n Args:\n lines: CSV lines\n csvheader: CSV header line\n quotechar: ``quotechar`` parameter passed to :func:`csv.reader`\n\n Returns:\n list of dictionaries mapping fieldnames (from the header) to values\n\n \"\"\"\n data = [] # type: List[Dict[str, str]]\n # an empty line marks the end of the block\n csvlines = get_lines_from_to(lines, csvheader, [None])[1:]\n # ... remove the CSV header\n headerfields = csvheader.split(\",\")\n reader = csv.reader(csvlines, quotechar=quotechar)\n for fields in reader:\n row = {} # type: Dict[str, str]\n for f in range(len(headerfields)):\n row[headerfields[f]] = fields[f]\n data.append(row)\n return data", "response": "Extracts data from a list of CSV lines starting with a defined header line and ending with a blank line."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef dictlist_convert_to_string(dict_list: Iterable[Dict], key: str) -> None:\n for d in dict_list:\n d[key] = str(d[key])\n if d[key] == \"\":\n d[key] = None", "response": "Converts a list of dictionaries to a string form."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef dictlist_convert_to_datetime(dict_list: Iterable[Dict],\n key: str,\n datetime_format_string: str) -> None:\n \"\"\"\n Process an iterable of dictionaries. For each dictionary ``d``, convert\n (in place) ``d[key]`` to a ``datetime.datetime`` form, using\n ``datetime_format_string`` as the format parameter to\n :func:`datetime.datetime.strptime`.\n \"\"\"\n for d in dict_list:\n d[key] = datetime.datetime.strptime(d[key], datetime_format_string)", "response": "Converts a list of dictionaries into a datetime. datetime object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef dictlist_convert_to_int(dict_list: Iterable[Dict], key: str) -> None:\n for d in dict_list:\n try:\n d[key] = int(d[key])\n except ValueError:\n d[key] = None", "response": "Converts each dictionary d [ key ] to an integer."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef dictlist_convert_to_float(dict_list: Iterable[Dict], key: str) -> None:\n for d in dict_list:\n try:\n d[key] = float(d[key])\n except ValueError:\n d[key] = None", "response": "Converts each dictionary in dict_list to a float."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef dictlist_convert_to_bool(dict_list: Iterable[Dict], key: str) -> None:\n for d in dict_list:\n # d[key] = True if d[key] == \"Y\" else False\n d[key] = 1 if d[key] == \"Y\" else 0", "response": "Convert a list of dictionaries to a bool."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreplacing a key with value in a list of dictionaries.", "response": "def dictlist_replace(dict_list: Iterable[Dict], key: str, value: Any) -> None:\n \"\"\"\n Process an iterable of dictionaries. For each dictionary ``d``, change\n (in place) ``d[key]`` to ``value``.\n \"\"\"\n for d in dict_list:\n d[key] = value"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nremove a key from a list of dictionaries.", "response": "def dictlist_wipe_key(dict_list: Iterable[Dict], key: str) -> None:\n \"\"\"\n Process an iterable of dictionaries. For each dictionary ``d``, delete\n ``d[key]`` if it exists.\n \"\"\"\n for d in dict_list:\n d.pop(key, None)"}